The Verse of Hajj-at-Tamatu was revealed in Allah's Book, so we performed it with Messenger of Allah ﷺ, and nothing was revealed in Qur'an to make it illegal, nor did the Prophet ﷺ prohibit it till he died. But the man (who regarded it illegal) just expressed what his own mind suggested.  

أُنْزِلَتْ آيَةُ الْمُتْعَةِ فِي كِتَابِ اللَّهِ فَفَعَلْنَاهَا مَعَ رَسُولِ اللَّهِ ﷺ وَلَمْ يُنْزَلْ قُرْآنٌ يُحَرِّمُهُ وَلَمْ يَنْهَ عَنْهَا حَتَّى مَاتَ قَالَ رَجُلٌ بِرَأْيِهِ مَا شَاءَ