"The Messenger of Allah said: "Were it not for the fact that my people' - according to the narration of Muhammad he said: 'Your people' - 'have recently left Jailiyyah, I would have knocked down the House and given it two doors.'" When Ibn Az-Zubair was in power, he gave it two doors. (Using translation from Nasāʾī 2902)  

إِنَّ رَسُولَ اللهِ ﷺ قَالَ «لَوْلَا أَنَّ قَوْمَكِ حَدِيثُو عَهْدٍ بِجَاهِلِيَّةٍ لَهَدَمْتُ الْكَعْبَةَ وَجَعَلْتُ لَهَا بَابَيْنِ» فَلَمَّا مَلَكَ ابْنُ الزُّبَيْرِ جَعَلَ لَهَا بَابَيْنِ  

[Machine] The Messenger of Allah ﷺ said, "Did you not see that when your people built the Kaaba, they limited themselves to the foundations of Ibrahim?" So I said, "O Messenger of Allah, shouldn't they have restored it upon the foundations of Ibrahim?" He said, "If it weren't for the recent disbelief of your people." This is a summary.  

أَنَّ رَسُولَ اللهِ ﷺ قَالَ «أَلَمْ تَرَيْ أَنَّ قَوْمَكِ حِينَ بَنَوْا الْكَعْبَةَ اقْتَصَرُوا عَلَى قَوَاعِدِ إِبْرَاهِيمَ؟» فَقُلْتُ يَا رَسُولَ اللهِ أَلَا تَرُدُّهَا عَلَى قَوَاعِدِ إِبْرَاهِيمَ؟ قَالَ «لَوْلَا حِدْثَانُ قَوْمِكِ بِالْكُفْرِ» مُخْتَصَرٌ