"The Messenger of Allah said: "Were it not for the fact that your people have recently left disbelief, I would have knocked down the House and rebuilt it on the foundation of Ibrahim, peace be upon him, and I would have given it a back door. For when the Quraish built the House, they made it too small.'" (Using translation from Nasāʾī 2901)   

لِي رَسُولُ اللَّهِ ﷺ «لَوْلَا حَدَاثَةُ عَهْدِ قَوْمِكِ بِالْكُفْرِ لَنَقَضْتُ الْكَعْبَةَ ثُمَّ لَبَنَيْتُهَا عَلَى أُسِّ إِبْرَاهِيمَ فَإِنَّ قُرَيْشًا حِينَ بَنَتْ اسْتَقْصَرَتْ ثُمَّ جَعَلَتْ لَهَا خَلْفًا»  

[Machine] I asked the Prophet ﷺ about the Black Stone, is it secure inside the Kaaba? He said, "Yes." I asked, "Then why don't they let people inside?" He said, "Your people cannot afford to provide for their expenses." I asked, "What about its elevated door?" He said, "Your people did that so that they can let in whomever they want and prevent whomever they want. If it wasn't for your people's recent departure from ignorance, I would have certainly had the stone removed and had its door placed on the ground."  

سَأَلْتُ النَّبِيَّ ﷺ عَنِ الحَجَرِِ أَمِنَ الْبَيْتِ هُوَ؟ قَالَ «نَعَمْ» قُلْتُ فَمَا لَهُمْ لَمْ يُدْخِلُوهُ فِي الْبَيْتِ؟ فَقَالَ «إِنَّ قَوْمَكِ قَصَّرَتْ بِهِمُ النَّفَقَةُ» قُلْتُ فَمَا شَأْنُ بَابِهِ مُرْتَفِعًا؟ قَالَ «فَعَلَ ذَلِكَ قَوْمُكِ لِيُدْخِلُوا مَنْ شَاءُوا وَيَمْنَعُوا مَنْ شَاءُوا وَلَوْلَا أَنَّ قَوْمَكِ حَدِيثُ عَهْدٍ بِجَاهِلِيَّةٍ فَأَخَافُ أَنْ تُنْكِرَ قُلُوبُهُمْ لَعَمَدْتُ إِلَى الْحِجْرِ فَجَعَلْتُهُ فِي الْبَيْتِ وَأَلْزَقْتُ بَابَهُ بِالْأَرْضِ»