In the lifetime of the Prophet ﷺ there was no wall around the Kaʿba and the people used to pray around the Kaʿba till ʿUmar became the Caliph and he built the wall around it. 'Ubaidullah further said, "Its wall was low, so Ibn Az-Zubair built it."
عَنْ عَمْرِو بْنِ دِينَارٍ وَعُبَيْدِ اللَّهِ بْنِ أَبِي يَزِيدَ قَالاَ لَمْ يَكُنْ عَلَى عَهْدِ النَّبِيِّ ﷺ حَوْلَ الْبَيْتِ حَائِطٌ كَانُوا يُصَلُّونَ حَوْلَ الْبَيْتِ حَتَّى كَانَ عُمَرُ فَبَنَى حَوْلَهُ حَائِطًا قَالَ عُبَيْدُ اللَّهِ جَدْرُهُ قَصِيرٌ فَبَنَاهُ ابْنُ الزُّبَيْرِ