We used to offer the Friday prayer along with the Messenger of Allah ﷺ and return (to our homes) while no shade of the walls was seen (at that time). (Using translation from Abū Dāʾūd 1085)
«كُنَّا نُصَلِّي مَعَ رَسُولِ اللَّهِ ﷺ الْجُمُعَةَ ثُمَّ نَنْصَرِفُ وَلَيْسَ لِلْحِيطَانِ فَيْءٌ نَسْتَظَلُّ بِهِ»