We used to offer the Friday prayer along with the Messenger of Allah ﷺ and return (to our homes) while no shade of the walls was seen (at that time). (Using translation from Abū Dāʾūd 1085)
كُنَّا نُصَلِّي مَعَ رَسُولِ اللهِ ﷺ الْجُمُعَةَ ثُمَّ نَرْجِعُ فَلَا نَجِدُ لِلْحِيطَانِ فَيْئًا يُسْتَظَلُّ فِيهِ