I asked the Messenger of Allah ﷺ about looking to the sides during prayer. He said: It is something which the devil snatches from a servant’s prayers. (Using translation from Abū Dāʾūd 910)
سَأَلْتُ النَّبِيَّ ﷺ عَنِ التَّلَفُّتِ فِي الصَّلَاةِ فَقَالَ اخْتِلَاسٌ يَخْتَلِسُهُ الشَّيْطَانُ مِنْ صَلَاةِ الْعَبْدِ