I asked the Messenger of Allah ﷺ about looking to the sides during prayer. He said: It is something which the devil snatches from a servant’s prayers.
سَأَلْتُ رَسُولَ اللَّهِ ﷺ عَنِ الْتِفَاتِ الرَّجُلِ فِي الصَّلاَةِ فَقَالَ إِنَّمَا هُوَ اخْتِلاَسٌ يَخْتَلِسُهُ الشَّيْطَانُ مِنْ صَلاَةِ الْعَبْدِ