I asked Messenger of Allah ﷺ about looking hither and thither in prayer. He replied, "It is a way of stealing by which Satan takes away (a portion) from the prayer of a person." (Using translation from Bukhārī 751)
سَأَلْتُ النَّبِيَّ ﷺ عَنِ الِالْتِفَاتِ فِي الصَّلَاةِ فَقَالَ اخْتِلَاسٌ يَخْتَلِسُهُ الشَّيْطَانُ مِنْ صَلَاةِ الْعَبْدِ