I asked Messenger of Allah ﷺ about looking hither and thither in prayer. He replied, "It is a way of stealing by which Satan takes away (a portion) from the prayer of a person."  

سَأَلْتُ رَسُولَ اللَّهِ ﷺ عَنْ الاِلْتِفَاتِ فِي الصَّلاَةِ فَقَالَ هُوَ اخْتِلاَسٌ يَخْتَلِسُهُ الشَّيْطَانُ مِنْ صَلاَةِ الْعَبْدِ  

Once the Prophet ﷺ prayed on a Khamisa with marks on it and said, "The marks on it diverted my attention, take this Khamisa to Abu Jahm and bring an Inbijaniya (from him.)"  

أَنَّ النَّبِيَّ ﷺ صَلَّى فِي خَمِيصَةٍ لَهَا أَعْلاَمٌ فَقَالَ شَغَلَتْنِي أَعْلاَمُ هَذِهِ اذْهَبُوا بِهَا إِلَى أَبِي جَهْمٍ وَأْتُونِي بِأَنْبِجَانِيَّةٍ