When Messenger of Allah ﷺ came to Mecca, he refused to enter the Kaʿba with idols in it. He ordered (idols to be taken out). So they were taken out. The people took out the pictures of Abraham and Ishmael holding Azlams in their hands. Messenger of Allah ﷺ said, "May Allah curse these people. By Allah, both Abraham and Ishmael never did the game of chance with Azlams." Then he entered the Kaʿba and said Takbir at its corners but did not offer the prayer in it.  

إِنَّ رَسُولَ اللَّهِ ﷺ لَمَّا قَدِمَ أَبَى أَنْ يَدْخُلَ الْبَيْتَ وَفِيهِ الآلِهَةُ فَأَمَرَ بِهَا فَأُخْرِجَتْ فَأَخْرَجُوا صُورَةَ إِبْرَاهِيمَ وَإِسْمَاعِيلَ فِي أَيْدِيهِمَا الأَزْلاَمُ فَقَالَ رَسُولُ اللَّهِ ﷺ قَاتَلَهُمُ اللَّهُ أَمَا وَاللَّهِ قَدْ عَلِمُوا أَنَّهُمَا لَمْ يَسْتَقْسِمَا بِهَا قَطُّ فَدَخَلَ الْبَيْتَ فَكَبَّرَ فِي نَوَاحِيهِ وَلَمْ يُصَلِّ فِيهِ