By Allah, I think that she must have heard it from the Messenger of Allah ﷺ. I think that the Messenger of Allah ﷺ had not given up touching both of them but for the reason that they were not on the foundation of the House (the Ka'bah), nor did the people circumambulate (the House) beyond al-Hijr for this reason.
أَنَّهُ أُخْبِرَ بِقَوْلِ عَائِشَةَ ؓ إِنَّ الْحَجَرَ بَعْضُهُ مِنَ الْبَيْتِ فَقَالَ ابْنُ عُمَرَ وَاللَّهِ إِنِّي لأَظُنُّ عَائِشَةَ إِنْ كَانَتْ سَمِعَتْ هَذَا مِنْ رَسُولِ اللَّهِ ﷺ إِنِّي لأَظُنُّ رَسُولَ اللَّهِ ﷺ لَمْ يَتْرُكِ اسْتِلاَمَهُمَا إِلاَّ أَنَّهُمَا لَيْسَا عَلَى قَوَاعِدِ الْبَيْتِ وَلاَ طَافَ النَّاسُ وَرَاءَ الْحِجْرِ إِلاَّ لِذَلِكَ