When you tell (your brother on Friday) to be silent while the imam is giving the sermon you are guilty of idle talk. (Using translation from Abū Dāʾūd 1112)
أَنَّ رَسُولَ اللهِ ﷺ قَالَ إِذَا قُلْتَ لِصَاحِبِكَ وَالْإِمَامُ يَخْطُبُ يَوْمَ الْجُمُعَةِ أَنْصِتْ فَقَدْ لَغَوْتَ