When you tell (your brother on Friday) to be silent while the imam is giving the sermon you are guilty of idle talk.  

أَنَّ رَسُولَ اللَّهِ ﷺ قَالَ إِذَا قُلْتَ أَنْصِتْ وَالإِمَامُ يَخْطُبُ فَقَدْ لَغَوْتَ  

The Prophet ﷺ said: Three types of people attend Friday prayer; One is present in a frivolous way and that is all he gets from it; another comes with a supplication, Allah may grant or refuse his request as He wishes; another is present silently and quietly with-out stepping over a Muslim or annoying anyone, and that is an atonement for his sins till the next Friday and three days more, the reason being that Allah, the Exalted, says: "He who does a good deed will have ten times as much" (vi.160).  

عَنِ النَّبِيِّ ﷺ قَالَ يَحْضُرُ الْجُمُعَةَ ثَلاَثَةُ نَفَرٍ رَجُلٌ حَضَرَهَا يَلْغُو وَهُوَ حَظُّهُ مِنْهَا وَرَجُلٌ حَضَرَهَا يَدْعُو فَهُوَ رَجُلٌ دَعَا اللَّهَ ﷻ إِنْ شَاءَ أَعْطَاهُ وَإِنْ شَاءَ مَنَعَهُ وَرَجُلٌ حَضَرَهَا بِإِنْصَاتٍ وَسُكُوتٍ وَلَمْ يَتَخَطَّ رَقَبَةَ مُسْلِمٍ وَلَمْ يُؤْذِ أَحَدًا فَهِيَ كَفَّارَةٌ إِلَى الْجُمُعَةِ الَّتِي تَلِيهَا وَزِيَادَةُ ثَلاَثَةِ أَيَّامٍ وَذَلِكَ بِأَنَّ اللَّهَ ﷻ يَقُولُ { مَنْ جَاءَ بِالْحَسَنَةِ فَلَهُ عَشْرُ أَمْثَالِهَا }