اِنَّمَا السَّبِيْلُ عَلَى الَّذِيْنَ يَسْتَأْذِنُوْنَكَ وَهُمْ اَغْنِيَاۤءُۚ رَضُوْا بِاَنْ يَّكُوْنُوْا مَعَ الْخَوَالِفِۙ وَطَبَعَ اللّٰهُ عَلٰى قُلُوْبِهِمْ فَهُمْ لَا يَعْلَمُوْنَ ۔ ( التوبة: ٩٣ )
Innamas sabeelu 'alal lazeena yastaazinoonaka wa hum aghniyaaa'; radoo biany-yakoonoo ma'al khawaalifi wa taba'al laahu 'alaa quloobihim fahum laa ya'lamoon (at-Tawbah 9:93)
The cause [for blame] is only upon those who ask permission of you while they are rich. They are satisfied to be with those who stay behind, and Allah has sealed over their hearts, so they do not know. (At-Tawbah  : 93)
1 Mufti Taqi Usmani
Blame lies, in fact, on those who ask you permission despite being rich. They are happy being with women who sit back. Allah has sealed their hearts; so they do not know.