It is not that japa of Harinam can be done only by chanting loudly.
namany eva haranty agham
tany evartha-karani ca
- Padma Purana
Diksha mantra japa can be done only mentally and Harinam can be done mentally as well as loudly. Material non-eternal sense-organs have got drawbacks, but there cannot be any defect in the spiritual sense-organs of a Tattvik Guru, who is established in the realised state. One who is established in the eternal nature of the real self, can chant correctly. Enslaved jivas with material egos and material sense-organs can never do any kind of bhajan, whatever they do that is only Karma, for material benefits.
We find evidence by Srila Bhaktivinod Thakur -
siddha-deha diya vrndavana.majhe sevamrta karo dana,
piyaiya prema matta kari more suno nija guna gayea
Srila Bhaktivinod Thakur has submitted his prayer to Supreme Lord Sree Krishna to bestow the embrosia of loving service to Him, by giving realised spiritual form. Supreme Lord, actually, by imparting prema to His associate, sings His own glories through the devotee and also hears His own glories. With material ego and material endeavour, actually you cannot do any kind of service to Supreme Lord.
Also, Srila Rupa Goswami has written in Bhakti Rasamrit Sindhu (1.2.234) -
na bhaved grahyam indriyaih
sevonmukhe hi jihvadau
svayam eva sphuraty adah
The Name of Supreme Lord is not material sound. In this material realm, material sound denotes a thing, material sound itself is not the thing. As word 'water' denotes the thing 'water'. Uttering water you cannot quench your thirst, you are to take the 'water' thing. So in material world material sound and the thing indicated by material sound are different, but in Transcendental Spiritual Realm, Transcendental Name of Supreme Lord and Transcendental Entity of Supreme Lord are non-different. Supreme Lord is the cause of all causes, and therefore, naturally, is Self-effulgent. His Transcendental Name is also Self-effulgent. Transcendental Name descends to a completely surrendered soul.
— Srila Bhakti Ballabh Tirtha Goswami Maharaj