You're inventing problems because you're using software for what isn't it's intended purpose. And why would you need to keep passwords a secret from people who you want to talk to in secret? You can kick, ban and make new private rooms whenever you want and send the password directly to the user you want to talk to. What's your issue?
It sounds like what you're trying to do would require your server to have a separate room for each pair of clients, then give those clients passwords for the rooms they're allowed to be in. You can make a script that does this automatically for all clients on the server and deletes the rooms when they leave, but this would be inefficient. Another/better option is to make a script that constantly reads what users write and listens for something like "call client_name" and makes a temporary room for the client who sent this and the other client ("client_name"), then make the server bot send passwords directly to the two clients.
Either that or a rewrite of the client software in a way that would allow two users on the server to connect directly to each other by pressing "direct call" on one another, which would place them in a temporary pocket server in a form of a direct peer-to-peer network (their temporary independent server) so that even if they're kicked out of the main server their call persists. This would mean that all client apps would need to include server functionality. File an issue on github about this and see if they'll agree to this. I'd like to see this feature, but this would mean that servers would become somewhat obsolete since they'd serve as nothing but a public phonebook of currently connected users.
I mean, if you just want calls then there are better options:
about.riot.im
app.wire.com
But yes, nothing beats Mumble when it comes to performance in it's *intended* way of use. It's primarily software for gaming groups with the addition of privacy and low latency. It's not meant to be a Facebook, Skype or mobilealls alternative, which is what you're looking for apparently.