If you host the server it is. Which is really easy. Then you can do stuff to add security like adding an IP whitelist perhaps along with other things.

Mumble! You can host your own mumble server, I used to it for privacy over Discord. Getting people to use it is another challenge.