Has anyone had success running video calls over Matrix without Element?
fwiw if you use the Jitsi widget, you can share the URL to your Jitsi room with anyone and they do not need Element or Matrix to join.
The whole voice/video thing is a design problem as protocols like Matrix or XMPP are designed for non-realtime communication, while voice/video is very much realtime. You cannot possibly federate a video stream, so Matrix or XMPP can, by design, only supply the routing (i.e. connecting the people who want to participate in a realtime chat) and the participants then need to agree on some server/software combo that provides the actual video conferencing.