I mean psychological safety, as in https://www.inc.com/justin-bariso/after-years-of-research-go... , or as in "will i face verbal bullying in this space". That is a different type of safety than physical safety, but it's very real. How much people experience not feeling psychologically safe varies widely, though.
I'm a visible minority, and my rights are frequently a part of political debate. In spaces where those debates are allowed to happen, it's often not safe for me to participate. It's a hard feeling to describe or put into words for someone who hasn't experienced others saying that they shouldn't be allowed to do participate in some aspects of society because of a characteristic they can't change.