They would be an oddity. That's often why one got reports of them. Because they were incredibly rare
Most of them like septimus severus you mention and the maurorum are north africans and then the historia augusta mentions a single black ethiopian. And it gets some weird amount of focus seemingly purely due to modern sensibilities in an anti racism drive and to create a sense of belonging. Same with how a lot of the story around the HMT windrush is wildly stretched or downright myth.
"It's not a stretch" .... "possibly manumitted later"....
Colonial era europe fairly certainly had many times more and outside of a few hotspots...they were still an oddity. They didn't just go back, didn't get selected out so that they didn't routinely show up in genepool research. They just aren't there in the demographic data. Fuck this is not that long ago. My great grandparents never saw one in the flesh.