it was not (solely) about slaves; this was debated in Congress during the process of drafting the amendment and resoundingly put down by contemporary legislators.
from Justice Jackson’s concurring opinion:
Senator Edgar Cowan, for example, argued that German immigrants’ children born in Pennsylvania should be citizens, but Chinese immigrants’ children should not—because Germans and Chinese were different. In response, Senator Trumbull emphasized that the law he had drafted drew no such distinctions. Undeterred, Senator Cowan would warn again—this time during debates on the Fourteenth Amendment—that the Citizenship Clause would let Chinese immigrants “overrun” California and “double or treble the population” of that State. Senator John Conness of California, where anti-Chinese sentiment was arguably most pronounced, responded that “the children begotten of Chinese parents in California . . . shall be citizens.” In fact, he said, the Civil Rights Act had already declared “that the children of all parentage whatever . . . should be regarded and treated as citizens of the United States.” No Senator rose to agree with Senator Cowan or dispute what Senator Conness had said. And no Senator said what the principal dissent says today: that the text at issue conferred citizenship only on freed Blacks and those in analogous situations.
- - -
further down, Justice Jackson cites the most forthright example of how blisteringly ahistorical the Republican party’s arguments are on this topic:
During the ratification debates, Senator Cowan took aim at the Roma people too, characterizing them as undeserving of birthright citizenship because they “wander[ed] in gangs,” “infest[ed] society,” and “impos[ed] upon the simple and weak everywhere.” And again, Senator Conness dismissed Senator Cowan’s prejudices: “The only invasion of Pennsylvania within my recollection was an invasion very much worse and more disastrous to the State, and more to be feared and more feared, than that of Gypsies. It was an invasion of rebels [at Gettysburg].”