If you were born in an agricultural settlement, that was probably the only lifestyle you knew
All of these societies would have had headers and hunting/foraging people on their periphery who they traded with. People regularly fucked off to go join those groups. This is well attested to in written records.