The studies mentioned here suggest nicotine, stress, and plastics may be to blame: https://www.nytimes.com/2017/08/16/health/male-sperm-count-p...
Estrogenic substances such as atrazine have been correlated with lower sperm count in agricultural workers (e.g. https://news.nationalgeographic.com/news/2005/04/0427_050427...) but I don't know of a study showing the effects through the food distribution chain, only ones on field workers.