The specific numbers would make a big impact, and it depends alot on what we define as 'harassment'.
Is it 20 among a class of 600? Or is it 20 among a class of 50? How pushy are they about it? Do they stop if I tell them I don't care once? twice? twenty?
These are all relevant questions but it feels like many folks are automatically treating it as if they make up half the class and they're doing the absolute maximum amount that's just under what would be considered 'harassment'.
For what it's worth, most of the people in my college classes (including myself often) didn't give a shit about any part of the class except for the grade.