I don't remember ever seeing a flag link.. so have never downvoted.
But I do upvote stuff I like and especially stuff I think is being censored by group think... so that could explain why my flag priviliges where never given or taken away before I could notice it..