What if the net number of votes a comment has received is hidden whenever it is between -10 and +10 (for example)? That way you are able to tell that a comment hasn't yet been rated highly, but also can't jump on the bandwagon of voting whichever direction the first couple of voters do. When a comment has a votecount somewhere near zero, you still have to think about it yourself (if you want to read it), but can also choose to ignore it if you only want to read already highly rated ones. If a comment at some point has, say, 12 votes and then receives three downvotes, it would return to the nebulous "small vote-count" status.
Another scheme I've liked on some sites is to show the counts for both up and downvotes, along with the total. A comment's having 10 upvotes and zero downvotes can be quite different from its having 100 upvotes and 90 downvotes, even though both equate to a net count of +10.