Ask HN: Performance hit for jQuery Download?

I'm using JQuery for the first time on a new site. I love it, but I had some worries.

I really want my site to be snappy, but it seems that downloading JQuery, even when minified, can cause a decent performance hit.

Is this worry justified? If so, is there anything I can do? Can something like GWT do the same basic stuff that JQuery does (and thus reap the performance benefits from less code to DL)?

Also, if this is a problem, and since I see a lot of sites using jQuery these days, why can't the browsers bundle the framework in their browser so the user doesn't have to download it each HTTP hit (of course, using the bundled framework would be optional)?

Thank you.