Page Weight Matters

http://blog.chriszacharias.com/page-weight-matters

by zacman85 • 14 years ago

717 164 14 years ago

blog.chriszacharias.com

plouffy• 14 years ago

I must not be very clever as it took me 3-4 attempts to understand the before last paragraph (e.g. that without Feather it took 20 minutes while with Feather only 2 minutes to load a video).

geuis• 14 years ago

Agreed. The wording of the story was a little hard to read. My first time through, it sounded like Feather increased load times, something about geography, and then the story ended without resolution.

Be sure to read it a couple times if you run into the same initial impression.

The clear point being made is that in many parts of the world, the original heavy page was taking a long time to load. Videos would only start after 20 minutes. Feather dramatically reduced the initial load time, but because of what was being measured (video start times), it looked as if Feather had increased load time. In reality, videos were being watched 18 minutes sooner.

zacman852• 14 years ago

I'll rework the wording. Also, it wasn't that the videos were necessarily taking 20 minutes to load. That was simply projections based on the numbers we were seeing at the time. There was not enough data of people actually waiting for 20 minutes to watch something, at least in the small subset of data we pulled at the time.

jarcoal• 14 years ago

I'm not really sure this post provides any useful information.

antiterra2• 14 years ago

The interesting bit is the somewhat paradoxical result of longer load times after optimization, due to the page suddenly being more tolerable to use by those with slow connections. This is similar to the (somewhat controversial) idea that adding new lanes to highways can increase congestion [1]. That may not have any apparent direct application for the general developer community, but it's at least a modicum more useful than your comment.

[1] http://www.fhwa.dot.gov/planning/itfaq.cfm

lloeki• 14 years ago

> This is similar to the (somewhat controversial) idea that adding new lanes to highways can increase congestion

This is non controversial and known as Braess's paradox[0]: changing the graph characteristics alters the Nash equilibrium solution for a worse one (i.e the Nash equilibrium is not optimal).

[0] http://en.wikipedia.org/wiki/Braesss_paradox

ricardobeat2• 14 years ago

> places like Southeast Asia, South America, Africa

Large swaths of South America have broadband access. I guess South Africa does too. Being a little more specific would be helpful.

RyanZAG• 14 years ago

While South Africa does have broadband access, most people sit on sub 3GB caps. The actual broadband penetration is also low - 2% for fixed line, and 11% for mobile access. The mobile broadband (3g) is metered per mb and frequently drops down to edge. It is improving, but as with the rest of the developing world, if you want to actually target most users you need to use SMS or very basic html sites that can work on older S40 Nokia phones and blackberries.

Obviously a fancy desktop sized page that relies on java script is just completely out.

ghshephard4• 14 years ago

While I appreciate (and agree with) the overall theme (Make your pages light, so people on < 10mbit/second links have access them, opening them up as a market to you - this works well for people on older GPRS/GSM links in the developing world) - I don't understand the math. Presuming there are parts of the world still stuck in 1997 analog modem technology, then 100 Kilobytes / 56.6 kilobits/second = 14 seconds.

At two minutes, there are people out there with 6.6 kilobit links connecting to youtube?

I suspect there might be a bit of hyperbole in this article as well, because, even if there were people connecting on ultra slow links, the average latency of those connections is likely to be be wiped out by the tens of millions of broadband links.

ShardPhoenix• 14 years ago

Perhaps there's a delay in doing those 14 different requests? My phone is very fast when downloading apps, but sometimes still takes a long time to load a page.

gwillen• 14 years ago

Most of the page-load delay tends to come from dropped packets and unnecessary round-trips, rather than from bandwidth limitations.

mdonohue• 14 years ago

So "page weight" isn't the issue then, contradicting the title. It's the pipeline depth of the page.

roopeshv• 14 years ago

single digit kbps is very common speed in some parts of the world, and you'd not think it's possible until you actually experience it personally. combine it with trying to multitask by loading multiple pages in browser, where by you are basically trying to share those single digit speeds among multiple pages.

ghshephard4• 14 years ago

I'm curious - what part of the world has single digit kbps links to the internet?

I spent a month all over Brazil, and, while it was painful trying to work from some areas near Belem - it was either 80kbits+/second on GPRS or no connectivity whatsoever.

toast0• 14 years ago

For small transfers, you're going to need to account for tcp slow start and latency instead of just dividing data size by bandwidth. You also need to include latency for DNS lookups. HTTP Keep-Alive and reducing domains referenced can help a lot here.

mamby• 14 years ago

No hyperbole, that's the truth. I'm an african and broadband is just a name! The net is faster but still very slow... Chris knows what he's saying :)

ghshephard4• 14 years ago

Oh, I know that developing country links are slow - I just thought it was a stretch to suggest that they are connecting to youtube so frequently after the page went from 2MB to 100 kilobytes that they shifted the average latency up.

It's okay though, sometimes a bit of hyperbole does a better job of getting the story across.

rtkwe• 14 years ago

It the page could download in one burst without any breaks the calculation would be correct. However those 14 requests aren't all run at once and end up triggering sequentially. It goes something like GET youtube.com, process reply, find other resources needed GET youtube.com's CSS, etc. etc.

Combined with link delays, dropped packets, long round trip times and the like you get a much slower nominal bandwidth.

chacham15• 14 years ago

My 1997 analog modem connected to the internet at an average speed of 3.3 kbps.

rglullis• 14 years ago

Let me FTFY: 3.3KBps

Joeri2• 14 years ago

On dialup (western europe) i used to get around 6 KB / s. Combine that with high latency (1 second roundtrips) and a slow machine and 2 minutes doesn't sound crazy to me for a 100 kb page.

And ofcourse, nowadays if i'm on a train my phone will often downgrade to gprs speeds, and it feels like i'm back on dialup having to wait a minute for a page to load.

jerrya• 14 years ago

I'm on a pretty fast line, and I am sometimes appalled with how long it takes pages to load, and often it comes down to loading content I have no interest in:

  * comments
  * ads and images
  * headers
  * sidebars
  * toolbars
  * menus
  * social network tools
  * meebo bars (really google, really?)
  * javascript to load all of the above crap

The amazing thing is how often I can't even begin to read the page for what seems like much of a minute as the page takes so long to render, or various pieces of the page jump, and shift and scroll.

I find that tools that localhost various ad servers help, and other tools that load the crap but keep it off the page help like adblock plus, but even more so, adblock plus' filters that let me shitcan all the crap.

One of these days I want to write an extension similar to adblock plus that seeks out and removes jquery crap. A lot of the reasons I can't read pages anymore seems to be jquery slideshows, jquery toolbars, jquery popups and the like.

I am pretty sure that graphing this out and we find the end of the web occurs sometime in 2018 when page designers and their bosses and engineers and marketing pukes have so larded down pages that the net runs out of available bandwidth and any page takes 4:33 to load.

SquareWheel• 14 years ago

If using Adblock Plus already, you might consider using fanboy's Annoyances list. It removes a fair bit of web clutter (though I often block unneeded elements on sites I frequent).

https://secure.fanboy.co.nz/

wtallis• 14 years ago

> "One of these days I want to write an extension similar to adblock plus that seeks out and removes jquery crap. A lot of the reasons I can't read pages anymore seems to be jquery slideshows, jquery toolbars, jquery popups and the like."

This might be something that NoScript's Surrogate Script feature could handle - it was originally developed to allow blocking Google Analytics without breaking sites that expected it to be loaded, and now includes surrogates for a lot of other junk.

erso3• 14 years ago

There seems to be an expectation these days that everyone's on a 50Mbit+ connection with an ultra mega blazingly fast computer, and nobody tests their sites on anything but that setup.

Example: The Verge. Chrome reports theverge.com as making 118 requests with 2.62MB transferred, taking 11.04s (onload: 6.42s, DOMContentLoaded: 3.32s). Kotaku: 234 requests, 1.35MB transferred, 11.93s (onload: 10.91s, DOMContentLoaded: 4.28s).

I tested this on a maxed out 2010 MacBook Air, and that's with AdBlock Plus and ScriptNo.

The Verge is basically unusable while it loads and Chrome struggles to render it after it has loaded while scrolling on my machine. This is all-too-common these days.

Retric2• 14 years ago

For comparison:

I got theverge.com in ~3 seconds from first click Using Chrome on a 40MB FIOS connection i7 2500k @ 3.6ghz w SSD & 16GB RAM on windows 7 without add-block or no script.

My PC is over a year old so it's probably the internet connection and then again the CPU is also 2-3+x as fast.

emp_• 14 years ago

Not disagreeing on ridiculous amount of requests, but The Verge (specially) and Kotaku audiences should lean towards high-end users.

nfm• 14 years ago

Is there a consulting market for fixing these kinds of examples? I quite enjoy optimizing these kinds of issues, but perhaps the value is too implicit or too low for them to bother.

keithpeter9• 14 years ago

Try it on a command line web client in your terminal. I use w3m as it has vaguely Vim like shortcuts.

Loads the front page in about 5 seconds on a 24 Kbit/s connection in the UK.

They could include a 'skip to content' link for those using text or audio interfaces, because the entire navigation stricture of the site appears to be included on every page. They also could make comments available in a way that makes them visible to a basic http client. Two minor mods like that would mean fast access to their reading material for cli nerds

cocoflunchy• 14 years ago

Just tried theverge.com on a not so powerful computer :

35 seconds and 6.50MB transferred !

lucian1900• 14 years ago

Ghostery takes care of several of those things.

lincolnbryant5• 14 years ago

Funny blaming jQuery (one of the leanest frameworks, striving to only fix the browser, not provide app architecture). Disable that and not much of the web will work. Remember these same jQuery-ites want to ignore IE6 and 7 which is what the people he describes from around the world are way more likely to be using.

lubujackson• 14 years ago

Just because people have bad net connections doesn't mean they run out-of-date software. Does anyone know how true this is?

001sky2• 14 years ago

The amazing thing is how often I can't even begin to read the page...

Cannot upvote this comment enough.

davars• 14 years ago

So the code to load the videos came in under 100KB. How large were the videos themselves?

stormbeta• 14 years ago

Keep in mind the request count too - lots of small packets can be considerably slower than their net size would suggest, especially if one gets dropped.

habosa• 14 years ago

I like this article a lot, it makes a good point that many of us often overlook. What's the dominant feeling about ajax to reduce page weight? It is ok if my page is 1MB but only the initial 100KB are sent synchronously and necessary for interaction? An example would be YouTube sending just the main page layout and video player and then async loading all of the related stuff and comments.

chubot• 14 years ago

It seems in theory that it should work. In practice the AJAX I've seen makes the perceived experience worse. There was a thread about Github latency a few months ago, and they use AJAX heavily in the source browser to hide latency, but it just seems janky to me.

I would be interested to hear if anybody is using AJAX to make very light sites that are accessible in Siberia and the places mentioned in this article. I can't really think of one. I'm sure it's possible though.

FWIW this is why craigslist is universally popular. Because you can use their site from the public library with IE5 from a dialup connection! When you're trying to overcome the chicken and egg problem of community sites, that extra population is probably pretty important.

danenania• 14 years ago

It can help, but still has to be applied carefully. A fast initial load is pointless if a bunch of widgets are making the page jerk and jump all over the place for the next 30 seconds.

The strategy I've used is to serve up a light frame with the meat of the content, while keeping all ajax/external site dependent stuff hidden (with equivalently sized placeholders to avoid shifting layout and scroll), then doing a single subtle fadein of the additional content when everything's ready.

You can tweak the timing--like if you have a few bits of ajax content from your server that take 1-2s to load, and a few social buttons that are taking more like 5-6s, you'd probably want to fade those in separately in two steps. The point is to carefully orchestrate how elements appear to the user, not just bombard them with everything at once.

lincolnbryant5• 14 years ago

People are not taking into account "browser crunch" of parsing and executing all of this javascript required to defer loading to ajax. This is painful to phones, but heavyweight frameworks like jQuery mobile are still used, even though we know just parsing jQuery can take phone several seconds. It could be quite a while before your $.ajaxes will fire.

zacman852• 14 years ago

I really like only using AJAX for content that is conditional. For instance, a comment form that opens up only when a user clicks on it. If they never go to add a comment, the AJAX request is never made. Anything that is actually going to be displayed immediately on the page should be loaded as soon as possible as there are major performance gains in letting the preprocessors do their work properly.

rorrr• 14 years ago

It helps a lot, assuming the user can use most of the page while you ajax-load the less important stuff.

Perceived speed is extremely important.

http://www.nytimes.com/2012/03/01/technology/impatient-web-u...

morpher2• 14 years ago

In addition to the developing world, page weight is a big concern for those of us with limited data plans. When a single log post weighs in at 2-3 MB (which is common), the day's allotment of a 200MB/mo plan disappears really fast. I often end up only reading the HN comments while on the bus since I don't want to play Russian roulette with the article size.

(of course, this is unrelated to YouTube, but to the general sentiment of the article)

STRML2• 14 years ago

There's at least one HN reader (I believe it's actually called "Hacker News Reader") that allows you to view the text-only cached version of article URLs. Definitely worth it if you're data-starved.

alec• 14 years ago

When I'm mobile, I like to read with ihackernews.com because it has a button that uses viewtext.org to just show me the text of the article.

orangethirty2• 14 years ago

Re: Page Weight.

I spend a bit more than a month studying how to build Nuuton in a way that it did not end up being a front-heavy piece of crap. Looked at a lot of differents options. Considered a client side Javascript MVC frameworks. Looked at the CSS side of things. Went as far as thinking about developing our own little app for Nuuton (No browser needed). But at the end, the simplest choice won. I decided to go with flat HTML/CSS. No bootstrap or framework either. Plain old markup and stylesheets. It uses very little Javascript (no Jquery either), just what the simple UI requires (an event listener for the enter key, and an event listener for 4 links). Nothing else. No animations. No fancy scrolling. No caroussels. Nothing like that. Its rendered on the server side (Django), and gets served to you in a very small package. I even went as far as breaking down the stylesheets so that you dont need to download CSS that does not apply to the page you are visiting. And you know what? It works, feels, and looks amazing. Even if the project ends up being a failure, acheiving such little victories are just deliciously fun.

Where did I get the inspiration to go against the current industry trends? HN's simple yet functional html setup. My god, its features a million tables, but the damn thing just works beautifully. By the way, Nuuton also uses tables. :)

dpe822• 14 years ago

Mind telling us what Nuuton is?

orangethirty2• 14 years ago

A new type of search engine. We call it an information engine, but just think of it as a seach engine that covers much more meta information than anything out there. nuuton.com to see the landing page. If you want to get an early look at it just email me.

nkuttler• 14 years ago

Fragmenting the CSS doesn't sound like such a great idea, as you will have additional requests on each page? But I guess you had that in mind.

neumann_alfred2• 14 years ago

You also have less stuff for the browser to consider when parsing the HTML. Depending on the amount of CSS, I'd rather have that and less memory footprint per tab. And its only as long as the browser doesn't have that permutation of CSS "includes" cached already, basically, once per each "page type" if you will, and only when you change from your current page type to a new one.

However, I don't say this from a usability perspective, but strictly from a "this is neat to code" standpoint. If you do it right, you should always be able to fallback to "include all CSS always" at the flick of a switch; so you have to code this first before you can even test if it helps ;)

Benferhat• 14 years ago

> I even went as far as breaking down the stylesheets so that you dont need to download CSS that does not apply to the page you are visiting

Doesn't this just result in five downloads of 1KB CSS files over the course of a five-page visit, when a single download (and cache) of a 1.8KB CSS file would have sufficed? Is that better?

aaronjg• 14 years ago

This is an interesting example of why randomization in experiments is important. If you allow users to self select into the experiment and control group, and then naively look at the results, the results might come up opposite from what is expected. This is known as Simpson's Paradox. In this case, it was only the users for whom page load was already the slowest that picked the faster version of the page. So naively looking at page load times made the pages look like they loaded slower.

However once Chris controlled for geography, he was able to find that there was a significant improvement.

Moral of the story: run randomized A/B tests, or be very careful when you are analyzing the results.

yuliyp• 14 years ago

You have to be careful of how you sample for the A/B testing. Even if they properly chose a set of users to get Feather, and a set of users to stay at baseline, their results would STILL get skewed, since the remote users who now got Feather would view disproportianately more videos than the baseline, pushing up average load time in the experimental group anyway.

001sky2• 14 years ago

Or be very careful when you are analyzing the results

== This.

Certain types of stratifications work, as in this case. Pure randomization in an expanding universe? I'm not so sure its foolproof. Though In general I agree with your comment.

winter_blue2• 14 years ago

Oh, I totally know it does because I've been stuck on and off on slow connections recently -- and a lot of pages (HN is big and awesome exception), load quite slowly. I even ended up using gmail's HTML interface for a while.

winter_blue2• 14 years ago

Right after writing this post I see OSFlash (Open Source Flash) on Canasse's profile. (Got there via https://news.ycombinator.com/item?id=4956809)

Eliezer• 14 years ago

Simpson's Paradox ftw!

tomfakes• 14 years ago

There's a great resource for testing page load speed for different browsers/locations/connection speeds.

Frankly, I'm stunned that they run this at no cost.

http://www.webpagetest.org

drivebyacct2• 14 years ago

Can someone with data give me a comparison of the people with the slowest connections and which browsers they use?

fudged71• 14 years ago

Posted above: http://i.imgur.com/JSQ48.png

marcamillion12• 14 years ago

This story reminds me of a founder I met recently, building a wildly successful project on the most meager of resources (continue reading, I include Google Analytics screenshots below).

This is a hacker in the truest sense of the word. He builds stuff just because he loves building stuff. He is a student at a University here in Jamaica. He isn't building it to be cool, or chasing the latest web 2.0 fad of the week. He didn't know HN until I introduced it to him, and he is young (say 19/20).

He built http://wapcreate.com - a WAP site creator.

That's right, WAP...not iOS or Android optimized HTML5 sites. Good, old fashioned WAP sites.

The most amazing part of the story though is that he is running it on 2 dedicated servers in Germany, it's a hack job (PHP, a bit of Ruby & a bit of Java). But once he picked up traction, he got so much traffic that he hasn't been able to keep the servers online.

In the first image - http://i.imgur.com/yEbyh.png - you will see that he got over 1.5M uniques. The vast majority of the time period covered here (the last year) was either very low traffic - pre-traction - or servers offline due to excessive traffic.

In the 2nd image - http://i.imgur.com/Pu8da.png - you will see that about 1.2M of those visits were in the 3 month period of June 1st - Aug 31st. His servers melted down towards the end of August and he ran out of money to pay his server bills. He eventually got it back up again a few weeks later, and the traffic spiked again and the servers crashed a few weeks later again.

In the 3rd image - http://i.imgur.com/HJ4gy.png - you will see that the vast majority of the visits are from Asia (even though he is 1 guy in the rural areas of Jamaica).

In the 4th image - http://i.imgur.com/JSQ48.png - and perhaps the most striking you will see the diversity of devices that the visitors are coming from. Most of them are from "feature phones". i.e. A multitude of versions of Nokia phones. Notice that this is just 1 - 10 of 508 device types.

He presented at a conference I went to, here in Jamaica, and he and I started speaking. I am helping him figure out how to proceed in a sustainable way. i.e. getting this thing stable, and then generating revenue.

After speaking to him for many weeks, I finally realized how insane his accomplishment is. Apparently, in all of this, he had been creating his site on computers that were not his. He either used his school computers, or borrowed machines from people. His Aunt is buying him a 2nd hand Thinkpad for Christmas - for which he is EXTREMELY stoked.

So while we are all chasing the billions doled out by Apple on the App Store and the newest, sexiest SaaS app idea with fancy $29/mo recurring revenue, with our cutting edge macbook pros and iPads - here is one guy using borrowed hardware, making a creation app for a technology that we have long since forgotten, generating crazy traffic and usage and struggling to even make a dime from his creation.

The world is a funny place, and this internet thing that we live on - is massive. As big as TechCrunch & HN are, there is so much more out there.

If you think you can help out in any way, either donating computing resources or anything else that can help us get this site back online and helping him start to generate revenue from this - then feel free to reach out to me.

P.S. If you want to dig into it some more, check out what some of the fans of the site are saying on it's FB page. I am not trying to trick you into liking the page. It only has ~1500 likes, but you can see passionate users commenting (both complaining about the downtime and praising some of the features).

zem• 14 years ago

truly amazing story. would have held my interest as a scene in a bruce sterling or neal stephenson novel. thanks for sharing it.

marcamillion12• 14 years ago

Thanks for the kind words...but I can take very little credit. The story kinda writes itself :)

keithpeter9• 14 years ago

"His Aunt is buying him a 2nd hand Thinkpad for Christmas - for which he is EXTREMELY stoked."

Thanks for posting this along with the data screen shots, very interesting. He has the right kind of aunt (I use recycled Thinkpads)! I take it you will be providing cloud storage/backup discs &c as well?

marcamillion12• 14 years ago

Well, Thank God for things like Dropbox - he has been using some amount of cloud storage & he has a backup external drive.

I will be doing my best to get it up and running - but given that it was initially written in PHP largely, and I am a Ruby guy we are trying to figure out the best way forward (in terms of a sustainability path with hosting resources).

Gertig• 14 years ago

Great story marcamillion, I had totally forgotten about WAP, it's so easy to get caught up with the tech that we use every day and forget about the rest. Good luck to your friend!

marcamillion12• 14 years ago

I know the feeling. I was floored (as in disbelief) - because of the same reason - when I saw his presentation and stats too.

Thanks for the kind words.

driverdan2• 14 years ago

That's a great story and congrats to him on building something like that under those conditions.

There's a good reason why "we are all chasing the billions doled out by Apple on the App Store and the newest, sexiest SaaS app idea": money. He's making no money and it will be very hard to monetize. How do you get someone on another contentent, who earns very little money, using old technology, to pay you and pay enough to turn a profit?

It's great that he's helping people but if it has no business model it's not sustainable unless he can turn it into a non-profit with donors.

lilsunnybee2• 14 years ago

That is really a sad state of affairs. You can create something that helps and is used by millions of people in the world, but unless those people have disposable income, you're likely screwed as far as earning enough to keep doing it. Better do something that a much smaller number of rich white people will pay you for instead.

suhastech• 14 years ago

very little money / person is fine as long as you have enough scale to back it up.

@parentcomment. I see that you seem to have a lot of traffic from India. You might want to check out http://vserv.mobi/ IIRC, they do have this payment processor which can pull money from your mobile prepaid balance. I don't think a lot of people here use credit cards.

mblake• 14 years ago

Have any specific steps been taken into monetizing his app?

Actually, let me re-phrase, what is the biggest problem he's facing?

I see that most of his visitors are from Asia, then why not reach out to Alipay payment processor? I am not overtly familiar with them, but maybe they have a cheap payment-by-sms product or similar (where users could pay a small price using their phone via txt message).

It is a very popular payment method in some areas of the globe for micro-payments.

marcamillion12• 14 years ago

Thanks for that suggestion. Never heard of it, but we will look into it.

The biggest problem, right this instant, is keeping the app up. I suspect it needs to be re-architected, because there is a lot of room for optimization so we can at least get the app working properly on the current hardware.

Once the site is online and stable, then it is a matter of monetization.

As for specific steps, yes...developing an ad network along side this app - and also using third party ad networks. The issue is that a lot of the sites that people create have porn.

I believe, when he gave the presentation the site had 19,000 WAP sites active. By active, he means updated in the last 7 days. He does a purge of all non-active sites every 7 days - because server resources are so scarce.

rwg2• 14 years ago

Since my home router and wireless access points all run Linux, I can play all kinds of sadistic games with routing, tunneling, traffic shaping, etc.

One of the experiments I did a while back was creating a "satellite Internet connection from halfway around the globe" simulator on a dedicated wireless SSID. Basically, I created a new SSID and used Linux's traffic control/queueing discipline stuff to limit that SSID's outbound throughput to 32 kbps, limit inbound throughput to 64 kbps, add 900 milliseconds of latency on sent and received packets, and randomly drop 4% of packets in/out. Very, very few sites were even remotely usable. It was astonishing.

I think one of the most useful products that could ever be created for web developers is a "world Internet simulator" box that sits between your computer and its Internet connection. (Maybe it plugs into your existing wireless router and creates a new wireless network.) It would have a web interface that shows you a map of the world. You click a country, and the simulator performs rate shaping, latency insertion, and packet loss matching the averages for whatever country you clicked. Then devs can feel the pain of people accessing their websites from other countries.

(Thinking about this for a minute, it could probably be done for about $30 using one of those tiny 802.11n travel routers and a custom OpenWRT build. It would just be a matter of getting the per-country data. Hmmmmm...)

scraplab2• 14 years ago

OS X has a handy GUI tool called the Network Link Conditioner, which lets you easily set up simulated network conditions. The inbuilt profiles are a bit optimistic, I find, but it's easy to add more.

It's just tc or the BSD equivalent under the hood, but it's point and click and easy to turn off.

More details at: http://mattgemmell.com/2011/07/25/network-link-conditioner-i...

rwg2• 14 years ago

Neat -- I'm still on OS X 10.6 and didn't know about this. Thanks for the pointer!

I guess the advantage of a dedicated pain box is that any device, regardless of operating system, could use it. Network Link Conditioner (and the two pieces of software mentioned so far by siblings of your comment) are all OS X-specific. (That's another webdev gripe for another day.)

madethemcry• 14 years ago

Oh that looks promising and it`s easier than my current solution. Thank you! For years I used the following small note saved on my mac to perform bandwidth tests of my application.

  #create firewall bandwidth configuration
  sudo ipfw pipe 1 config bw 15KByte/s

  #add config to firewall 
  sudo ipfw add 1 pipe 1 src-port 80

  #change. during testing
  sudo ipfw pipe 1 config bw 115KByte/s

  #delete. after testing
  sudo ipfw delete 1

melvinram• 14 years ago

There are software solutions for this, though you would need to specify the speed. For example, http://mschrag.github.com.

nwh• 14 years ago

There's a tampering proxy named Charles that allows this sort of thing on the fly, and for mobile devices if you so wish. I find it absolutely indispensable for finding the heavier parts of websites I'm working on.

http://i.imgur.com/69xFu.png

nanidin• 14 years ago

Brilliant idea! I've been traveling through less developed parts of the world and it is painfully obvious when you encounter a site that wasn't tested on anything less than broadband.

Also, the Facebook app on my iPhone (pre-rewrite) would almost always cause me to lose my edge connection to the cell tower when launched in certain parts of Indonesia.

grego• 14 years ago

Indeed, many who extol the virtues of cloud computing somehow have a blind spot to the fact that network coverage on this globe is not even. Actually it would be nice to see a color coded world map of network strength, is there one?

Bad network coverage is a problem that can be addressed many ways. Shameless plug: I've been working on offline wikipedia that by its nature is always available and fast, everywhere: http://mpaja.com/mopedi I'm sure there are many unexplored niches where the cpu power in your hand can be put to good use without needing the network.

dainix• 14 years ago

We are working on the same thing, get site as lightweight as possible! Oh man, thinking that for somebody site takes 20 min to load makes me cringe!!!

nikcub• 14 years ago

There are a few ways to see this data in Google Analytics. You can view a world map of your site with page load time and server connection time color-coded.

The first, and easiest way is to go to Content > Site Speed > Overview. By default this will show you a chart of page load time over time.

First, to get enough data change the time scale to a full year. Underneath the date picker there is an icon with 16 dots in it in a 4x4 arrangement, with some dots filled in. click on that and move the slider all the way to the right. This will ensure higher precision and will capture some of the slower page loads.

At the bottom, in the 'Site Speed' section instead of 'Browser' select 'Country/Territory'. It will change the data from pages to countries. Now click on 'view full report' and you will get a world map with page load times.

It will look something like this:

http://i.imgur.com/uCxcx.png

The site I just did it on doesn't have enough data, but if you have a fairly popular site you should see a nice variation in page load times.

Google have a post about this on their Analytics blog with much better maps and more information:

http://analytics.blogspot.com.au/2012/04/global-site-speed-o...

Their maps look a lot better.

The other way of doing it is to create a custom report. I have one called 'Global Page Load'. Add 'Country/Territory' as a dimension, and 'City' as a sub-dimension. You may also drill in further by adding browser version, mobile, javascript, etc.

As metric groups, I have - in this order: DNS Lookup time, Avg Server Connection Time, Avg Server Response Time, Avg. Page Load Time.

This then gives you a pretty report where you can immediately see which visitors are getting slow responses, and you can further drill in to see what type of connections and which browsers or devices are slow. I was surprised that my light page, with compressed CSS and every static/cached was still taking ~20seconds to fully load from 30% of countries.

That is for pages that load, you need to add another tab to the report with error metrics to see who isn't getting through at all, and you would need to look at server logs to see who isn't even loading the Google Analytics javascript. All very handy, and eye opening in terms of the types of connections and the speed of connections that a lot of web users are on.

To many sites are guilty of having pages that are just far too heavy - like they only test from their 100Mbit city based connections. I am in Australia with a 1st world internet connection at 24Mbit and I avoid theverge.com desktop site because of the page load.

Edit: If anybody can work out a way to share custom reports in Google Analytics, let me know - I would be interested in sharing reports with others, for specific cases such as this.

comforteagle• 14 years ago

Google's sitespeed tool only has a, usually quite small, 'sample' of your speed time.

igrigorik• 14 years ago

You can increase your sampling to rate to as high as you want, even 100%: https://developers.google.com/analytics/devguides/collection...

The cap is 10K visits/day.

HarshaThota• 14 years ago

> Edit: If anybody can work out a way to share custom reports in Google Analytics, let me know - I would be interested in sharing reports with others, for specific cases such as this.

Haven't tried it myself but it seems like you can share reports and dashboards in Google Analytics:

http://analytics.blogspot.com/2012/03/share-your-custom-repo...

https://support.google.com/analytics/bin/answer.py?hl=en&...

spyder• 14 years ago

What about compression? Is he talking about gzipped size or the uncompressed page?

KMBredt• 14 years ago

Feather even exists today: http://www.youtube.com/feather_beta and works as far as I can tell. You can opt into it using the link.

keithpeter9• 14 years ago

I live less than one and a half miles from the centre of a major European city. We currently get 0.24Mbits/s download and 0.37Mbits/s upload on ADSL over the phone line. My broadband provider's help line have confirmed no faults on line, or with my router (I've swapped the cables, router, and tried several computers). They think its an automatic profiling change because we are often away from home and switch the router off. I'm battling my way through the help line 'levels' which is proving to be 'interesting'. I may go to a smaller more expensive provider in the new year just so I can run an rdp session and get an interface I can work on.

The low bandwidth experiment has been educational. On Firefox/Ubuntu, you get the little status bar at the bottom that shows the requests. Some pages have a lot of those, and take ages to load. Distro-hopping is feasible (I'm trying out different interfaces), a CD-ROM downloads overnight quite easily. Software updates are a killer (go and have dinner, listen to some music...).

I've started using a command line web client (w3m in my case) to get to the text more quickly. You get to see which sites have 'skip to content' links embedded early in the page and with CSS set to not show them on Firefox. You also get to see which sites provide all content via html and which sites 'wrap up' their content in javascript functions &c.

As many here provide Web content and applications, just try a command line Web browser on your site...

pretoriusB• 14 years ago

>They think its an automatic profiling change because we are often away from home and switch the router off.

Why switch the router off? Just leave it running, it's not like it uses much power...

keithpeter9• 14 years ago

>Why switch the router off? Just leave it running, it's not like it uses much power...

I'm eccentric. I switch off the electricity, gas and water when leaving the house for longer than an overnight stay.

I'm also dubious: I've had this connection running for 4 years with the same use pattern and this 'profiling' thing has only been happening for the last couple of months. I'm also not clear why if it is software controlled rate limiting, someone can't just go and click a button somewhere!

It is surprising what I can do on 0.24Mbit/s/40mS latency. It makes fat Web pages slow, and YouTube silly. RDP is the killer, my work Windows 7 desktop is just painful to use. I pop down-town to a coffee shop.

adrinavarro• 14 years ago

I'm three miles away and I easily get 3 Mbps. Check your line (mostly attenuation and SNR). I've got 53dB attenuation and a SNR (ratio) of 10dB. Plus, uptime is pretty good (over 13 days now).

keithpeter9• 14 years ago

They claim to know that it is not the phone line itself. What are you using to measure the attenuation/signal to noise? I have to say the upload speed being faster than the download speed, together with the lack of any drop-outs or instability in the connection did make me wonder about rate limiting.

dsr_• 14 years ago

All of you people pointing out that AdBlock and NoScript and using w3m really helps reduce load times:

you're right. And missing the point.

A large chunk of the world is permanently bandwidth starved, and most of the mobile world lives on transfer caps and long latencies. If those are plausible scenarios for your audience, you need to reduce the quantity of invisible bits you are sending them.

What's an invisible bit? Any bit that does not directly create text or images on the screen is invisible. Anything that runs client-side. Some of it may be necessary, but most of it is probably boiler-plate or general-purpose when what you actually need is more limited. Reduce!

gregpilling• 14 years ago

I had this argument about 4 years ago with a young colleague who said I was out of date, and I am not a developer. Brett Tabke of Webmasterworld.com has been arguing for years that lightweight matters. It used to matter for dialup in the USA, now with widespread broadband it is less important to those people. Now it is important for mobile users and global users on dialup or slow connections. In a few years it will matter to the people that got their first android phone in some country with poor infrastructure.

Even personally it matters to me :) since I live in a 50 year old house that has slow DSL (YouTube buffers constantly) and I live in the center of Tucson, AZ.

I recall Google doing a study that increasing the speed of browsing increases the use of the internet. Page weight will always matter to someone.

neumann_alfred2• 14 years ago

Page weight will always matter to someone.

I'd even go further and claim it matters to everyone in some situations. Open youtube in one tab to listen to a long lecture, open more tabs.. slim webpages still load fast, but "regular" (derpy!) ones take ages.

And then it's also a resource. I turn water off when I don't need it, because wasting water sucks, not because I cannot afford wasting some. Why should bandwidth be much different? I'm all for using what you need, or for making experiments, but the quest for shinyness over content, putting all sidebars on all pages, including jquery plugins and whatnot at the drop of a hat etc.. all that leads to crappy sites and a crappy infrastructure. That is, the same infrastructure could allow people to achieve much more, if more sites were spartanic like, say, HN ^^

Because at the end of the day, I don't even care if I notice it. That argumentation is lazy and shortsighted. See, if the site loads in 200 ms instead of 250 ms, and I load the site a thousand times per year, that's 50 seconds per year -- over 10 years that's over 8 minutes. That may not sound like much, but consider that this is not the only site, and that I am not the only human (and that load time reductions of much more than 50 ms would be possible on a lot of big sites). In total, the amount of time wasted by petty BS is probably quite staggering.

cedricd• 14 years ago

This post makes a ton of sense. I spent a year traveling in the developing world a couple years back and was literally unable to watch a YouTube clip the entire time. It's amazing what parts of the Internet you can't see the moment you're on an insanely slow connection.

I was in Myanmar for a bit and their internet was so slow that I couldn't check even news or email -- no need for a China-style firewall.

lilsunnybee2• 14 years ago

It would be nice if newer browsers could start including average speed data in the http header. Then you could have servers that would offer up more or less weighty versions of the website based on projected load times. For browsers not attaching this header info, you could just assume the worst case scenario.

Even if maintaining multiple versions of the site would be a pain, it would at least be nice to do something like this during a site redesign. If a user has a fast enough connection, and load times aren't projected to be terrible, serve up the new heavier site. If not, serve up the older site. Then you could get some metrics about what percentage of your users still need the old design around, and how important it would be to cut weight on a redesign if needed.

I'm a technical lightweight so i might have messed up some of the details, but hopefully the idea is still interesting.

justnoise• 14 years ago

I wanted to post this last night but the internet went down in Vientiane (again) so I just went to sleep.

Having spent the last couple of months connecting from various parts of SE Asia, I'm happy to see this getting the attention it deserves. You don't realize how utterly painful or unusable much of the web is until you experience a network that is not only incredibly slow (by US standards) but also incredibly unreliable and prone to 5-20 minute outages many times per hour or day depending on where you are.

Before today, I hadn't clicked on a youtube video in months. The site was unusable.

Most evenings in smaller cities (not villages, that's a different story entirely) the broadband will be OK with downloads averaging around 3-15KB/sec. This allows you to access most sites or check gmail (usually I still opt for HTML mode). However, it's the frequent and intermittent outages that cause the most headaches. In many places I've visited they happen a couple of times per day and in other places, they happen many times per hour. These outages are a larger problem for AJAX heavy sites where clicking a button doesn't give a page saying "This webpage is not available" instead, the site just sits there, waiting for the request to complete, giving the user no indication that the rest of the internet is unreachable.

I have no idea what causes these outages (it's not the local network) but they lead to an awful experience on some sites. Hacker News, Wikipedia and other places do a great job of providing useful info that can be retrieved in a reasonable amount of time, countless others are so fat I've stopped visiting them entirely.

Think about the people and parts of the world you'd like to reach and design your site to have an enjoyable experience in those areas. I'm grateful for features like gmail's HTML only mode. I only wish other heavy sites had an equivalent feature.

vii• 14 years ago

I agree that browsers should make it much clearer that clicking on a button or link which does nothing is having no effect. In my mind this violates good UI design: that whatever action the user takes, they deserve some feedback. There are many very popular sites (including Facebook, at which I am advocating for new approaches) where pages are essentially unusable until JavaScript has loaded but little indication is given to the user.

The other point that the Internet is slow and high latency in foreign countries: true, but in the SF Bay Area it's slow and high latency on mobile devices and this should be a common experience shared by Hacker News readers.

Sami_Lehtinen• 14 years ago

Posting from Egypt. On a good day I might be able to download 100kB/s, usually during evenings it's more like 10kB/s with high packet loss & latency.

I often use Opera Mini, because most browsers and sites are just unbearably slow or do not work at all.

nnq• 14 years ago

> traffic from places like Southeast Asia, South America, Africa, and even remote regions of Siberia

...and now someone will figure out that this traffic isn't generating any revenue and you'll be back to catering the folks with high bandwidth and deep pockets.

WA• 14 years ago

What I don't get from the article:

Why could people in low-bandwith regions suddenly watch YouTube videos? Having the website itself down to 98KB is fine (which still took 2 minutes to load in some areas). But then you only get the first video frame.

So then you'd go and watch a video and that file is still the same size (several megabytes at least), which should take them still hours to load the entire video.

I understand that less requests and a smaller page weight does matter and I try to keep my webpages as light as possible, but YouTube seems to be the least interesting example for that, since the content itself can't be reduced much further without reducing either the image quality or the screen size.

twistedpair• 14 years ago

This has been available in the Google Web Toolkit since 2006. I'm not sure why a Google employee would not have been aware of it. If you use GWT, only the bare minimum of resources (JS) are loaded to render the landing page. Further JS will be loaded for subsequent pages/actions as needed. Images and CSS are inlined too so that there are no requests for them. "Perfect caching" of resources is built in through MD5 names for resources so that with a simple apache filter you never download static content twice.

I'm glad to Feather worked so well. It always drives me nuts when some major site/portal needs 250+ requests to load. If you like the sound of it (only 3 requests to load a complex app), checkout GWT as you get all of these optimizations (i.e. Closure) and more by just compiling your web app.

lnanek2• 14 years ago

YouTube is written in Python, which was just bought by Google around 2006, not even developed in house. Not even Gmail uses GWT last I checked...

erickhill• 14 years ago

My experience is this is most applicable for medium to large sites with very healthy monthly uniques. But it is excellent advice for all, particularly as more and more mobile devices access your site.

Some excellent best practices from the Yahoo team here on how to increase front-end site performance (which can account for "80% of the end-user response time"): http://developer.yahoo.com/performance/rules.html

gprasanth• 14 years ago

I thought this was the reason apps existed: to cut down loading time. I hate it so much when someone, in their app embeds a browser and load their site.

bevan• 14 years ago

This post prompted me to commission a script that calculates itemized page weight: https://bountify.co/ruby-script-to-display-itemized-page-wei....

Page Weight Matters

Related Stories