How Node.js has revolutionised the Mail Online (Worlds No.1 Online Newspaper)

http://www.nearform.com/nodecrunch/how-node-js-has-revolutionized-the-mailonline#.UpMzaWTfw2E

by darsee • 13 years ago

34 56 13 years ago

This is a fascinating article, but I can't help thinking that I wish they were more technically incompetent so it harmed their business. The Daily Mail has to be one of the most repugnant media organizations and if there is ever a justification for using the word 'evil' in relation to an institution then they are it.

JonnieCache2• 13 years ago

Read this new yorker piece for an insight into how cynical they are: http://www.newyorker.com/reporting/2012/04/02/120402fa_fact_...

Apparently they realised that there's zero chance of the two readerships ever intersecting, so they let their guard down. It's pretty shocking, even to someone who already reviles them.

teh_klev• 13 years ago

My own thoughts were along the the lines of "what a waste of talent on such a scumbag publication".

"Tables are co-located, meters away from the news desk ‘the customers’, meaning it only takes minutes if something breaks for the communication between the developer and the user."

Meh, can you imagine sitting across from journalistic horrors like Melanie Phillips or Peter Hitchens, and being torn between fixing your shit quickly or doing the nation a favour and delaying publication of their right wing tripe from poisoning the weak minded for just a few minutes/hours longer.

slashdotdash• 13 years ago

Their journalists hold their opinions because they can earn a good living by doing so, not because they particularly believe them.

The Mail attracts writers, who ought to oppose it, because it pays them top rates on one condition only: they say exactly what the editor wants them to say. You can get at least £1,000 for a morning’s work, and Dacre will fill your pockets even if he decides not to use your piece. Writers will bark like a performing seal for money as easy as that.

http://blogs.spectator.co.uk/nick-cohen/2013/10/who-is-the-g...

lhnz2• 13 years ago

The Daily Mail position was paying upto £100k for their permanent software engineers.

That + being able to offer people a greenfield project using Clojure and Node.JS probably attracted great engineers.

enko4• 13 years ago

The Daily Mail has to be one of the most repugnant media organizations

Hm, I don't know. I sort of respect them, in a way. In some ways I think of them as the authentic voice of the impotent lower middle classes, lashing out at everything around them, greedy, cynical, childish and shallow, blaming others, ever seeking some outraged scandal to distract them from their otherwise mundane lives. Hate the Mail if you will, but you're basically hating their audience too.

I don't like the Mail, but at least they're independent, seemingly serving no-one's agenda but their own. Murdoch is much worse, in my opinion.

hipsters_unite4• 13 years ago

I hate the Mail because it encourages the worst in its audience. I have to believe that if the people that read it and I could reach a compromise if we weren't being so polarized by their reading material.

Being "greedy, cynical, childish and shallow, blaming others" are all vices at the end of the day (FYI despite the tone of my comment here, I'm actually not religious).

I agree totally that Murdoch is worse, but I think the Mail does hold more sway in a vocal area of the population with political power. The fact that they never acknowledge their hypocrisy or the damage they do I think is also offensive on an intellectual level that the Sun et al never quite reach as they're so cartoon-like.

oneeyedpigeon5• 13 years ago

As a fully paid-up member of the impotent lower middle class, and a fervent opponent of pretty much everything the daily mail stands for, I resent that implication!

Maybe the core audience of the daily mail IS lower middle class, but only the segment of the lower middle class that is intolerant and reactionary.

vixen992• 13 years ago

Is HN the place for this kind of witless snobby sneering unsupported diatribe?

vixen992• 13 years ago

Perhaps Reddit would be a better venue for you to get this sneering unsupported diatribe off your chest. Says much more about you than the Mail.

jbigelow76• 13 years ago

    >I think of them as the authentic voice of the impotent lower middle classes, lashing out at everything around them, greedy, cynical, childish and shallow, blaming others...

Odd, "greedy, cynical, childish and shallow, blaming others" sounds like an accurate representation of a large chunk of the elites and 1 percerters, it must be a species thing.

Joe8Bit• 13 years ago

It's interesting, I know several engineers who've worked at (or currently work at) the Mail Online and they all talk about the 'Daily Mail Tax' salary bump they required in order to slice off a little bit of their soul to work there.

Technical recruiters I know talk about it too, how they describe this very cool roll their trying to fill, deliberately obfuscating who it's for, before eventually having to reveal it and usually loosing most of their potentials.

Goladus• 13 years ago

I'm curious, can you be more specific about why they are "repugnant" and "evil?" Preferably something longer than 318 characters and shorter than the rambling 8,500 word New Yorker essay.

nodata3• 13 years ago

Worlds (sic.) number 1 online newspaper? Riight. http://www.mailwatch.co.uk/

nailer2• 13 years ago

They're not claiming to be the most ethical or nice paper, just number one in terms of readership. Which, like it or not, they are.

nodata3• 13 years ago

The world's number one newspaper in terms of readership? Show me the numbers!

babuskov2• 13 years ago

I expected that some Chinese, Indian and US news sites would have way more readers. It seems hard to get any hard data on this.

The original article says: "Every month they service over 150 million unique visitors"

This article from Forbes(2011) claims that The New York Times is the leader, followed by Times of India.

http://www.forbes.com/sites/kenrapoza/2011/09/05/tech-savvy-...

The article was based on Alexa ranking, which have changed since:

http://www.alexa.com/siteinfo/dailymail.co.uk

http://www.alexa.com/siteinfo/nytimes.com

http://www.alexa.com/siteinfo/china.com.cn

I guess they are using Alexa rankings to support those claims.

EDIT: If we use Alexa, the Huffington Post ranks above:

http://www.alexa.com/siteinfo/huffingtonpost.com

UK-AL2• 13 years ago

A lot of this personal responsibility, freedom to make choices stuff sounds like a system to place blame on individuals, instead of fixing the system.

anu_gupta• 13 years ago

My hosts file always contains this entry:

127.0.0.1 dailymail.co.uk www.dailymail.co.uk

This ensures that there is no possibility of my giving any traffic to this utterly repugnant shitfest of a newspaper.

I would encourage everyone to do the same.

CmonDev2• 13 years ago

"From a content management perspective, writing code in a very functional way is very low friction vs traditional OO models of building content management systems". It sounds like "using letters is very low friction vs using numbers". A new silver bullet/hammer?

PS: I like functional programming a lot.

coldtea• 13 years ago

It sounds like "using letters is very low friction vs using numbers".

No, it sounds like "using flexible, isolated, easily changable functional components and avoiding side-efects" is very low friction vs "designing and creating an OO class hierarchy, with all the tedium it entails in most common languages".

CmonDev2• 13 years ago

I think I misunderstood him but it did sound like OOP vs functional :). What he meant was probably something like "we stopped writing bad code with lots of mutable state and caught up with modern best-practices in OOP".

phillmv• 13 years ago

My theory goes, it's part and parcel of phase 3 in the language adoption cycle: it's now what a lot of people learn straight out of college.

It's really popular and quickly maturing - so lots of cool stuff that pays money is being built with it - but it hasn't crystallized into a morass of legacy work. Oh shucks, version 3 came out but it's largely incompatible with 50% of existing libraries, blah blah.

If your main experience developing software came from tweaking wordpress or uni C++/Java 101 then that's totally a reasonable feeling to have - "my goodness I am so much more productive".

dodyg• 13 years ago

You don't read the Daily Mail, you look at pictures. They put tons of pictures whenever event happens.

spindritf• 13 years ago

Excellent submission. I wish it went into some more detail on how they integrate those mini apps into the site or handle their failure but it was probably already long enough.

This

a Table also rebuilt the entire legacy Java frontend replacing the entire 155,000 lines of have with about 4000 lines of Clojure

was particularly impressive since a Table is just a small team of up to six people.

I also skimmed the front page of http://www.dailymail.co.uk and don't understand the allergic reactions in the comments here. A guy built a hobbit hut, some celebrity gossip, sports, more gossip... rather unremarkable.

For a website this big (32k pixels according to the article) it does load pretty quickly. Faster than a product page on Amazon (at least for me from Central Europe).

choult• 13 years ago

I believe the use of the phrase "all grown up" juxtaposed with loud headlines about the evils of paedophilia is one such reason the Daily Mail is reviled.

JonnieCache2• 13 years ago

The online version is a different beast to the print version.

alan_cx4• 13 years ago

There is a difference between the Daily Mail newspaper, which is a scare mongering right wing hate rag which has as much relation to truth and news as say Fox News, and its website which is pretty much a celeb stuff and sort of a soft porn peddler.

Its a weird pairing under the same brand. I don't know, but I don't think there is much crossover between the users of the paper and website. Personally, I find both hideous, but Im at a loss to accept that some one who gets all up set about gay marriage, immigrants, and the alleged loss of Englishness, what ever that is or was, is in any way interested in a Kardashian thing.

oneeyedpigeon5• 13 years ago

It's very difficult to hold any extreme view - right or left - without begin hypocritical[1]. The DM just excels at it.

[1] And, at an individual level, without encouraging the trashier elements of the media to expend enormous amounts of energy to expose those hypocrisies, however significant.

hipsters_unite4• 13 years ago

A recent scandal in the UK happened when the Mail publicly called a leading politician's father 'evil' as he was a Marxist. He was also a respected academic at a leading institution and fought for Britain against the Nazis, whereas the Mail was pro-appeasement and its owner friends with Hitler. Without wishing to appeal to Godwin's law, it's that kind of bald-faced moral pronouncement (with no regard to institutional hypocrisy) on subjects from immigration to gay rights that makes the Mail so divisive.

UK-AL2• 13 years ago

I think programmers these days focus too much on having whizz bang technologies.

90% of successful project implementation is culture, good practices and so forth.

I've seen well run cobol projects, and badly run projects using all the latest trends.

It doesn't really matter if your software is using Java, ruby or node.js. I can guarantee you from a business perspective, success does not rely on that. I much rather have well tested, well maintained Java Spring Code base, then a badly tested, unreadable clojure codebase.

The success of the story is that they probably went from a bad codebase to a good one. And you can in any language, node.js or not.

babuskov2• 13 years ago

I wish I could upvote this more than once.

Except maybe for the 4th paragraph. Using node.js instead of PHP cut my server costs to 20% of what it was before (simply because I can run the same thing on less number of cheaper servers). Lower cost means more profit and I count profit under "success from business perpective". ;)

alan_cx4• 13 years ago

Sure, but if you don't get that "90%" right first, your "20%" becomes irrelevant. Too many don't get the "90%" right.

jasonwocky• 13 years ago

Jerry Weinberg has an essay on fads in computer programming[1]. Basically, it's written such that you can swap in whatever today's hottest fad is, and the article still makes sense (it was originally written for Structured Programming, but OO Programing, Agile Development, Node.js, et al would fit well within it).

One of his conclusions is that the organizations that are likely to succeed with the latest whiz-bang technologies are also the ones who were most likely to succeed without them. Sure, what's new can be better, but it's hardly sufficient.

Based on my experience, I would agree, and so I think you're right. With the exception of this being specific to "programmers these days". I think our industry has always been plagued by this trend. It's just that now, with the internet, the stories have a longer reach.

[1] http://secretsofconsulting.blogspot.com/2011/06/beyond-agile...

petercooper2• 13 years ago

The entire backlog of 1.2 million articles are stored in elastic-search, which is used to drive the site

Elasticsearch is awesome and I'd heard rumors of people using it as a data store rather than just a search index but it's awesome to see it being used as a data store at such scale.

lost_my_pwd• 13 years ago

At the last place I worked, we used ES for both search and storage of 2.5MM+, and quickly growing, items (the data was also stored in DynamoDB for safety). It worked quite well, saved us some development time, and reduced the latency of search requests a little since there was no 2nd network call to fetch the documents found by ES from separate storage.

petercooper2• 13 years ago

What are the basic tradeoffs at that scale? I'm only using it for search at a very low scale but it almost sounds like a dream database if it works well at scale and for canonical storage.

JunkDNA• 13 years ago

Enter Fred George (of Programmer Anarchy fame). Fred came on board as Chief Architect, and focused on supporting the organisational move from traditional enterprise technology team to a nimbler, highly skilled organisation built around a strong culture of personal responsibility and trust.

I think all the other details in this write-up are not as important as this. The specific tech choice isn't the driver, it is this cultural shift. This doesn't mean the tech choice doesn't matter though. In fact, I would argue that the most rapid way to build such a culture in your organization is to self-select for people who would do well in that environment. I hate painting people with too broad a brush, but my experience has been that choosing safe, established technologies as your stack will cause the sort of people who like safe, predictable work to show up for jobs. It can be hard (though certainly not impossible) to recruit the alpha geeks and the people with a more entrepreneurial perspective. There are plenty of super-talented people who have very pedestrian technologies as their preferred platforms. But if you're looking to mix things up quickly and go for a more nontraditional environment, the shortcut to getting that is to choose tech that is beloved by people who primarily want to work in a nontraditional manner. The sort of person who dabbles with go or node in their spare time does it not because they expect that it's going to be their meal ticket to a cushy job doing software development at a bank somewhere. They do it because they have got intrinsic motivation to learn new things and be on the cusp of the latest tech out of sheer enjoyment.

barrkel• 13 years ago

it’s not possible to serve a 32,000 pixel webpage to a mobile device

I reject this prejudice.

The biggest annoyance I have with browsing websites on mobile devices is that they nearly always disable zoom.

Zoom is the probably the single easiest way to navigate large documents, because it lets the user scale effective panning speed. Without zoom, you're forced to pan at a fixed speed, and if the document is very long, it takes forever. But if you can zoom out, you can pan much much faster, and focus in on interesting bits of content much more easily. Many many times I've given up on websites because I wasn't able to get to the desktop view on my phone.

As to the infeasibility of a 32k page? The Nexus 7 tablet has the same resolution as my 24" main monitor. My phone isn't far behind. If anything, the question should be if 32k pages are infeasible for desktop machines, not mobiles.

whizzbang• 13 years ago

It's a interesting presentation but can anyone tell me why you would build an analytics tool and an advertising tool when there are already tools out there to do this? (Chartbeat, Doubleclick, etc...)

Maybe there have very specific use cases or have grown to the point were it makes economic sense to rebuild these but seems a bit redundant.

How Node.js has revolutionised the Mail Online (Worlds No.1 Online Newspaper)

Related Stories