How to avoid picking the wrong technology just because it's cool

https://blog.bradfieldcs.com/you-are-not-google-84912cf44afb

by scarhill • 9 years ago

398 149 9 years ago

blog.bradfieldcs.com

AznHisoka• 9 years ago

I used to work at Standard and Poors. They do media monitoring for lots of financial companies.

When I joined, I thought they using state of the art tools to monitor millions of sources.

Turns out they just had a team of thousands in India that manually visited certain websites to check for press releases, transcripts, earnings reports, etc and push them to a database if they found them.

sharemywin• 9 years ago

wonder if that could be crawled and save money.

bm1362• 9 years ago

Woah there cowboy

sjg007• 9 years ago

or just set google alerts.

dismantlethesun5• 9 years ago

Accuracy. Every solution I've seen that relies on automatic crawling will eventually have a parsing error when someone changes their sentence structure of a press release.

It's not so obvious when you're looking at the breaking releases for a few stocks or companies, but historical records have at least 1 error per stock per year.

thehardsphere4• 9 years ago

Would writing and maintaining the crawler be less expensive than paying the small army of third world employees to manually check pages all day?

tomascot3• 9 years ago

financial companies usually use the worst formats to extract info from.

steve371• 9 years ago

this somehow made me laugh so hard. made my day~

kobeya• 9 years ago

laugh, and then cry

westoque• 9 years ago

Related. I know a company that does analysis for soccer games. I thought it was state of the art too until they told me they just hired a bunch of people from China that logged every move (distance ran, kicks, etc) from eveyone and also put it in the database.

tomascot3• 9 years ago

go on...

ci5er2• 9 years ago

Well, that makes sense, doesn't it? Back in "the day" when certain US companies were trying to convert all the Whitepages and Yellowpages nation-wide into CD-ROMs, they'd ship phonebooks from each zip-code to China to have them triple entered (error correction), and then that's where the nation-wide CD-ROM phone-books came from. This is the same deal, no?

rkwasny• 9 years ago

@AznHisoka I would like to hear more about this! This is exactly a problem we are trying to solve (http://evolution.ai)

One cannot even imagine how many people in the world have a boring job of looking at word document and C&P some of the data into excel spreadsheet. Other half have a job of looking at some document and effectively making Yes/No decisions all day.

Disclaimer, I am the CTO

tedmiston• 9 years ago

Sounds similar to Scale (https://www.scaleapi.com)

gaius5• 9 years ago

I'm sure they'd rather no job than a dull job... Oh wait.

s_kilk• 9 years ago

Humans are the cheapest machines.

komali22• 9 years ago

Regarding "UNPHAT": Is this... serious? Does the author genuinely hope that we will use this acronym as a means to help guide our technology choosing decisions? Is it not their creation and is just something I wasn't aware of yet?

Finally, do these forced acronyms ever help anybody else out there? I mean seriously, the "N" standing for "eNumerate?" The "P" standing for "Paper," which barely correlates to the actual meaning "consider a candidate solution."

Seems to me just saying "apply a principle of unfattening your technology decisions" would be a hell of a lot easier to remember.

paulddraper• 9 years ago

I doubt it. I think it needs to be three or fours letters. (E.g. Always Be Closing. Keep It Simple, Stupid.)

---

Trying my hand:

- understand the DOMAIN

- find the OPTIONS

- research a CANDIDATE

- know the HISTORY

- consider the ADVANTAGES

- apply deliberate THOUGHT

DOCHAT

komali22• 9 years ago

Hmm. That's a much better acronym, but it makes me come back to my original question - an acronym I guess is a way to make a process (dare I say algorithm) easier to remember, yea? So like, when you're switching to a new technology, the steps are as you listed, which are basically just

1. Understand what your problem is, and the potential solutions to that problem.

2. Use your brain.

3. Make an intelligent decision.

Or really just

1. Use your brain

I mean I just don't see the need here. Could be arrogance, I guess? Is this the very problem the author was trying to solve? Create a defined process for technology choosing?

jaclaz3• 9 years ago

Maybe simpler:

DOE

Don't Over Engineer

(which more or less brings us back to KISS principle)

ng12• 9 years ago

It seems to me just a complication of YAGNI.

nine_k• 9 years ago

YAGNI is dismissive. That other approach with a comparably silly acronym is an actually useful checklist.

mcrad• 9 years ago

Try DMAIC six sigma - Define/Measure/Analyze/Improve/Control

or OSEMN data science - Obtain/Scrub/Explore/Model/iNterpret http://www.dataists.com/2010/09/a-taxonomy-of-data-science/

vec• 9 years ago

Checklists and acronyms help some people, so more power to them. But I think the central observation of most of the anti-cargo-cult articles boils down to a simple maxim of "don't use a technology unless you understand what it's bad at".

There are very few technologies that can reasonably be thought of as strict upgrades, and the few that do exist (i.e. MySQL to Postgres) tend to be incremental enough that switching rarely justifies the migration costs. Instead, many solve one or two exceptionally dramatic problems (gargantuan datasets, huge write volumes, partition tolerant master-master replication, etc.) and are willing to make equally dramatic tradeoffs to achieve it. Saying Technology X is good at Problem Y is only half the story.

In my own practice I've found that forcing myself to stop and explicitly enumerate both the pros and cons of a new technology is usually enough to get my professional intuition to kick in. And 99 times out of 100 it tells me to just use boring old SQL and move on.

markrages• 9 years ago

GARABA!

joshribakoff4• 9 years ago

I've seen this in action. Using code generators to convert XML configuration to a few API end points. Or using a DSL/rules engine because you don't want to write code. Or having APIs that hit other APIs ad infinitum when the whole thing runs on one server because "micro services are the only right way". The result was we spent time gluing together what was already a monolith disguising as microservices, rather than adding features the customers wanted

More recently I had to solve time drift on 1000s of devices. The problem was someone installed puppet to manage those devices which uses NTP. The devices are behind firewalls so if they block the puppet master or mess with SSL puppet doesn't even phone home. Or worse it gets incorrect time from NTP peers on the network. The solution was to throw out the shiny tool "puppet" and just call "date". Puppet and NTP are great in theory for getting time down to the millisecond but totally backfired when some devices were off by over 24 hours. For our purposes as long as all devices were within 5 minutes we were good. The irony was after disabling NTP puppet just started it again. And we couldn't use puppet to fix that since 50% of our users had it blocked. No other choice but to throw out puppet and start over from scratch. The guy who spent months setting up puppet was not happy.

liveoneggs2• 9 years ago

the real issue is why the firewalls were randomly blocking puppetmaster and/or ntp and why the puppet ssl stuff stopped working (apparently randomly?)

Everyone involved sounds like they need a lot more experience.

joshribakoff4• 9 years ago

With all due respect, you don't know the real issue. Your response is the same thing the guy who installed Puppet said to me... just have them unblock it.

Our sales pitch is "these devices use plain http and will work behind your corporate firewall". The blockage wasn't an issue that could be solved, it was our whole business model to workaround the blocks by using simple http instead of https, proxying everything through our IP, and things like that.

Even the puppet documentation says not to run a puppet master when you have devices that are behind firewalls or limited network. The guy who added puppet apparently didn't read that.

I wasn't the one who decided the business model just the guy who fixed it to work as advertised while dealing with the pressure of everything crashing & burning. You're right no one had experience but thats not the point.

My point was that the fancier tools sometimes just add new issues without solving your real issue. Despite my lack of experience I solved the time drift using a linux built in "date" to set the date time. It didn't account for network lag like NTP, and an NTP developer would probably laugh at my solution, but now all devices are accurate to within a few minutes & that particular problem was solved. So don't always go for the most complex tool is all I'm saying.

For what its worth I do plan to bring back puppet but run it in "puppet agent" (offline) mode. We'll using custom scripts to copy in new puppet configs so puppet does not need to phone home.

Animats• 9 years ago

The author's acronym is silly, but it's a real problem. Soylent liked to blither about their "infrastructure", for a product that sells a few times per minute. They could be using CGI scripts on a low-end hosting service and it would work fine.

Wikipedia is some MySQL databases with read-only slaves front-ended by Ngnix caches and load balancers. That seems to get the job done. Wikipedia is the fifth busiest web site in the world.

Netflix's web site (not the playout system) was originally a bunch of Python programs.

The article mentions a PostGres query that required a full table scan. If you're doing many queries that require a full table scan, you're doing something wrong. That's what indices are for.

godzillabrennus2• 9 years ago

Queries that require full table scans? What are we talking about? WooCommerce?

claudiulodro• 9 years ago

Hey! Those all have indexes now. :)

squeaky-clean2• 9 years ago

I remember reading a scaling-out article from some startup. Some of the things felt a little over-engineered, some were impressive, some seemed wrong. But then they get to the point where they brag about their scale, and the metric they used was that they can handle thousands of requests.... per day.

opportune• 9 years ago

Be honest, were they hosted on their founder's smartphone?

aqme28• 9 years ago

https://engineering.hellofresh.com/scaling-hellofresh-api-ga...

Consultant32452• 9 years ago

Hey, that was a major milestone for a RoR app.

vacri• 9 years ago

At the other end of the scale, an acquaintance of mine is a sysadmin for a government department, and can't get their web framework changed to his frustration (politics...). He gave an example of the websites he has to look after, one being a boring legal website ("so you can imagine how low the traffic is already...") and the software could barely serve one request per second!

rrdharan• 9 years ago

Wikipedia is some MySQL databases with read-only slaves front-ended by Nginx caches and load balancers.

You're not wrong, but they are slightly more complicated than that: https://meta.wikimedia.org/wiki/Wikimedia_servers

They have Kafka, ElasticSearch, some workflow engine thing, and they manage a distributed object store (Swift) deployment.

Apparently in Wikimedia's case they also have a stated company policy / philosophy around self-managed infrastructure that prevents them from using public cloud for a bunch of stuff (I've heard this from a former employee but can't find any further documentation about it).

puzzle• 9 years ago

Wikimedia actually uses Kubernetes, but not on the serving path (yet?):

https://kubernetes.io/case-studies/wikimedia/

y4mi• 9 years ago

Kubernetes is essentially platform as a service. The parent didn't address the backend infrastructure,only the services serving the web content.

cooper12• 9 years ago

That's just for the Tool Labs (tools.wmflabs.org).

didip• 9 years ago

Netflix's web site (not the playout system) was originally a bunch of Python programs

That is fascinating, I would love to read up articles about this.

trhway• 9 years ago

Netflix's web site (not the playout system) was originally a bunch of Python programs.

seems that the bunch has grown slightly since then:

https://cdn-images-1.medium.com/max/600/1*OhONoCsb1KvesLy-n5...

(from https://medium.com/netflix-techblog/announcing-ribbon-tying-... )

Or this view https://image.slidesharecdn.com/devopsisraelpdf-141030000609...

(from https://blogs.mulesoft.com/dev/microservices-dev/api-approac... )

vim_user• 9 years ago

Yes, forget the acronym.

It's simply, "think for yourself." Do you ever talk to someone on a topic (in technology or politics) and it feels like you're talking to list of commonly accepted prepackaged talking points?

ceejayoz• 9 years ago

This happens all the time on StackOverflow/ServerFault. "What sort of server cluster do I need? I have thousands of hits a month!" Usually the answer is "an average smart watch could do it".

strken• 9 years ago

In fairness to Soylent, their marketing revolves entirely around the perception that they're a tech company and not one of many meal replacement shakes. The problem they were solving might have been "Techcrunch hasn't written an article about us recently", which an over-engineered stack does actually fix.

wwweston2• 9 years ago

Yeah, you can't buy that kind of publicity.

Wait...

pram• 9 years ago

A lot of this just sounds like Resume Driven Development, not people thinking they're Google or Amazon.

OpenDrapery• 9 years ago

I was thinking the same thing. I wonder how much of it is due to the way we make devs work and the hours we make them keep.

For example, I'd be more than happy to use the same old, tried and true, boring tools to just get the job done, if it meant that I could then go play golf or otherwise not be in the office.

But if you insist that I be in my seat 8 hours a day regardless of workload, then goddamn let's take this shiny new tool for a spin!

Do I want my resume to show that I used the same tool for every job for the last ten years? Or do I want it show some new hotness?

The industry and employers are as much to blame for this as the engineers, if not more. When you use middleman firms to find your employees, and all they understand is buzzwords, well then guess what game the devs are gonna play?

allcentury• 9 years ago

Fellow golfer and tinkerer - we are a product of the job markets. Hot employers typically want new and shiny on the resume in addition to fundamentals, seems like everyone is playing the same game...

alecco• 9 years ago

I recently worked at a company with a working batch legacy system and an ongoing update to Python and SQL with newer tools. It processed large data but it fit in a big server. 3x the amount of people required.

They replaced it with Spark and Scala. Rewrote the whole thing and bumped into the same previously fixed problems for 8 months. It was not faster and it was a pain to extract the result data for the online website. Oh, and they had only 1 server running Spark. On top of the existing one. They also dumped IRC with very useful plugins for proprietary Slack channels.

Yeah, their resumes are very competitive now with the Big Data hype. Worst part is the higher ups liked it because they had more keywords in the bag for investors.

We are living in very strange times.

mparr4• 9 years ago

This to me is the real problem, especially in a larger company. I mentioned "Resume Driven Development" at work the other day and got the response from a peer of "that's how it should be."

Convincing others to not chase the shiny new toy is harder than picking the right technology. The other posts under this parent are right: we are a product of the job market.

gtirloni2• 9 years ago

This is just a "my technology stack is better than yours" post like countless others we see daily. Sorry to dismiss it so abruptly but it gets tiring.

gtirloni2• 9 years ago

I think people downvoting tend to ignore the fact that the proposed "optimal" solutions for non-Google companies were at some point novelties themselves. If the same logic is applied we'd be using CICS app servers, IMS, and buying terminals.

At some point things change, the new normal changes, etc. The shift we are seeing in some areas also contributes to finally accepting the realities of distributed systems.

icebraining3• 9 years ago

The people are probably downvoting because you have missed the point of the article. Nowhere does it say that one shouldn't change things, or adopt new solutions.

eropple• 9 years ago

If you have the mindspace to be sorry, you have the mindspace to not do it.

And, since you clearly didn't actually read the article: he never says a word about his own technical preferences except "don't pick ones that don't fit your scale and scope".

thehardsphere4• 9 years ago

Wouldn't that actually require the author to promote their own technology stack?

Kiro• 9 years ago

Nothing I've built has ever needed anything more than a $5 Digital Ocean droplet and one of my services gets around a thosand requests a second at peak. Purely anecdotal and I'm not doing anything CPU intensive but I really feel startups are overdoing their infrastructure.

sidlls3• 9 years ago

It isn't just startups. And I agree with another commenter: it seems like resume driven development.

__jal2• 9 years ago

There's also a weird peer-pressure involved. Overheard a conversation a while back that summarizes it nicely - someone was talking about a scheduling system written they've used for years, and mentioned it was written in Perl. Another participant guffawed, and after the requisite Perl-bashing, the original person allowed that, yes, even though it worked fine, they should rewrite it.

No idea what company that was, but I'd love to work in a place where that was the most pressing concern on my plate.

draw_down2• 9 years ago

OK, but the other side of that is all of us looking at resumes and going "wow, that's cool that this person built that" instead of "that sounds overengineered and silly".

XorNot• 9 years ago

Startups? I work for a finance firm, and while we certainly have a need for large farms of servers to store data, my current team keep talking about web request latencies as an important infrastructure concern when the literal maximum number of users is in the tens of thousands.

Which is, you know, 1 maybe 2 machines with nginx plus 1 for redundancy. Our internal services are slow because people cohost them with batch jobs, and no other reason.

Twirrim• 9 years ago

It's so frustrating, because when it all boils down to it, the story is really quite simple:

1) What metrics reflect your customer experience? Monitor them, alarm on them. Target them as a priority for improvement. Check in on them weekly at a minimum.

2) What metrics make up the metrics that reflect the customer experience? Monitor them as well, and consider whether alarming on them is the right thing to do. Use these to direct what you target to solve 1.

3) What do you need for SPOF resilience? If you've got something critical, you need a minimum of two servers running it, and no more than, say, 40% total CPU usage so that you've got sufficient overhead to cope with a single host failure and any unexpected work increase.

copperx2• 9 years ago

You're not using Rails, aren't you? What stack do you use to handle 1000s of requests per second on a 512MB VPS?

therealdrag02• 9 years ago

I don't know about the specific overhead of rails. But if you had a uniform distribution of 2000 requests a second, and each request took up to 50ms, then you'd only have 100 requests at any given time. So each request could take up to 5mb, which is generous unless you're doing something explicitly fat.

dismantlethesun5• 9 years ago

Working in the D.C. area has given me a high tolerance for acronyms and backronyms (seriously: P.R.O.T.E.C.T. Act stands for "Prosecutorial Remedies and Other Tools to end the Exploitation of Children Today").

U.N.P.H.A.T does raise a smile to my face for trying, but if the author is reading, I'd suggest you change it to a prescriptive paragraph where the first word in each sentence becomes a letter in the acronym (e.g. B.A.M.C.I.S).

====

Here's my best try:

UNPHAT:

Understand the problem.

Nominate multiple solutions.

Prepare by reading relevant research papers.

Heed the historical context.

Appraise advantages versus disadvantages.

Think!

dismantlethesun5• 9 years ago

Drat, it's too late to edit my comment and take it out, but "Consider candidate solution" wasn't meant to be included in the acronym. It was part of my brainstorming.

dang5• 9 years ago

Ok we took that out for you.

gaius5• 9 years ago

Tupac was the master of this

GHETTO: Getting Higher Education To Teach Others

RUG3Y2• 9 years ago

BAMCIS: Begin Planning Arrange Recon Make Recon etc...that BAMCIS?

dismantlethesun5• 9 years ago

Yep. That one.

dang5• 9 years ago

We temporarily replaced this article's baity title with the text's more accurate self-description.

If someone would care to suggest a good title—i.e. accurate, neutral, and preferably drawn from the language of the article itself—we can change it again.

stickfigure• 9 years ago

FWIW, I think the original title was pretty good. I have had the unfortunate experience of screaming pretty much exactly those words (albeit replace "Google" with a different company from which one of my CEO's advisors came from).

EDIT: how about combine the two?

You Are Not Google: Another "Don't Cargo Cult" Article

nemild• 9 years ago

I like your proposed title, "Another 'Don't Cargo Cult' article" on its own seems dismissive, when the content seems quite useful for many engineers (the acronym need more work, though).

dang5• 9 years ago

"You are not Google" is a linkbait trope ("you") and also one very tired cliché.

aleksharma12• 9 years ago

"UNPHAT: one answer to cargo culting"

yumaikas• 9 years ago

I suppose another point worth bringing up is that hardware has made some pretty strong advances in recent, especially with SSDs being widely available. Stuff like that has raised the ceiling on what a single box can do compared to 2000 and earlier, when Google was building MapReduce at first.

jethro_tell2• 9 years ago

Pretty sure that's in the article is it not?

merb• 9 years ago

Rule 1: don't use any modern javascript framework.

marktam264• 9 years ago

I'm typing this impromptu but this article seems to be a qnd informal carnation of the Architectural Tradeoff Analysis Method (https://en.m.wikipedia.org/wiki/Architecture_tradeoff_analys...).

wwweston2• 9 years ago

Don’t even start considering solutions until you Understand the problem. Your goal should be to “solve” the problem mostly within the problem domain, not the solution domain.

I'd guess that this guideline alone would stop 2/3 adoptions of JS SPA frameworks (and 4/5 Angular adoptions!) if followed.

BigJono2• 9 years ago

At least the SPA frameworks themselves have a reasonably common legitimate use. The tooling around them is the major problem. Most projects I've worked on, even complex ones, could comfortably trim from 100 dependencies down to 10 and have the developers working on them be an order of magnitude more productive.

People wilfully wrestle with thousands of functions worth of APIs every day and don't even notice the immense slowdown it's causing them. It's especially bad in React land, which is ironic seeing as Sebastian Markbage at Facebook has an excellent talk about reducing API surface area.

lacampbell3• 9 years ago

I am fortunate, in that I got a lesson in not over-engineering things very early in my career.

My first programming job was a 3 month contract at the maintenance department of an international airport. They had a bunch of information in large, unwieldy ERP system and wanted to automatically generate job sheets for the different maintenance crews. So I did the simplest thing possible - I generated an excel file from the ERP system, then using that file as input, I outputted different excel worksheets for the different crews.

It was very plain GUI app that had one or two buttons. I remember being a bit worried that it wasn't nearly fancy enough for 3 months work, but everyone seemed pretty happy with it.

Later on I found out that - before me - they had hired an experienced software developer who had worked on the same problem for 6 months, and at the end of that 6 months had apparently not produced a solution. I had done the dumbest, simplest thing - not because I had any insight or wisdom, but because it was really the only thing I had the skills to do. But I delivered.

It was a brilliant, accidental first lesson in not over-engineering.

Can_Not• 9 years ago

Same here, I've watched multiple devs come into my first job with agendas of extreme over-engineering and unnecessary rewrites. I think the company is actually pretty lucky that they haven't gone under from all the damage.

gaius5• 9 years ago

I generated an excel file from the ERP system, then using that file as input, I outputted different excel worksheets for the different crews.

As a complete aside, you might be surprised how far you can go with Excel these days. Do you know it has a built-in in-memory columnar database now? You can have millions and millions of rows of data in there that you can use in tables and charts completely independently of the size of the grid. Pull back a huge chunk of data from the DB and slice and dice it to your heart's content locally.

I look at people buying expensive "business intelligence solutions" and I think, it's right there on your PC all along and you don't even know it...

collyw2• 9 years ago

The problem is people using Excel for everything that it shouldn't be used for.

NTDF9• 9 years ago

"Don't use tensorflow to predict everything"

dlwdlw• 9 years ago

Many people may not work at Google scale, but name would probably like to work at Google

How to avoid picking the wrong technology just because it's cool

Related Stories