Buy Don't Build

by jrott • 6 years ago

232 217 6 years ago

didip• 6 years ago

I agree.

If you are in decision making position, I struggle to see why would you want to spend so much energy installing Prometheus + Grafana stack instead of just using DataDog.

Even worse if your company attempts to build Prometheus/Grafana from scratch. Just why? What a huge waste of money.

If it contributes to your core business then ok, I can see why you may want to build custom solutions.

rantwasp2• 6 years ago

i think there is a huge difference between using prometheus and paying for datadog. the article argues for not building prometheus which makes sense in 99.95% of cases.

btw, the build vs buy should be a topic of discussion whenever people want to build something. at an absolute minimum they need to understand (and use) what’s already out there before they embark on the journey of building themselves

YetAnotherNick• 6 years ago

so much energy installing Prometheus + Grafana stack instead of just using DataDog.

Because it could be done in few commands the second time. And learning is only possible when wasting time on this kind of things, rather than trying hard to custom fit your usecase using some buyable tool.

That being said, I kind of partially agree with you and the author though. Don't maintain service just to feel that you are in control, but with the exception that if you are doing it for the learning purpose.

flumpcakes• 6 years ago

On premise Prometheus/Grafana could be lower than 5% TCO compared to DataDog, at least at places I have worked, and I am including employee time. This isn't particulary a discussion about cap ex. vs. op ex. either.

lhopki01• 6 years ago

Interesting you should bring this up as we are currently in the process of moving from Datadog to Prometheus/Grafana. What we've found is all the time you save in not setting up Prometheus we lose in support tickets to Datadog to get them to explain undocumented functionality and hidden "magic" they put in place. With Prometheus/Grafana the answers are always out there because of the sheer number of people out there. We've even resorted to sending known metrics on known intervals just to understand what Datadog does differently for monitors vs graphs. Also the majority of the time for a monitoring system is not spent on setting it up but rather on generating the dashboards and alerts.

Also with all buys systems there is a hidden costs in managing the legion of user accounts to access these new services. It's rarely as easy as just setting up SAML or single sign on.

The final thing that people never talk about is cost. Yes there is the cost of an engineer to set it up but often that costs is cheaper than the bought service. This is especially true of pay per api call type services.

phsource• 6 years ago

I'm totally on board with the overall message that building (i.e. engineering) your own internal tools come with lots of overhead, but take issue with this one point:

My counter-argument to that is there is also lock-in with internal systems. The most common version of this is the keeper of the spreadsheet.

The author then disparages spreadsheets as becoming the exclusive domain of one employee who wouldn't want processes to change.

In reality and my experience, though, spreadsheets are one of the most versatile and accessible systems, and their close cousins (Airtable, Notion) great as well! You can customize it to your own processes and they're pretty universally understood, so the barrier to change is pretty low.

jrott2• 6 years ago

Author here I actually think spreadsheets are great because they are so accessible. What isn’t great though is when there is a business processs that is a spreadsheet and knowledge that exists in one persons head.

holri• 6 years ago

It is a management failure when this is in only one person head. Outsourcing tools will not solve internal management problems.

rantwasp2• 6 years ago

i don’t think it’s the spreadsheet itself. it’s about the process and people who defend the process no matter what

systemvoltage3• 6 years ago

their close cousins (Airtable, Notion) great as well!

Their close cousins are locked, proprietary, slow, bloated, exceedingly complex, poorly designed (ui/ux), emojized, vc-backed feature extravaganza and have a subscription fee.

Experts of Excel use keyboard exclusively, their keystrokes are a melody of efficiency, expressivity and productivity that is continued to be mocked in similar fashion as the 2007-era Mac vs. PC advertisements.

cma• 6 years ago

Excel has a subscription fee as well.

tenaciousDaniel• 6 years ago

I found Notion to be fairly simple to get started. Whereas when I opened Airtable my head exploded and I closed it immediately and haven't revisited it since.

crave_• 6 years ago

It's also a strawman argument.

The alternative could instead be a system with open standards that many vendors implement, and only relying on standardized behavior.

This works to some extent with for example SQL or C, where you can migrate from one DB or compiler to another with limited effort.

serial_dev• 6 years ago

And it's also important to keep in mind that if your team is using various external tools, there is a high chance that only a couple of people will actually know how those tools work and even less of them will be able or happy to use those external tools.

cortesoft• 6 years ago

All of this is true, but sometimes people forget that buying a solution doesn't mean there will be no work. Sometimes, the integration and maintenance of that integration ends up being more than building and maintaining your own solution, especially if you have to do any large changes to how your systems work in order to integrate.

victor9000• 6 years ago

While I agree with this comment, if someone buys and ends up in the position you describe then they did a poor job at buying.

dnautics• 6 years ago

there may be no options. For example, I (profitably) ran a small cloud. My supervisor kept questioning me "why don't we just use openstack instead of your software"? The answer is even in this article!

Most enterprise systems require an engineering team to keep them running.

If we wanted a team of 3+ to run the cloud then we could buy openstack, or cloudstack, probably 2+, but that's also without pushing features. And suddenly we wouldn't be profitable anymore. I left, so I guess they will find out.

jeromenerf• 6 years ago

Not always a poor job, but sometimes a bet that turns out wrong. I have seen great products and companies disappear in unforeseen mergers or API prices skyrocketing for instance.

Risk is low for commodities, but high for anything "disruptive".

JMTQp8lwXL• 6 years ago

The people that have to perform the technical integration between two independently developed products (and could warn that the integration work is more involved than starting from scratch) are a different group from who decides the acquisition is happening.

aeternum• 6 years ago

Nowadays you're often buying SaaS so not only do you need to ensure the product is a fit currently but you must predict the fit going into the future.

It's not an easy thing to do. I've found that open source libraries are often more stable, require less ongoing maintenance due to API changes and have better support lifetimes compared to the SaaS equivalent.

q3k• 6 years ago

That's very true. Especially for tooling/products/services that sells themselves as shrinkwrapped products that only allow for a particular flow - sometimes reimplementing your own subset from scratch that's geared towards your usecase is less effort in the long run than hacking around a blackbox solution.

I'd much rather occasionally work on feature for a well-engineered internal reimplementation than constantly fight with shellscripts wrapping around a solution with a huge impedance mismatch.

themodelplumber• 6 years ago

Exactly. I've even had this happen with things like CSS, where a master stylesheet changes somewhere upstream on another server, and hundreds of hours of my own custom, vendor-supported modifications now make my web application or website look like doo-doo.

Plus maybe I'm weird but I think it's enough to love the energy you can get from building something. It's fun. And considerable portions of work should be fun. Especially if you work for yourself, where building stuff for money, for other people, often isn't enough without a strong connection between the work, customer, and the subject's own interests and values system.

(IMO a lot of those kinds of projects also build on the edge of one's core expertise, extending it outwards, almost like a recon mission, so it's less of a binary yes/no core expertise condition.)

marmaduke• 6 years ago

I think it's enough to love the energy you can get from building something

I think this is good if you can get there. The IKEA of software is pure dopamine. If had to grow and saw those trees less so. If you are pouring concrete and get it delivered, pure dopamine. If you have to crush the aggregate by hand, less so.

lexicality6• 6 years ago

My job involves a lot of integration work so I'm probably biased, but I find successfully cracking open a poorly documented API and extracting all the data in a usable way to be deeply satisfying, especially if you can be confident enough in your edge cases that you think it'll run until the next unannounced and undocumented change comes along.

NoOneNew• 6 years ago

True. At that, one thing about the buying vs building argument that's missing is picking and choosing your battles. Competitive advantage comes from that. Build what you are capable of producing effectively based on your environment. Obviously you can run into roadblocks, but part of being an adult is planning and adjusting based on your circumstances and needs, not along someone else's. Too many of these arguments fall into the realm of, "Everyone, listen up! I have the single solution for everyone's problem for all of eternity!"

blackoil• 6 years ago

That is why likes of Office 365 are popular, each component individually may not be best, but you get dozens of apps that are active in one go, for most parts are integrated well and have a simple billing.

gentleman11• 6 years ago

In unreal and unity, most of the third party code I’ve tried to use ended up unusable because of either bugs, performance issues, or limitations. There are great exceptions but it’s been frustrating overall. The ones that do work, well, the documentation is often terrible and it takes longer to learn and integrate than it would to make your own. Finally, unity has a history of retroactively changing their terms of service to squeeze more money out of you for asset store buys, so it’s scary to trust them. I write a lot more of my own systems now.

gfxgirl• 6 years ago

unity (and maybe unreal) have a real problem in that the majority of users are hobbyists, students, and new programmers. None of them are really willing to pay much for 3rd party code and they need $$$$$$$$$$$$$ of support because they are new.

For example make a JSON serialization library and you'll be asked "how do I spawn enemies from JSON" which is arguably entirely orthogonal to your library but don't answer and get downvoted for bad support. Do answer and you'll just be asked more questions about how to adapt your example to their personal project and you'll have to teach them programming in the process.

These incentives make it almost impossible to make money selling 3rd party code (plugins/add-ons). If you charge what it should really cost given the amount of work put in the market won't bare it. If you charge what the market will bare you'll go broke.

Maybe Unity should split the market into "Pro" and "Hobbyist" and the Pro market would have prices more inline with what it actually costs. Check out the prices of libraries like Radgametools.com (you'll have to google the prices) for comparison.

CyberDildonics• 6 years ago

Unity also changes their APIs around with every tiny version increment, meaning that any app store package has to be constantly on top of installing new versions of unity and updating all the little parts that break every week.

mauriziocalo• 6 years ago

+1. And I'd add to that that sometimes there's significant work in the pre-buy phase as well:

- Researching what your options are / what's already out there.

- Comparing different alternatives.

- "Hopping on a call" with a sales rep to get a product demo (there's this super annoying trend where many SaaS companies' landing pages don't explain what they do and the only option they give is to "schedule a demo").

For CRUD-like internal tools or simple 3rd party integrations, my experience has been that it's often much faster (typically < 1 hour) to build a production-ready app on Retool (https://retool.com) than it is to even get started with SaaS vendors.

lumberjack242• 6 years ago

For CRUD internal tools, I'd still consider taking a look at other software vendors in the space. Everyone is moving so fast really.

- https://www.forestadmin.com // Fastest way to build self-hosted admin panels on top of SQL (Postgres, MySQL...) and Mongo databases

- https://www.appsmith.com // Open source alternative to retool

- https://www.internal.io // A no-code alternative to all the above

- https://www.basedash.com // Very spreadsheet like XP, YC20 startup

pc86• 6 years ago

We recently purchased an enterprise scheduling software that I will not name (you may have heard of it, but it's unlikely). Told numerous times throughout the sales cycle by their architects and engineers that their API could do all the things we needed "easily" and that they could finish off the last few features we needed within 2-3 minor releases.

Fast forward almost a year, at least 7 releases, and probably $200k of payroll on our end and their product barely runs, and doesn't handle all of our standard workflow, let alone edge cases. Best we can assume is they were writing the API from scratch because we needed it and didn't have the skill to do so. Documentation was offensively bad when it was correct, which most of the time it was not, to the point where there were several instances of the endpoints in the docs being wrong, causing us to call support to ask what the real endpoints were we were supposed to be calling.

All this to say yeah, buying is great if the company is professional and has a well-documented product, and the sales and technical pre-sales folks know what they're talking about. But that isn't necessarily the case, even for six-figure purchases. And when it's not the case you can easily burn a non-trivial amount of money before you ever realize it.

corytheboyd2• 6 years ago

I have been in the boat of the company that ripped you off before. Sales will promise the world, product/engineering caves “just one last time” to help win the deal with the easiest shittiest thing. Years go by and things inexplicably break all the time.

It’s such an easy trap to fall into, because I hear similar stories all the time.

I guess the moral of the story is to assume sales is lying to you until proven otherwise? Haha not sure, just do as much research as you can but you’ll probably get burned anyway from time to time

pjc50• 6 years ago

Ah yes, this is how SAP and Oracle make their money: you "buy" a "COTS" solution, and then spend ten times as much having them build something for you .. which in the end you don't own.

Scarbutt• 6 years ago

Many in the "serverless" all crowd are blind to this.

signal11• 6 years ago

Precisely. There is a cost to integrating off the shelf software which people often forget. Less so in small startups, but in large enterprises this is non-trivial.

beckingz2• 6 years ago

I've heard that large companies don't mind SaaS because they used to budget 30% or more of a software purchase's sticker price on annual support and maintenance.

jrochkind1• 6 years ago

Good article, I am definitely filing away to pull out in some future team discussions.

This paragraph is confusing me, either it's missing a 'not' or something, or I'm just confused. Is anyone following?

The question you should be asking is what else could be done instead of tuning your own stuff or building a new internal system. The answer is usually spending more time coming up with the correct architecture instead of fighting fires or developing actual customer-facing features.

Instead of building internal systems you would be coming up with the correct architecture instead of fighting fires or developing actual customer-facing features? Wait, what? there are too many 'instead of's in there and i'm not sure they are all meant how they are said.

I think developing actual customer-facing features is the goal, but here it sounds like the thing you would like to avoid... and I'm not really sure about fighting fires (ideally you want a system where you do less of that, right?) or coming up with the correct architecture (?).

jrott2• 6 years ago

Yeah developing actual customer facing features is the goal. That paragraph needs to be edited

blakesterz• 6 years ago

I think he's right, if you're at a small/medium sized place, all the reasons he lists there are solid reasons to buy and not build things for yourself. That being said, there's no reason to make that an absolute rule, I think it's ok to build some things and buy some things. Just think of all the things we've ended up with because someplace built some thing and it ended up being some big thing that we all know. Isn't that how Slack and Docker both got started?

whoisjuan3• 6 years ago

I own a Screenshot API service (shameless plug: https://getscreenshot.rasterwise.com/) and this exactly the argument that I make for small utilitarian services like mine. You shouldn't build them because buying them is several orders of magnitude cheaper.

I have a small paragraph from a blog post where I explain this with some math that is probably crappy but captures the idea:

"When you spend time in areas that are not the core of your product, you're actually being financially inefficient. Taking screenshots is likely not a core task of your business so it doesn't makes much sense to waste development resources in this area.

Here is a brief example of how inefficient it gets:

A mid-level developer in a small market earns $90K USD per year, working 40 hours per week. His/her effective hourly rate is $47 USD per hour.

Writing a basic implementation of a screenshot utility will take at least 5 hours. But this is just a simple prototype. Many use cases will need to be addressed, and there's likely going to be a large amount of time spent in optimizing, securing, provisioning, testing, etc.

A realistic utility that can be used for a development workflow is going to take at the very least 30 hours of development time, but likely more.

Other un-accounted areas of development time are: documentation, training and maintenance.

Given this scenario is very likely that writing a semi-good solution will take more than a week and maintaining it, will take at least an hour every month.

This means that writing this solution will cost almost $2000 USD of development time plus another $500 USD or so, just to support it every year. And of course, this doesn't include the cost of the infrastructure you're paying to run it."

My argument is that the more utilitarian the service, the more cost-effective that is to buy it instead of writing it. Writing it could be simple but there's a drain in engineering resources when you try to maintain it in the long run.

mewpmewp2• 6 years ago

That's cool. How much money do you make with that service?

whoisjuan3• 6 years ago

About $500 USD/month. Haven't put too much effort to promote it though.

ianmcgowan• 6 years ago

As an aside, love your Lincoln, Hamilton, Jackson pricing plan, super creative!

whoisjuan3• 6 years ago

Thanks! I wanted to do something creative with the pricing. :)

ummonk2• 6 years ago

I think the screenshot service is a perfect example of a service suited for buy don't build - it's something complex to implement but with a really simple API to integrate with.

elcritch• 6 years ago

Yah it’s a pretty ideal example. Another might be CI. It seems from reading other comments that bad examples in buying external products are from complex “business apps”.

lexicality6• 6 years ago

One of the few things I appreciate from my abusive first boss was making me acutely aware of the value of my own time.

He was incredibly stingy with money and always wanted to know how much it would cost him to have his engineering team implement something vs how much it would cost to buy it and integrate it.

Admittedly he was pathologic about this to the point of having us create a timer that counted in pounds sterling that he'd start running at the start of every meeting, but I feel like it made me way more efficient with my time (and hate long meetings)

human2• 6 years ago

I had a similar client early in my career. Working for him was a love/hate relantionship. He tripled my hourly rate telling me my time was highly valuable but then had very high standards and aggressive timelines.

FpUser3• 6 years ago

I think you had great boss (assuming he did not start timer each time you stopped typing on keyboard). Those endless meetings become unproductive pretty fast. I suspect that in big part they become a self sustained cancer used to justify time and salaries of unproductive company members.

As for the subject itself: frankly I do not see any advantage of hosting on say Azure over hosting myself. Either requires a good deal of maintenance. And no you can not really rely on Azure doing it for you.

Uptime for me is probably just as good on my servers as each one I have in my office also has ready to use up to date shadow copy located elsewhere. Azure is way more expensive of course.

The amount of hardware/processing power I have on my own server would cost me a fortune to have on Azure. Scalability does not matter much either because:

I am not Google and do not have to serve the rest of the world.

On top of that my servers are usually high performance native C++ applications that can handle thousands of requests per second sustainably and without breaking a sweat.

Vertical scalability with modern CPUs and multilevel storage is absolutely insane.

lexicality6• 6 years ago

frankly I do not see any advantage of hosting on say Azure over hosting myself.

In my opinion, cloud hosting only makes sense if your systems are architectured around being hosted on the cloud.

If you can build your application in such a way that it runs in the free tier of serverless, then it's going to be considerably more cost effective than renting a dedicated server - but if you can't, then it absolutely won't be.

wenbin2• 6 years ago

Totally agree.

Some engineers love to say "I can build this over a weekend". But the main cost is almost always on the maintenance side.

I forgot where I saw this number - in the entire lifecycle of a piece of software, (on average) maintenance cost is at least 8x of the initial dev cost.

Yes, buying a solution also costs time (and $$$) to integrate and maintain, e.g., using a 3rd party API - time to write code, time to set up monitoring / alerting, error handling... There are always exceptions, edge cases, special situations...

But in general there are fewer and fewer things that a web company needs to build from scratch. It takes less engineering time to launch a web product than before, thanks to all those out of box solutions, e.g., SaaS, apis...

I like building a software/web business today than 10 years ago :) One-person (or tiny team) web businesses will be very common (I'm running one [1]), and running an API business is not bad because it actually saves customers time & money(vs building one in-house) [2] thus they are willing to pay a small fee (compared with hiring one or more full-time engineers).

[1] https://www.listennotes.com/blog/the-boring-technology-behin...

[2] https://www.listennotes.com/blog/how-i-accidentally-built-a-...

satyrnein2• 6 years ago

I think "maintenance" somewhat mischaracterizes it. In my experience, it's often that an engineer can build the typical use case over a weekend, but will have to continue pouring in more dev time while it's in production to handle a huge number of edge cases that weren't properly handled originally. That work is only "maintenance" because you already claimed the work was completed, and finance has it marked as having been put into service, so you can't capitalize further costs against it.

wenbin2• 6 years ago

Yea, in software development, it’s always work in progress. There’s not a clear finish line.

The word “maintenance” can’t perfectly capture the meaning of all the continuous development, small incremental improvements, bug fixes, operational tasks...

dilyevsky• 6 years ago

Buy if you want to save money, build if you need it to work.

thinkharderdev4• 6 years ago

These sorts of discussion always seem somewhat confused to me because "buying" vs "building" is not a binary choice. It is more often a choice between:

1. Buy some platform/framework which you will then need to hire a small army of costly consultants to integrate and customize for your particular business need. Or

2. "Build" your own solution by orchestrating a bunch of open source technologies to solve your problem.

Moreover, it is not quite clear up front whether 1 or 2 will be costlier in terms of development effort and overhead.

recursivedoubts2• 6 years ago

excellent point

barnaclejive• 6 years ago

This is 100% correct

dvtrn• 6 years ago

It would seem to me there are still two choices to make, meaning it’s still a binary choice-merely with more caveats and words involved to describe the options?

thinkharderdev4• 6 years ago

Yeah, that was phrased somewhat awkwardly. The binary choice I was rejecting was between

1. "Buying" a solution that solves your problem exactly and doesn't require engineering resources to implement and maintain

2. "Building" a solution where you have to solve every problem from scratch where you don't have any particular expertise.

localhost8000• 6 years ago

I understand your general point of there being a continuum of paying for building blocks to be integrated. But,

is not a binary choice

Then, offering two options. :)

thinkharderdev4• 6 years ago

Haha, yes that it is a bit awkward

bdcravens3• 6 years ago

Sometimes though we are dishonest about #1, and we want a deeper level of integration or polish than is necessary.

Raidion• 6 years ago

I think it's a matter of unknown future needs that's the problem. If you want to do feature flags, you can basically build a service that stores key value pairs segemented however you please, throw it behind redis cache, and call it a day. This is faster and probably cheaper than buying a SaaS!

However, if you want a bunch of neat features, custom rollouts, and all that stuff, you're simply not going to be able to get the same value (unless you have HUGE scale) building it, and should just buy it.

Most organizations aren't comfortable saying "this is all we'll ever need" and are worried about both building, buying, and then migrating, which can be significantly more expensive than either of those options in a vacuum.

Aeolun• 6 years ago

In my experience, integrating a thing like feature flags is a major pain if you do not have control over the implementation. It ends up being kludges and suboptimal choices everywhere.

fuzzythinker• 6 years ago

Agree with @Aeolun's comment. Feature Flagging is one exception in buy vs. build where there just isn't any good SaaS out there that makes it "just buy it" worth it. More so if you need customizations beyond what is offered.

crocktucker• 6 years ago

s/binary/simple/ ... and you're on to something.

jiveturkey• 6 years ago

cost is often not clear up front, but neither is benefit!

this is where experience matters.

api• 6 years ago

This is usually true except for two cases:

1 There is some other expense you can save significantly on by DIY. An example would be DIY k8s on bare metal via bare metal providers for a massively bandwidth intensive service where paying AWS or GCP outbound data rates would be thousands and thousands a month.

2 You are at sufficient scale that there is an economy of scale that can be accessed. This overlaps a bit with the first point but can also happen if you are just big enough. The first point is more about some special need.

x87678r• 6 years ago

This is where free & opensource has really changed the world. Its not buy vs build any more its download vs buy vs build and its best to use a free package that is out there. Its incredible what is available now, in the future I wonder if anyone will need to build anything.

mirekrusin• 6 years ago

Nonsense without a context.

Sometimes "building" means writing 6 shell scripts (each <10 LoC) and "buying" means getting CKA/CKAD certificates for all people involved and spending half a year on training.

fuzzy2• 6 years ago

I'm a little confused. When I have to build software (for a client), it's usually to support some business process or make it automated or whatever.

What am I supposed to buy, except some PDF/Excel/whatever library every now and then?

But maybe I'm just not the target audience for this article.

satyrnein2• 6 years ago

Yeah, doing work to automate some idiosyncratic process, either for clients or internal users, may be different. You do have to be careful you're not slowly implementing a CMS, CRM, email marketing tool, etc, feature by feature, though.

z3t4• 6 years ago

I think there is a balance in the "Yak Shaving" business... There is a swaying "the best code is the code you did not write".

leowoo91• 6 years ago

I have to disagree.. Not everyone has enough time to learn "ready" systems either, they can get pretty overwhelming. When I decide to build instead of buy, it's mostly because I don't believe in the design of the product that is favored by thousand people with reason being "it just sounds cool".

lexicality6• 6 years ago

This works great until you hire a second person who now has to figure out how your system works from the documentation you wrote, rather than the documentation and blogposts on the internet written by the thousands of people who thought the other thing was cool.

human2• 6 years ago

From an engineer perspective there is a learning opportunity that can’t be underestimated. If you have free time to build it’s always interesting how much you can grow by doing it yourself. However, if you are working for a startup and time to market is essential, than buy it for sure.

recursivedoubts2• 6 years ago

Every build v. buy decision should ask: "Is this a place where we want to bet on our ability to derive competitive advantage."

If the answer is no, lean towards buy. If the answer is yes, lean towards build.

For ops, netflix would say "yes" whereas most companies would say no.

magicalhippo• 6 years ago

It's all trade-offs. At work a customer could only send some data via mail, our integration could only accept via SFTP.

I had just had a look at Azure and had seen their Logic Apps, and quickly whipped up one which downloaded the mail and uploaded the files to SFTP. Nice.

Until a few months ago when it suddenly stopped working because for some reason Azure doesn't like the key exchange algorithms Bitvise SSH server provides. Or at least, that's as far as I've been able to determine. Zero logging on the Azure side so no clue what's wrong.

After spending hours trying to figure out what went wrong, I made my own program that does the same. It took a bit longer than the Logic App, but at least I can fix it when it breaks...

papito2• 6 years ago

The most egregious cases of violating this involve trying to create your own CMSs, search engines, and God forbid - databases. There are of course, exceptions. AirTable wrote their OWN database engine, but they knew what they were getting into, and they did it right.

hansvm• 6 years ago

I can't speak to CMS, but are search engines and databases really such a big deal? If postgres or influx or something kinda fits your workload then it's probably not worth the time, but on the other hand I've seen both custom databases and the cludges people have written to work around off-the-shelf DBs, and I vastly prefer the former both in terms of performance and maintainability in a problem space (similarly with search engines).

papito2• 6 years ago

The problem with search engines is that people think it's just tokenizing the search query and then doing an AND query for the words, but then the fun starts. First of all, there is specific search theory that one has to dive into and have the Knuth tomes nearby. Then, different languages, thesaurus, taxonomies, "did you mean". And we are not even at indexing speed, replication, performance, etc. There is absolutely no chance you would be able to create and maintain a reasonably good search engine part time.

rq1• 6 years ago

Sorry but no.

It’s cost dependent and you should carefully study both options.

It’s part of the engineering: study fixed and variable costs.

Outsourcing is cost effective when the market is mature enough.

You should maybe not rebuild cloudflare’s core services to operate a website.

Depending on your scale, you certainly should build your own ML stack for instance.

mlthoughts2018• 6 years ago

I think all four points in the article are deeply wrong.

1. It is easy to run your own services. You should be investing in internal developer tooling that makes this easy, and in fact that developer tooling should sit in front of any vendor solution you ever buy so that the way it integrates with your alerting, monitoring, data exporting, etc., is completely standardized to be uniform with service delivery in any other system in the org.

2. and 3. You do need complete control over what the application does because it absolutely always is unique and special on a per-use-case basis every time. A good example is search. Anyone who thinks search is a commodity service you can just throw ElasticSearch or Algolia in front of is sorely mistaken and dangerously naive. Every different search use case is going to have different success criteria, different data privacy concerns, different timeliness and freshness concerns, etc. and you need business software to control these elements in ways that fit into standard internal product management and QA procedures.

4. Vendor lock-in is a critical problem. If you choose GCP vs AWS, you are defining culture and you are defining experimentation and exploration that you cannot do. You’re essentially cleaving away many future possibilities from even being testable. It’s much worse than just having a crufty old system to maintain, it’s about brittleness and lack of ability to appropriately empower engineers to consider whatever part of the solution space they decide is needed. Companies that “get it” will prioritize “ease of swapping” so that you can constantly improve and leverage autonomy without needless parochial constraints on what can be considered. Thinking, “yeah but just buying it solved our problem today” is such a death knell of weak leadership who cannot fathom strategy or how to leverage real solution ideation from their staff.

lta• 6 years ago

Thank you for your answer. I was desperately scrolling in search of one like this to support.

I'm especially angry at the dismissal of the vendor lock-in problem in the main article. I've seen quite a few start-ups being trapped by a vendors which were very cheap at first (trial period, starter plans, etc.) and ended up eating much of the profits. Not to mention the innovation cost...

libria• 6 years ago

The desire to build custom versions of everything seems to come from a few places: 1 [...] 2 [...] 3 [...] 4

There are personal reasons as well:

5. Innate curiosity and excitement about technology

6a. Get experience

6b. Resume talking point

Many of us picked a career in IT because just like building things out of Legos is fun, building a streamlined full CI/CD pipeline is fun, building a full stack application is fun, etc. Speaking of career, one needs to acquire experience in the new technologies to pad their resume and it's convenient to do it on company time.

I'm not justifying putting your interests ahead of your company's, but understand that's what some of us do. I'd say #5 and #6 are often stronger drivers for decisions than #1-4.

mchusma• 6 years ago

The answer here is not so simple as "always buy". It depends heavily on context.

I think there are some things that do make little sense for anyone to build unless it's super core. These tend to be low level, cheap developer tools that do a couple of things well.

Trying to "buy and then integrate" larger pieces of software is often more work. Lots of people "buy" because they don't understand the problem. It's a mental opting out. This leads to disjointed experiences internally and externally.

But it all depends on your situation.

monadic3• 6 years ago

This doesn't account for vendor incompetence at basic customer support, which is far and away the largest reason to avoid renting commodity services. Being able to debug and fix things on demand without waiting on another party is massively valuable.

jvanderbot• 6 years ago

Build the secret sauce, buy the packets.

philjohn• 6 years ago

Also worth noting that "buy" in this context can also mean using a piece of open-source software, not just buying something commercial.

coward86753092• 6 years ago

The way I put this to clients is that you should own the bricks that your product is made of and not worry so much about the mortar. The bricks are the SQL queries, the logic inside the container images you run, the data stored in the tables and storage buckets. Oh yes, and the business relationships and brand equity.

As long as you're wiring those things together in ways you can replicate across cloud providers or in your own datacenter or on your own server in the closet, you've achieved the sweet spot in terms of deployment portability.

Running your own (Kubernetes cluster|database instance|pubsub system) on a cloud provider is crazy given that you now have at lease $150K+/yr piece of overhead in the form of at least one (dev)ops person. Save that expense for when or if you decide your needs have become predictable enough that you can start building your own software/hardware/network infrastructure.

Unless advanced devops fu is an important part of your company's mission or valuation, don't do it unless its saving you a lot of money over the alternatives.

jtsiskin• 6 years ago

I thought one of the main advantages of “running your own (Kubernetes cluster|database instance|pubsub system)” was for exactly that reason - so that “you can replicate across cloud providers or in your own datacenter or on your own server in the closet”.

coward86753092• 6 years ago

I don't pack a desk in my messenger bag: a laptop, pencil, and notebook suffice.

woleium• 6 years ago

A lot of the maintenance issues change with low code solutions, as do a bunch of other assumptions made in the article. We will be seeing a lot more low code developed without rigor by end user teams in coming years.

drej• 6 years ago

I used to have a boss who took this logic to an extreme and basically preached “always buy, never build”

Problems mounted - cost, security, integration issues, performance, ... - we couldn’t do anything about any of that, because we just wrote glue code to combine all the SaaS stuff.

I think about that job from time to time and I have two theories as to why this guy was like this - a) lack of trust/knowledge his engineers can build stuff, b) bragging about using a shiny-new-tech is better than talking about writing a Python script that saves data to Postgres.

All in all - good points in the article, but there is more to the decision (as hinted in the summary), building stuff is not just about scratching one’s itch.

veesahni• 6 years ago

For Enchant [0], we followed the "buy don't build" philosophy a lot in the early years. However, the biggest challenge has been reliability.

For example, a service provider will have a 5H outage without any clear indication about what's going on or when it will get resolved. This leaves us in a difficult position to raise an incident with an unknown ETA where we can do nothing but wait. Then when it's all resolved, we either don't get a post-mortem or a hand-wavy one.

Or a service provider will lose data, which may have been minor to them but critical to our operations. And all we get from them is an "oopsie!"

So over time, we've ended up insourcing mission critical pieces (within reason) since we can engineer solutions with guarantees that are inline with customer expectations and our own targets.

Separately, in some cases, buying and integrating turned out to be a ton more work than what building in house would have been. Because when we build in house, it can be built to meet our full requirements in the first place.. whereas when buying, we may find ourselves working around limitations in the APIs provided to us.

0: https://www.enchant.com - shared inboxes, knowledge bases, live chat

rachelbythebay• 6 years ago

It’s not buy vs. build. It’s _rent_ vs. build.

Unless you are actually purchasing whatever it is you’re using... and I’m guessing they aren’t doing that.

lambdatronics• 6 years ago

That's really incisive. Just as with rental housing, the responsibility for the maintenance gets shifted, not just the responsibility for building. It seems to me like software maintenance has gotten more and more important relative to software building over time... but it might just be an optical illusion. (I mean, Windows 3.1 didn't update itself overnight...) I wonder if this is true, and if so, what's driving it?

paragraft• 6 years ago

Yes, and you only organisationally learn this distinction really once you've been say, bitten hard by a contract renewal that completely up-ends your architecture, requiring huge efforts to stay in compliance with the new terms because by this point you're well and truly locked into the product.

taylodl2• 6 years ago

Systems of Differentiation should employ a build-first strategy. These are the systems that differentiate you from your competitors, they are your competitive edge. As such you typically need tight control over these systems and don't want to rely on 3rd parties - either vendors or contractors.

Systems of Engagement & Systems of Record should employ a buy-first strategy. These are typically more operations-focused systems and as such you're usually better off buying them, but not always (see below). It's possible that some of these systems may also be Systems of Differentiation.

What is a buy-first strategy? It means consider buying first. How much would your operations be impacted by the necessary changes to your workflow? How much customization will be required in order for it to be usable and how are those customizations maintained throughout product upgrades? How much integration is required with existing systems? Buying software is rarely a buy, deploy, and done proposition! Usually the TCO is lower if you buy, but not always - depending on how much customization and integration is required.

bdcravens3• 6 years ago

The phrase AWS came up with for this, "Undifferentiated Heavy Lifting", is quite apt.

taylodl2• 6 years ago

I should have pointed out this is the strategy that should be employed by an established company. If your company is just starting out then buy everything if possible - your competitive edge is the fact you're small and nimble. You can respond to market changes faster than the established companies can. Over time you'll recognize what are to become your Systems of Differentiation and then and only then should you consider building that yourself. When you're starting out you typically don't have the time or capital to wait and build the software you may need. For such companies doing business and generating revenue now is more important than having a perfect application portfolio. And yes, this is why systems are never as perfect and pristine as developers would hope.

doonesbury4• 6 years ago

Like the post, and use of definitional concepts. Useful. Thank you.

bdcravens3• 6 years ago

The hardest part of learning to not look through your developer eyes. Often the 80% solution is "good enough" for your customers or your stake holders, and they'd never appreciate the extra 20% a fully custom solution provides.

CapitalistCartr• 6 years ago

Building anything is step 5 of a good process, not step one. Whether you buy or build is far secondary.

First, check the motivations. Emotional or logical. If the driving motive(s) is ego, that's already a big, red flag. Consider running like a bull in reverse.

Second,intel. Gather information about the situation; educate yourself. Find the apples on the road.

Third, formulate a vision, an image of the final result that all the stakeholders (those whom you cannot ignore, no matter how hard you try) agree on. This is where beautiful diagrams are created. Which don't matter. All that matters is each stakeholder signs & dates it.

Fourth, plan. Use your knowlege gained in step two to formulate a plan for getting from the current situation to the "vision".

Fifth, execute said plan. Find more road apples.

Sixth, debrief. Analyze the outcome. Get a drink with the ones who did the work.

This can all boil down to just you spending 2-3 days running around doing it yoursels, or multiple teams spending most of a year herding cats. The important part is knowing the steps, and mentally looking for them. And for those who push to skip.

burnte• 6 years ago

Meh. In 2007 I wrote a webapp for a friend to help him manage his business. It's in PHP and other than a few fixes here and there, it's running just fine in 2020.

bird_monster• 6 years ago

I work at a small startup. My boss comes from only enterprise level companies. My boss believes literally everything we touch has to be custom, and we have to make it. It's an incredibly exhausting process of getting in front of and/or retroactively nullifying any planning the company does based on his suggestions, because most of the time they're quite harmful to our startup's tech. I've gone to our CEO about it and he agreed that we have to curtail this behavior whenever we see it. It's just an extra layer of "stuff" I have to think about.

No, we don't need k8s, Kafka, a home rolled CDN, custom authentication, etc.

zapataband12• 6 years ago

There are some good points in there, but it's worth noting build/deploying is becoming easier than ever. For a simple static site you could pay 300$ a website-builder-thing or you can build an app inside a docker container and your host will reboot it for you if it crashes. I use a docker-compose.yml file and I was able to quickly download an open source service that automatically renews my SSL cert. But yeah it depends on the scope, and the project, and what skills you have freely available or wouldn't mind learning.

zapataband12• 6 years ago

good real-life example i just ran into: i host my own website, and now I want to run a newsletter/email campaign. I don't desire doing this work and it's cheeeaapp/free from certain services that guarantee high delivery.

temporallobe• 6 years ago

This is not a universal truth. I’ve worked with COTS products that required a ton of work to customize and get working properly - sometimes costing as much in time and manpower as it would have been to build the same thing from scratch. Often this happens with managed server all-in-one “business solutions” systems that let non-developers build UIs and reports (and charts/visualizations, etc.). Nearly 100% of the time this is inadequate and often completely fails, requiring management to bring in actual software engineers to “fix” things and eventually just rebuild an entire dashboard from scratch using real development tools and techniques. Many of these kinds of shops underestimate the importance of strong software engineering practices and fail to understand the complexity of their own requirements, much less the complexity of their data.

tus88• 6 years ago

It’s usually a major mistake, that ends up costing a ton of time and money

LOL this is hilarious. Because outsourcing to consulting firms (cough IBM cough) has never resulted in blow-outs and poor delivery.

Buying almost always means being trapped in a snake-oil contract that requires ongoing consulting costs that quickly outweight the costs of the actual software.

This can be avoided by making sure that the key differentiators for your business are in-house.

Oh....so you are saying we should build now?

Blogpsam.

blindm2• 6 years ago

Some people take lifetimes, even multiple reincarnations to come to the conclusion that they are buyers, not sellers. Some people are born marketers and are very good at persuasion, hyperbole, and other tools to sell a product/service.

Others are natural consumers: they develop tastes and grow to be rabid shoppers. It's good to sit in the middle of these two opposites, but often better to decide which extreme you want.

I know I've made the decision too late in life, when I wasted a vast portion of it trying to sell some product/service/idea when really I just wanted the latest Porsche!

dasil003• 6 years ago

I'm sure a lot of people would love to go all-in on the buy side. Where do you get the money to buy something if you don't sell anything though?

blindm2• 6 years ago

Where do you get the money

In this case we assume there is a cashflow of some sort. How the cashflow comes into being is a public matter for most, but sometimes a private matter. It is assumed that a certain amount of disposable income can be used to acquire tastes and become a 'shopaholic' of sorts.

einpoklum• 6 years ago

This is Capitalist logic, and more specifically - an argument for the concentration of capital in the computing/IT sector.

We, software developers and users - people and small organizations - should adopt the opposite slogan:

Built, Don't Buy.

When we build, we improve our knowledge and understanding of systems. We help affect them, through engagement with their developers or through modification and forking. When the challenges and overhead of building exceed our abilities - this will drive us to:

1. Strive for better-fitting software - easier to use, deploy, build and maintain; and

2. Cooperate and collaborate with related organizations and communities/groups of developers and enthusiasts, either to share knowledge on how to get things done more easily, or to distribute the load of work between many parties, each of which can't do it on their own.

Even if a lot of FOSS contribution these days are made at for-profit corporations - the above is key to the future of both software freedom and social freedom.

doonesbury4• 6 years ago

I like this comment a lot. It's new perspective here. Of course capitalism is great. The US is only 25% of China's population and still we produce well. China copied of lot of capitalism. Silicon valley is hard to replicate in other countries. But we Americans (I'm one at an American company) do have a penchant for confusing R&D and longer term investment distorting it through the lens of a short term bean counting dollar mentality. Usually in the medium to longer term is the loss of customers and market credibility by anemic products of crappy quality. And as far as that goes hasn't this been talked into the ground through the 1990s under the broad topics of SPC and TQM? I could mention pithy quotes from Iacocca, Ishikawa, Deming, Drucker and so on but I wont. To compete world wide usually requires deep knowledge in teams, organizations that lead to client centered services that require mature processes and engineering talent to build. You cannot buy that. That deep knowledge helps demarcate build vs buy.

Let me mention one story here to make the point in the small. During the 80s/90s HP did selectively less OEM work. They bought Fujitsu wave soldering and pick n place machines. That is something like 1million per line. That kind of equipment comes with a serious stack of user manuals. HP threw those out and provided their own set with training. Why? Any fool can plunk down money. Smart OEMs needed a way to get their line engineers to "know how the line thinks" and to move jobs in and out of the line smoothly in an integrated process with attention to spc, quality and metrics tracking. Yes they bought but not by going ignorant.

je42• 6 years ago

You can only buy stuff if you know that stuff is aligned with what you need.

Further, you need to consider that the stuff you buy also needs to aligned with your needs over time.

However, you also need to consider this when building stuff.

The alignment is the tricky thing it is easier to achieve when building in-house as long as you engineering is up to the job.

Also, note that building in-house doesn't mean engineering "from scratch". Usually it means use lower level components over higher level components/services.

Silhouette• 6 years ago

The longer I run small businesses, the more sceptical I become of this sentiment. I wrote a cathartic rant here at first, about all the different ways external services have let down my businesses over a period of many years and how often our in-house systems have been the only thing that kept us going in those situations. But instead of letting rip and naming names in rather undignified fashion, I have decided to settle for just posting the conclusion.

I want to agree with the principle that you build what is essential to your business and you buy in the rest. As both an entrepreneur and a developer, I want to focus on the unique, value-adding parts of whatever my business is doing. I want to let someone else handle the mechanical stuff and the formalities, and I have no problem with paying a fair price for that.

But that whole argument is predicated on the idea that outsourcing will get an overall better result than building in-house, so that in some way it saves time and/or money that you can better invest elsewhere. So many services we've used have changed or discontinued functionality and forced us to relearn or reintegrate just to avoid going backwards, so many services we've used have pushed their prices up in real terms while adding little of extra value to us, so many services only ever reach 75% of what we'd like from them and the missing 25% of functionality hurts more and more, so many services we've used just don't work so often that you wonder how they manage to stay in business at all, that I now find it hard to trust or expect good results from any external service at all.

Sometimes you have no choice about outsourcing. No small business has the resources to do some things, particularly in heavily regulated areas like payments and tax, or in large-scale infrastructure like operating a data centre, unless that is part of the purpose of the business. So you choose the least of evils and hope for the best.

But today, as someone just starting to set up another business, I am looking for services that handle everything in a particular area where for practical reasons we can't do it in-house. I want to specify what we need, and have it done and the results delivered. I don't want to care about anything in between. I don't want to be distracted by any related legal and regulatory compliance matters. I just want the job done, properly and legally, for a reasonable price. I have little interest in working with anyone offering less than that, unless I have absolutely no choice.

Maybe this means I actually do agree with the original principle after all, and I've just learned from experience that an extreme interpretation of it is the way to go.

zmmmmm• 6 years ago

It's interesting that the whole article is pitched around a context where there is a basic assumption that there is a whole engineering team, devops etc., and the question is about how their time is best utilized. But if you chose to buy instead of build everything most of that would not exist.

So there's a paradox I see which is akin to the credit system where only people who don't need credit are offered it: you can only "afford" to buy instead of build when you have the engineering competence to build - that's when you can intelligently choose not to.

On the other hand, if you lack the internal competence to build and for that reason choose to buy? That's when all the bad things happen. You're going to get screwed by your vendor - they are going to know you aren't technically capable of supporting yourself, they will give you stupid timelines, blown out costs, unreasonable constraints ("it has to have a 16 core, 256GB RAM server or we won't support it") etc etc. You will end up with the worst case of vendor lockin and a system everybody hates.

So one of the pitches I make when people are making this decision is that you should be building at least some things, because the strategic cost of not doing that affects everything else you do.

bobbydreamer• 6 years ago

This blog has only one article

simonw5• 6 years ago

One thing to pay close attention to when making a build v.s. buy decision is the impact that billing models will have on your usage of a tool.

Take logging for example. If you buy a log aggregation platform like Splunk Cloud or Loggly the pricing is likely based on the quantity of data you ingest per day.

This can set up a weird incentive. If you are already close to the limit of your plan, you'll find that engineers are discouraged from logging new things.

This can have a subtle effect on your culture. Engineers who don't want to get into a budgeting conversation will end up avoiding using key tools, and this can cost you a lot of money in terms of invisible lost productivity.

Tools that charge per-head have a similar problem: if your analytics tool charges per head, your junior engineers won't have access to this. This means you won't build a culture where engineers use analytics to help make decisions.

This is a very tricky dynamic. On the one hand it's clearly completely crazy to invest in building your own logging or analytics solutions - you should be spending engineering effort solving the problems that are unique to your company!

But on the other hand, there are significant, hard-to-measure hidden costs of vendors with billing mechanisms that affect your culture in negative ways.

I don't have a solution to this. It's just something I've encountered that makes the "build v.s. buy" decision a lot more subtle than it can first appear.

telesilla• 6 years ago

We've often gone down the route of 'modify an open source project' for this reason. For the price of a developer, we get the product in perpetuity and don't have these exact limitations you specified of headcount or worrying about usage. For almost everything out there, you can find a version of it that's been made available with source code. Granted, unless you are essentially a software company this is probably difficult to maintain, given the additional level of expertise required.

Another option we've found works is buy, to just get started, and invest moving forward in our own project. Getting something off the ground quickly tends to help us understand better the requirements.

WrtCdEvrydy• 6 years ago

Honestly, I'm fan of the open source self hosted (especially if only some of the enterprise features are paid). We set it up, configure it, evaluate it and buy the enterprise license (it's usually a per-box license).

tixocloud2• 6 years ago

Very interesting perspective. Ran into the same issues where Salesforce was charging us by the number of columns we had that ended up forcing the team to overload a column with a JSON object so we could get it to work without overpaying. Integration and maintenance turned out it be a downright nightmare. Would be interested to hear various pricing models to understand the pros/cons for each especially in the logging/analytics space. Does per storage, per event, per user make sense in certain cases?

nitrogen• 6 years ago

For things that are directly generating revenue, it makes sense to have per-X pricing because it aligns your vendors' interests with your own. In that case, you want to incentivize the vendor to maximize your usage, because that usage is making you money.

beckingz2• 6 years ago

I have lived through this same thing with Salesforce.

A brutal experience.

supernovae• 6 years ago

I hate to break it to ya, but if you do cheats like this when you use reliable off the shelf COTS solutions, imagine the shortcuts they're going to do if/when they try it do this themselves.

Salesforce isn't cheap, but its not exactly expensive either. However, with this short sighted shortcut you did, you certainly cost your company big. That data is useless. If you upgrade, transition, migrate you now have to spend employee time fixing what was a fixed cost.

Human time is expensive. But what is more expensive is putting artificial costs where they don't need to be.

gshulegaard3• 6 years ago

I love this comment because it articulates a dynamic that I think is often overlooked much better than I could. I might even go a step farther and suggest that anything that is core to your business flywheel should be built and not bought (in general).

Mostly because you don't want pricing to act as a disincentive to exercising the flywheel.

pseudalopex• 6 years ago

Then time is the disincentive.

dmlittle3• 6 years ago

Even for built systems the cost of ownership and maintenance will grow as the system grows. Take the logging example above and say you're using the Elastic Stack. Things are going pretty well and usage is increasing. Over time your system grows and logging is now present in all of your applications. You've seen your logging rate go from 100 logs/s, to 1,000 logs/s, to 10,000 logs/s and now are encroaching the 100,000 logs/s rate. Over time the amount it cost to host your ES cluster has surely gone up. You probably went from a single node cluster when developing the solution, to a multi-node cluster, to now a hot-warm-cold cluster with several more nodes than you originally had. You also have a dedicated team making sure your in-house logging solution is working and stable. The cost of ownership and maintenance of this system is real. Not only do you need to pay for the huge clusters you'll likely have a dedicated team supporting it. Most people I've talked to in the industry don't like maintaining their in-house logging solutions and would gladly pay for one if it wasn't for the fact that it was too expensive to switch then. Because you need specialized people that know this logging solution it's also hard to hire for this role when the person currently maintaining this system eventually leaves (because he doesn't want to do this anymore). At the end of the day it's cheaper to pay for a logging solution even if it's in the hundreds of thousands of dollars a year than it is to build your own.

pembrook3• 6 years ago

“anything that is core to your business flywheel”

The mechanisms by which your product is marketed, sold and distributed are core to your business flywheel. For example, if you’re a B2B Saas, distribution is often way more important than the product itself.

So by the “core to our flywheel” standard you need to build your own email marketing software, your own version of salesforce CRM, your own analytics tool for triggering marketing funnels, etc.

That’s just hilarious. As someone who runs a business, I wouldn’t spend a single minute of engineering resources on anything that we don’t sell directly to a customer.

I’d go even further and say if you have less than 5000 employees, ALWAYS buy. Never build.

jokethrowaway2• 6 years ago

Memories of developers not logging stuff on datadog not to go over limit.

We ended up running our own ELK stack and it was much better.

KenanSulayman• 6 years ago

ELK is killing it. Especially with their apm server.

yagi_baba• 6 years ago

I've lived that life with Datadog. I was the dev who was the most into logging so I got voluntold to be in charge of the Datadog rollout. Told everyone it was not a good idea and presented cost estimates to why, we're better off just keeping our own logs and using one of many open source projects that let you pull in logs and analyze them. But all the cool kids were on Datadog, so onto it we went.

What killed us was the combo of the desire to really aggressive log on the staging-master and release-candidate environments (five envs total) and keep the logs for more than 7 days so the QA team could compare the logs from the current release candidate to the previous release candidates. Between the volume of having all the services set at the info level and the desire to keep it more than 7 days, Datadog was >30k a month.

Which at least finally got me permission to set up an open source log manager and switch everyone over to that. Once the initial panic over the Datadog bill died down.

dmlittle3• 6 years ago

I haven't used DD logs personally but from what I've been told if you send the majority of your logs straight to archive to an S3 bucket and only rehydrate the logs when you need to access them their pricing is pretty reasonable. Because you're storing the logs in your S3 bucket you can keep them however long you want and rehydrate them when you need them (say 30 days later).

The bill will still be in the low thousands but paying 30-40k per year for a logging solution is probably worth the cost if it means you don't have to maintain it. Having logs and metrics (and the APM if you're using it) all in one place is _really_ nice.

dosethree• 6 years ago

Whats the point of even having it in data dog? Typically we pay a ton for services like Newrelic for monitoring, the APM, and insights. Like you, we have tons of logs, so I wouldn't want to pay Newrelic or Newrelic-similar prices for a service you can just use ELK for yourself much cheaper. So basically I don't really see why you signed on to data dog at all

mch82• 6 years ago

So true! If the tool can’t be used by everyone on your team, then it is unlikely to be used.

I’ve experienced this many times.

novok• 6 years ago

You can still run into that budgeting conversation when you build, but this time it's with an internal team.

pembrook3• 6 years ago

I don't have a solution to this.

You basically just articulated the solution. Spend the money to give more of your team access to the analytics tool, using the argument made above.

It will still be infinitely cheaper (by like 100X) than building anything.

And the more you spend with a vendor, the more they’ll be willing to completely customize their product for you.

simonw5• 6 years ago

That's an easy answer if you control the budget.

If you're an engineer in a larger software organization, you probably don't have that spending authority. Which means you have to spend time convincing the people with that authority to spend that money.

That's enough of a friction point that many engineers won't bother - which is why I talk about the invisible cultural damage that this problem causes.

You and I know that it will be cheaper. The challenge is getting organizational buy-in.

acdha• 6 years ago

I’ve seen this repeatedly over my career. At the large organizations I’ve worked at (.edu, .gov), unless you have a large general tools budget it’s surprisingly hard to beat the open-source tools - simply running a procurement takes months and might require arguments with non-technical staff who don’t understand why certain options are not truly equivalent even if the salesperson swears they are. This is how you end up spending 7+ figures on enterprise tools while the technical staff are all using Prometheus, Grafana, and ELK because they actually work and are available to the entire group.

On the “clearly crazy to invest in building your own” front I think it’s useful to add the nuance about how much of that time is purely related to the product and how much is the tuning, integration, understanding the data, etc. which every product requires. I think we’re prone to underestimate the latter and have something like Splunk save less time and cost more than expected because the commodity part it optimized for wasn’t as much of the whole as anticipated.

simonw5• 6 years ago

I posted an expanded version of this comment on my blog, with a suggestion that maybe there are some internal process improvements that could help address the problem: https://simonwillison.net/2020/Dec/13/build-vs-buy/

"Thinking about this more, I realize that this isn’t a technology problem: it’s a process and culture problem. So there should be a process and cultural solution.

One thing that might work would be to explicitly consider this issue in the vendor selection conversations, then document it once the new tool has been implemented.

A company-wide document listing these tools, with clear guidance as to when it’s appropriate to increase capacity/spend and a documented owner (individual or team) plus contact details could really help overcome the standard engineer’s resistance to having conversations about budget."

Viliam1234• 6 years ago

Once we moved a database to cloud, because we were told it would be cheaper, since we would only pay for the space we actually used.

It also meant that whenever the data size increased, however slightly, the invoice also increased, and managers started investigating whether this increase of data size was truly necessary. Sometimes even when then data size didn't change, managers kept asking whether it could be further decreased.

Making the billing proportional to X means that X will become a political topic. Choose your X carefully!

simonw5• 6 years ago

"Making the billing proportional to X means that X will become a political topic. Choose your X carefully!"

That's such a great way to put it.

giovannibonetti• 6 years ago

This can have a subtle effect on your culture. Engineers who don't want to get into a budgeting conversation will end up avoiding using key tools, and this can cost you a lot of money in terms of invisible lost productivity.

For exactly this reason I wanted to build a developer screening tool from scratch [1]. In all the companies I worked for, I never got the budget to pay for one of the available tools in the market (most have "contact us pricing"). The result is that most teams I've met spend a lot of time manually evaluating coding challenges. This is a waste of time both for the company and the candidate. A custom screening tool allows the company to calibrate it according to the perceived average candidate level, and avoid those algorithm questions that are useless for most small to medium companies.

In my previous company it was used in a few hiring processes and it worked fine - both for back-end and front-end development. It is not open source yet because your need some basic Docker and Rails skills to use it - even though the screening test can be in any language(s) you want (that runs on Docker). If it matches your experience and you want to try it out, please contact me.

[1] https://github.com/skilltest/skilltest

figers• 6 years ago

A huge customer went with our custom build software because we controlled everything, all built in house, we could integrate with them exactly how they wanted vs a competitor who bought their technology platform and couldn't control the changes asked for...

doonesbury4• 6 years ago

Interesting read. Coincidentally this is a current conversation at our office (10K+ FTEs): use/extend open-source or develop institutional proficiency at aspects of distributed computing?

Ours is the usual case: we have homegrown brokered message queues plus Kafka and 21 more. We have homegrown single-master replicated DB, but also use DB2, Oracle, Postgres, MySql. We have private in-core caches with speciality code to knit together data and also use Memcache, Redis. Some of that code is installed from DPKGs; some if it slightly edited code and installed as service so internal users aren't bothered with the icky-details.

We have the Jekyll & Hyde behavior mentioned in OP's write-up and from commenters: on stuff considered core to the company and clients for a sufficiently complex system, we're not going to rely on 3rd party trouble ticketing systems and support. It's in-house. But that resolve falls away in stages and never lasts. We also have people who strongly push the business practical side: we're not here to write Azure or Kafka. We're here to make business apps. Get focused, and get client focused. That comes with an interesting blend of boredom and brand awareness gone wrong: I looked around & I saw or heard people in Dept X are working on caching solutions so ... what'd be the point of you doing it?

Now, here's the interesting question: suppose we needed a distributed ledger (without blockchain) or we needed 2PC for some homegrown caching. Now what? Do we write that in-house as a reusable components? Dig it out of PG or MySql code if they have it and we know were to look? Suppose we make a business case to upgrade a cluster to kernel bypass NICs. Should we go into Redis and self-modify the code to support that I/O path? No? Frankly, what then is the purpose of a CS guy/gal with a MS in distributed computing otherwise?

My own perspective on this:

- The first 1/3rd of this back and forth is the proxy conversation that flies above reality which is more about risk management and perception of failure i.e. company reputation & gossip mongering.

- The second 1/3rd of this is what people talk about because we can't talk about the underlying CS concepts. Most company employees usually have no ability to formally describe, or otherwise even sketch 2PC giving pseudo-code for it on a whiteboard. Or take this another way: see if you can get your lead on your homegrown replicated DB to explain how it works and why it works like we'd see in a senior level CS class: clear, specific, and yet abstract enough not be talking about superfluous implementation details. Can the audience understand it?

- Insufficient grounding in customer needs so one can better determine what clients really need, and what's lacking in the current offerings. Failure here will send you on goose chases to no where.

Gedanken experiment: you and I start a new software company broadly in distributed ledgers, caches. We are able to get 25% of MS' distributed computing leads --- not all --- but none of these guys and gals are slouches. What now? Do we write it in house or expand and extend some IBM offering? The the difference is we have talent, and there's some reason to believe they can deal with a distributed system's worth of risk.

To end this with humor, here's a nice analogy I once heard: On a Saturday, mom and dad were sick of the noise of their three kids. And the house was a mess. So they send them outdoors to work their energy off. At sunset the house is clean, and they're sitting on the porch watching the sun go down over some wine. They see the kids coming back: they are filthy dirty. They see work: all 3 kids needs baths. The clothes have to be washed. And they'll probably destroy the bathrooms they just cleaned. So the father looks at the mother and says: As I see it, we could clean these kids up or make a new one. Not sure if making a new one is wisdom, but it's a hell of a lot more fun.

dmantis• 6 years ago

The very important point of today's world - data protection - is not covered at all.

For customers, it's better to give their data consciously to one provider within one chosen jurisdiction than moving personal data, documents, activity logs, etc within multiple 3rd parties, as well as to do due diligence on any web service they provide any data.

Please, respect your customers and don't share their activity when it's not necessary and cost just a few hundreds bucks per month.

Treating humans as "units" instead as people with rights is a big problem of IT businesses.

ummonk2• 6 years ago

The problem happens when Ops software tends to be GUI-based and overly complex, and you just want some programmatic / command line tool you can properly integrate into your software / processes.

It's often simpler to take open source software and adapt it to your needs than to buy commercial software.

hootbootscoot• 6 years ago

Mmmm probably depends upon what it is. Core competencies of your business? Probably not the best thing to use an external service for. (Assuming your core competency is not cloud computing & storage) Your core will need to constantly be extended, honed, refined, redefined, poked at, prodded etc...

say_it_as_it_is• 6 years ago

And then the day comes when you've been laid off and your lack of product development expertise keeps you from a world of opportunity.

XCSme• 6 years ago

I think the answer is somewhere in-between. I really like the idea of "paid open-source", where someone creates a software product and you pay a small license fee to self-host it. This way you still have full control over the software, very low costs but the product is still externally maintained so you don't have to worry about it breaking. The only thing you have to maintain are the servers, but with all the VPS and managed cloud platforms out there, running a server, backing it up or monitoring it is done in a few clicks.

The biggest selling points of such a distribution system should be:

1) It's REALLY easy to set-up, no technical knowledge required.

2) It doesn't require any type of maintenance (eg. can run perfectly at least one year without changing anything, this means that the software should allow for OTA updates).

3) It is a lot (10x) cheaper than a SaaS alternative.

jokethrowaway2• 6 years ago

This implies that employees actually have more valuable stuff to work on or that the leadership in the company know what's valuable, which is not always the case.

Given the limited number of projects creating values, it's often a choice of working on the next failed project or saving the company 100 grand.

dave_sid• 6 years ago

Nah

Ataraxy• 6 years ago

I tend to favor abstractions so that services can always remain agnostic. It's one thing to buy, but it's another matter to be locked into something. Especially if it's core to your business.

marsdepinski• 6 years ago

If you know what you're doing building and running yourself is usually a good path

mjgs• 6 years ago

Great article and some really interesting comments in this thread.

I wrote a piece just a few days ago talking about the same topic, although it was aimed at earlier stage developers, so there is less detail, but the overall message is similar:

Deciding when to build a custom solution in web development

https://blog.markjgsmith.com/2020/12/11/deciding-when-to-bui...

inshadows• 6 years ago

This is very often misinterpreted to mean that you should use exiting stuff even if it doesn't fit your purpose. Obviously there need to be products that fit your purpose BUILT by someone for you to buy one. Several times I've been in a position where people around me pushed me into using something that remotely looked like it solves my problem. Since I was the one doing actual analysis and work, I knew I'd spend a lot of time digging through internals of said product to bend it for the use case. They also claim that they don't believe that there's not some existing software that exactly solves my problem and I'm creating something just for the sake of it. Let's be honest, these people are full of crap thinking about software engineering only superfluously. There's no silver bullet.

talkingtab• 6 years ago

If you are talking core technology then I disagree. My example is Oauth. There are lots of libraries that make it easy to use Oauth, but then what you have learned is not Oauth, but some library and how to use it. Libraries can also add a layer of complexity because they are generalized. And finally, what if the service you buy and don't build needs to be modified for some use case that is particular to your application? While I agree that all of these points should be considered in the general case, core technology for your business presents a very different set of considerations.

ideamotor• 6 years ago

Talk Don’t Do

So much effort is spent on hypotheticals. If you think about a problem for too long, it will seem like a worse problem than it is.

watermelon592• 6 years ago

I hope this doesn't get buried, because there's something that I feel like I'm a bit of a unicorn about.

The problem then with that is everyone who is working on those services is usually trying to get off of them. After all, no one wants to work on something that their boss doesn’t care about.
Yes, your CI/CD system is absolutely critical, but it’s easy for executives to not think about. This leads to a failure mode where you have a lost garden of internal tools.

I love maintaining stuff. I don't care that my boss doesn't care about something. I care about... what I care about :)

I don't mind the "boring" tasks like keeping library versions up to date, being on the latest runtime version, and using a recent version of the framework we've adopted (we're currently running on an ancient version of it). I like doing that stuff. I like keeping things tidy. If things are up to date, I can move on to making our CI better, or improving our test coverage, or really anything that improves the whole team's productivity. There's always something to update or polish or improve.

My dilemma is that as much as I'm willing to do all that stuff, I'm essentially not allowed to. My lead and their boss say my skills are too valuable to be spent on that, so instead I must do things like lead the new team they're forming to build a new service... which I'm not interested in at all. I'm not keen on the whole lead thing. Glad to be a follower. I'm also not thrilled about building new stuff. I like to say I'm more like a car mechanic rather than an engineer: I like tweaking and tuning and fixing existing stuff, not creating new stuff.

Does anyone else feel like this? I've tried bringing it up in chats with coworkers and everyone looks at me like I have two heads.

presentation• 6 years ago

I more or less feel that way - I don’t have a lot of interest in churning out yet another feature in the never ending treadmill of feature development.

madrox2• 6 years ago

Consider that while you may care about maintaining something, whoever comes after you likely will not. Just like writing and documenting good code, solutions should last beyond a single engineer.

So yes, your skills are too valuable. All engineering skills are. That's the point of buy vs build.

If you truly feel you aren't being allowed to work on the the things you care about, it's possible you're simply in the wrong role or company.

bcbrown• 6 years ago

I feel similarly. I find the most satisfaction in maintenance of an existing system, as opposed to greenfield development. Finding the optimal way to add new features inside an existing system is deeply satisfying. Improving tooling and infrastructure that leads to quality-of-life improvements across the team is very fulfilling.

somuchdata• 6 years ago

I have also seen this from time to time. We have a few engineers in my org and another on my wife's team at another company that have interests similar to yours.

I think people with your interests are really valuable. We've changed around what we index on for performance evals/promotions and what work we assign them so that we can better accommodate these sorts of work preferences and skills. If someone is happier working on an essential but unsexy, unloved corner of your systems or infrastructure, and they're also many times more productive in that area than other engineers who have no interest in it, then you might as well take advantage of it!

The key factor is making sure that the work those people are doing is truly high impact (for example, CI improvement might reduce deployment failures, improve team deployment velocity) and not simply maintenance for sake of abstract cleanliness.

watermelon592• 6 years ago

We've changed around what we index on for performance evals/promotions and what work we assign them so that we can better accommodate these sorts of work preferences and skills.

It's really cool that you've that. And I'm glad to see I'm not alone in this.

Mandatum2• 6 years ago

Seems like your interests align to some SRE and DevOps roles, but you'd definitely need to shop around and it'd only work at large companies.

Renevith2• 6 years ago

I agree that you need to shop around; some places value this far more than others. But I don't think the company needs to be huge. We have small but dedicated teams for this on each platform (backend, iOS, Android, web) and we only have ~100 total engineers.

0xbadcafebee• 6 years ago

Reframe what you do as "upgrading legacy infrastructure" or "applying modern best practices to existing products", etc, and it makes more sense. Usually people don't need to build new things, they need to fix their old things. But you have to sell it the right way and throw some numbers at them so they realize, hey, this is actually a better idea (financially and speed-wise). Can't fall victim to sunk cost fallacy or end up with a bad system if you're starting from a working one.

You could probably make a career out of being "the optimization guy". Product isn't working well? Ask the optimization guy to take a look before somebody spends a million bucks trying to replace it.

Renevith2• 6 years ago

This is very much how I feel, which is why I've started a dedicated team for this at Wealthfront, with the support of our CTO, our VP of engineering, and our president (who used to be the CTO). My team is specific to the backend but we have equivalent "infrastructure" teams for the frontend platforms too.

If anyone feels like it's hard to get your company to care about this stuff or even let you work on it, maybe it's time to consider a move? :-) You can reach me at andyf@wealthfront.com if you prefer not to reply here.

rmason• 6 years ago

Like all things challenge absolutes. I agree you shouldn't always build, otherwise you'd build your own office suite!

But do you know how many times engineers started to build something and made a discovery that resulted in inventing something significant. Or created something that gave their company a significant advantage. A not small number of times.

nitwit005• 6 years ago

This misses the rather obvious issue of price. Building your own software is the cheaper option some of the time.

This is particularly true when you have to pay to add some missing feature you need. They will, naturally, try to ask for as much as possible to get what you want.

madrox2• 6 years ago

What this doesn't mean, though, is Buy Everything. Buying everything is also how you get poor quality and poor process. It's important to recognize what's core to your culture. It's why I prefer discussing this as a what is core vs commodity decision.

If there's a general rule for making these decisions, I haven't found one. What's helped me, though, is asking whether or not the problems my team has is any different from most other teams. Sometimes it's true, like when I was at an entertainment company working on asset management. Sometimes it isn't, like when I was at the same company figuring out an AB testing solution.

bern4444• 6 years ago

DSS?

trabant00• 6 years ago

There are several good comments in here explaining cases and reasons to build instead of buy. But they are all techical and miss one critical thing: human motivation.

Nobody cares about your bussiness as much as you. The motivation behind your product/service provider is to take your money. Buying will always be a battle with the other party to get value for your money. They are strongly motivated to offer as little work for as much money as they can get away with because there is where profit is made.

The provider might even be more competent than you in building. But think about hiring a good programmer that just doesn't want to work vs a mediocre one that is highly motivated. I think everybody has come across these examples and knows the outcomes.

doonesbury4• 6 years ago

Amen to that! Giving a damn is absolutely needed. See longer response elsewhere this comment section.

nine_k• 6 years ago

When you have more money than time, prefer buying. When you have more time than money, prefer rolling your own. It's a well-known principle.

For a VC-backed startup with millions galore and a pressure to beat competition, buying is natural.

For a bootstrapped company or a side project, building your own may be best, especially when the advanced features that make market-leading solutions more expensive does not add a lot of value at the company's current scale, and the growth is not exponential.

beshrkayali• 6 years ago

The article makes some good points, but the title is somewhat clickbaity and reductive. Blanket statements such as "Buy don't build" are unlikely to be a good rule-of-thumb to follow imho.

However, at the end, a somewhat more sensible advice is presented:

I suggest that you prioritize buying to building, unless building will provide a real sustainable advantage for the business.

I generally only buy things to save time when I really don't have it, and usually as a temporary solution. Buying isn't an all-in or all-out solution. There are many levels at which you can decide to either buy or maintain the thing yourself. In a lot of average/usual situations, reading how to properly setup the service rather than just buying it from x-as-a-service will provide valuable learning not just for the setup itself, but also for tooling used around it, which will make developers even happier and more productive. For example, I wouldn't buy Redis/Elasticsearch as a service if I can spare a couple of days to read how to set it up (and of course there are always exceptions). Care about security first, then efficiency. It will take time, but it's not as difficult as it's made to seem.

Quick example, a few years ago, I witnessed a company save more than 80% of their hosting cost by switching from Heroku to a simple EC2/Ansible setup. They're still buying, but there are levels. Also, it helps to know what are the actual requirements when you're buying.

Thinking that buying everything as-a-service will save time is a misconception imo, as you'll likely end up having to spend time reading the docs of whatever you're buying. Decision changes will also cost more time because unless that service was using an open standard, you'll likely end up having to change a few things.

There's great value in learning open standards and how to automate setups. With that being said, no one can learn everything at once, so if the need is urgent and you're short on time and have properly studied the financial model of the service, yes, buying is a good option.

Not related to this, but I've been having a lot of fun messing around with Nomnad. I've also learned some very interesting things about how docker networking work because of it.

PeterStuer• 6 years ago

'Buy' often is not available at the granular cost:benefit price-point that your specific application needs, especially in the startup and scale-up phase.

For the service provider there is a fairly fixed to linear base overhead so it does not make sense to make an special offering for the 1/10th or less of functionality you might require at 1:10th of the price. It doesn't make economic sense.

So for the small business it can mean they get away with a homegrown minimal service at a fraction of the cost, even though the 'quality' of the produced for general applications might also just be a fraction of integrating the 'off the shelf' offering.

I've seen some startups running in major deficits simply because they went all out on integrating external services the aggregated cost far outstripped the margin their market allowed for.

lumberjack242• 6 years ago

Another great read on the topic is from Calm's CTO - https://lethain.com/build-vs-buy

Some interesting takes

1) There's more to it than the 'risk' factor, consider value & costs!

2) Ask yourself "What can I actually build right now?" and be brutally honest before attempting any comparison (vs external tools)

3) Solutions that hinder your efforts to move any component outside their ecosystem are best avoided. These prove to be costly in the long run.

codebolt• 6 years ago

Completely disagree with this sentiment. I personally built a small program that scans our production environment for logs with errors and sends the team an email if something's up. Spent two days writing the initial version, have only made a handful of small extensions since then. Been using it for years now, with no complaints from anyone. But countless problematic situations that were resolved before they could cause major havoc.

Would have probably spent a lot more time wrestling with a more general solution to make it fit our particular environment.

Buy Don't Build

Related Stories