The resource leak bug of our civilization

http://countercomplex.blogspot.com/2014/08/the-resource-leak-bug-of-our.html

by userbinator • 12 years ago

120 88 12 years ago

countercomplex.blogspot.com

waps2• 12 years ago

The bug in this article. Economically speaking (in other words: in the real world, where you are not the only player), which is optimal :

1) optimizing resource usage

2) maximizing resource usage

(it's easy to see why some others are suboptimal, e.g. minimizing resource usage)

From a long term perspective (ie. which of these guarantees you'll survive and doesn't provide another player with an easy way to destroy you). Other way of putting this question, which is the Nash-equilibrium answer ?

The answer is : 2.

(Is it pareto-efficient ? yes !)

In other words, no matter how good it sounds to conserve resources, it's a mistake. If a civilization did that, either it would get run over by another civilization, or, if it came to that extreme, by evolution itself.

So this is not a bug, it's a feature.

tree_of_item• 12 years ago

Nash-equilibrium and pareto-efficiency are extremely technical terms, you shouldn't abuse them here to get some faux sense of authority. I say abuse because we aren't in a context well defined enough to give them an interesting meaning.

deciplex2• 12 years ago

Can you expand on this? It seems like you're saying that, for example, taking all the world's oil and lighting it on fire, or (more dramatically) firing all the world's resources out of a giant cannon and into the sun, is an optimal economic strategy. I'm not an economist, but that doesn't sound right.

It seems like the optimal economic strategy, assuming finite resources, is to optimize your use of resources. Then, there is a complementary defense strategy, which is to monitor other civilizations so that they don't get the drop on you. This might also include acquiring resource pools for future use, or simply to deny them to adversaries, but it doesn't mean you need to burn through them as quickly as you can.

mattdw3• 12 years ago

Maximising deprives others of the resource, optimising doesn't. I think that's the point being made.

An economy that first maximised, then second optimised, would beat an economy that first optimised, then second maximised, because the latter would have nowhere to expand.

An economy that optimised and maximised would beat an economy that only optimised.

clord• 12 years ago

You're assuming resources are liquid (meaning: able to be relocated for low cost). This is only true if you're trading, and civilizations trading energy can't really be separate civilizations.

The only civilization we're stealing resources from does not exist yet because it's in the future. We have one global civilization that is pooling a single resource base via highly liquid trade. There might be civil war between factions, but don't think a real resource war can occur under the current environment. Now, might we split into an 'eastern empire' and a 'western empire' like the Romans did? Quite Possible.

baq• 12 years ago

economy, when confronted with physics and mathematics, always loses. exponential growth is not sustainable. i recommend thinking how much energy we can release before the oceans boil off and how long will that take assuming 3% growth yoy. (hint: sooner than you might think.)

sergiosgc3• 12 years ago

It depends. Any complex question gets a complex answer. If we release the energy in the form of heat, and if we let that heat get trapped in our planet, then you'll boil off the oceans in a predictable horizon. However, those two ifs are very big ifs. Two examples:

If you start using energy to bond complex molecules, you are using energy, but the energy is getting trapped in chemical bonds. You can easily achieve 3% YoY energy use growth without any ocean boiling in the predictable time horizon.

If you use the energy to finally expand away from Earth, then clearly you can achieve 3% energy usage growth with no ill effects whatsoever.

In the end, the fact of the matter is that civilization evolution is very highly correlated to the ability to manipulate energy, and consequentially to the amount of energy used globally. Locking ourselves out of increasing energy usage is paramount to locking ourselves out of pie-in-the-sky projects such as planet terraforming.

boomlinde• 12 years ago

I'm glad you are sharing your conclusions, but your post seems completely devoid of an actual argument. Saying "Pareto" and "Nash equilibrium" does not count.

orbifold• 12 years ago

I think resource usage is also a matter of control. At least during the cold war large amounts of resources were in danger of vanishing behind the "iron curtain" forever. If socialism had successfully caught on in latin america, indochina and more of the gulf states had fallen under soviet influence, it would have been lights out for western economies, which largely rely on raw materials imported from those regions. To a smaller degree this is even still true today, with Europe dependant on Russian gas. In a market where everyone has equal and free access to natural resources 2) might be indeed the best option. Otherwise a strategy where you exploit the resources of other nations first to enrich yourself seems like a far better option. Of course this has happened to various degrees all over the world to the benefit of a few industrialised nations.

bonobo3000• 12 years ago

There is no mention of all the benefits that encapsulation gives us - to take the blogs argument to the extreme imagine writing your own web server, caching layer, load balancer, JSON/HTTP parser, frontend logic for every platform, and a hundred other layers of abstraction I may not even know about every time you want to write a blog.

There is a real tradeoff between programmer productivity and resource usage, because every library that isn't tailored for your exact use case (i.e every library) is an inefficiency. Given programmers wages vs. cost of CPU/RAM/storage, and the long term trends (technology makes one cost lower, the other cost higher!), it is economically insane to reject encapsulation.

I definitely see the appeal viewing programming as craft. In that case, its no coincidence that "reinventing wheels" is good advice to learn how to program. Its just not how to run a business.

mattdw3• 12 years ago

Perhaps, though, computing resources are priced artificially low due to externalities (e.g. third world wage inequality, environmental impact not paid for), and perhaps a sane long-term approach would find a different balance between programmer wages and physical tech components.

AnthonyMouse• 12 years ago

Moore's law eats the difference between first world prices and third world prices for breakfast. Changing how the hardware is manufactured wouldn't materially affect the choices programmers make even if it doubled the hardware cost, and it probably wouldn't even do that.

userbinator• 12 years ago

to take the blogs argument to the extreme imagine writing your own web server, caching layer, load balancer, JSON/HTTP parser, frontend logic for every platform, and a hundred other layers of abstraction I may not even know about every time you want to write a blog.

I see the argument as saying that all those layers are not really necessary. Take HTTP for example: it's a large complex protocol that requires plenty of code to handle correctly (e.g. the size of a request, and the various pieces inside it, are not specified and the parser has to read through the whole request sequentially to figure this out), and the majority of applications using it never need the full functionality. What if it was a simpler, easier-to-parse format that was as extensible?

One of the points I get from his argument is that encapsulation has encouraged us to build massively overcomplicated systems by piling layers upon layers with little understanding of how everything fits together, when we could instead be building much simpler and more efficient systems that could still be using abstraction but in a far more cautious manner.

crpatino4• 12 years ago

I think you are partially right, but the situation (I'd rather not label it as a problem just yet) goes on many levels.

By example, Moore Law has allowed us to pack incredibly large amounts to transistors in reasonably affordable machines, and what we do is... virtualization, a.k.a. write software that (presumably) simulates being a (much slower and limited) hardware system so a third party can run other programs on top of it, for a fee.

That kind of makes economic sense (because customers have a big appetite for computing power... just as long as they do not have to keep those pesky sysadmins on the payroll), but from a purely technological point of view is craziness. Every component you add into your stack is a potential source of error, and you do not gain any extra features because you are simulating real hardware... so, why to do it in the first place?

I think the answer is: because nobody would buy all those transistors if we didn't do so.

jerf• 12 years ago

"What if [HTTP] was a simpler, easier-to-parse format that was as extensible?"

Then it would become HTTP. It would take time but it would be inevitable.

HTTP didn't just happen. Using Fred Brooks' terminology, the real world is essentially complicated. We may pile accidental complexity on top of it at times, but the real world is essentially complicated.

While I agree that there is room for accidental complexity to occasionally be swept away and a new clean slate presented, I do not believe that a new protocol for HTTP could solve all the problems HTTP solves without being at least nearly as complex as HTTP. The set of problems that HTTP solves is a great deal larger than "request page -> get page". There's caching, proxies, security, encryption, policy enforcement, needs that schools have, needs that Facebook has... and that's still just the high-level table of contents of the table of contents of the world of HTTP. And if you think you can just ignore those cases and simplify... try it and you'll see how wrong you are.

Note I do think there is room for improvement in HTTP, and it probably isn't the direction of 2.0. But it's not like there's some sort of brilliantly simple protocol in there struggling to get out, buried under layers of cruft. It's a thorny hard problem and the result will be a thorny hard protocol. It often fails to work not because the protocol itself is broken but because people doing work in HTTP have truly failed to account for a fundamental use case properly, like caching or the use of proxies.

Also note that there's no guarantee that "simple protocol + time" will yield something any simpler than HTTP. Under HTTP is a fairly powerful idea, and there's a reason why so many Internet protocols look very similar to it. It would be easy to start with something "simpler" that turns out to grow much, much worse.

jdmichal• 12 years ago

http://xkcd.com/927/

Sure, you can create a simple transfer protocol that meets 80% of the use cases. Now add on the other 20% of the use cases, and you will end up with equatable complexity. And then, congratulations, you now have two competing standards.

meric3• 12 years ago

Every once in a while a company like Tesla comes along and reconstructs a particular product gaining incalculable advantages over its competitors.

Every once in a while a doctor is forced to re-break someone's arm which didn't recover properly.

Every once in a while severe economic recessions comes along and wipes out all the badly managed companies kept in life support by banks/government.

Every once in a while a programmer writes a piece of software that has almost the same functionality as an existing one but, is, some how, just better.

And every once in a while there are revolutions that completely destroys the existing system of governance, when it can no longer fix itself properly.

And that's when resources get deallocated.

Civilization will fall, and it will rise again from its ashes.

Looks like the world uses stop-the-world garbage collection on particular threads keeping the rest of the world running.

tantalor• 12 years ago

Reminds me of "crash-only thinking" http://www.ribbonfarm.com/2014/10/29/crash-only-thinking/

Most things work fine until they catastrophically fail and are rapidly replaced, hopefully with something better.

amirmc• 12 years ago

Every once in a while severe economic recessions comes along and wipes out all the badly managed companies kept in life support by banks/government.

Except we didn't allow this for the banks (iiuc), so not everything gets GCd.

meric3• 12 years ago

tantalor posted a good link - artificially preventing a complete crash would only cause the next crash to be bigger.

Central banks can prevent bank crashes - but only at the cost of partially compromising its own integrity (See central bank balance sheets and the amount of toxic assets they hold). Eventually a crash would become so big the central banking system itself would fail. One possible failure mode is hyperinflation.

Joeboy• 12 years ago

Civilization will fall, and it will rise again from its ashes.

Yes to the first bit. The second bit seems like an application of the Peter Principle.

brc3• 12 years ago

This happens at the micro level (one employee is swapped for another who is better) and at the macro level (banks/tesla) and at the country level (socialism is replaced by more efficient and humane capitalism as in ussr/GDR).

The problem is when the bad employee cannot be removed, the bank is bailed out or the people are unable to get themselves a new form of government.

The key to efficient reallocation is not letting too much power concentrate in one location. And I think that is the thing at risk the most right now. Governments have become too powerful and are preventing the necessary creative destruction, like an operating system refusing to run the garbage collector and preventing new programs from running.

FireRabbit• 12 years ago

The problem with the USSR wasn't its economic system, it was the fact it was a corrupt dictatorship.

aethertap2• 12 years ago

I think you nailed it.

This is a good example of the adaptive cycle that's used to explain behavior in complex adaptive systems of various kinds. The cycle goes through up to four phases: rapid growth, conservation, release, and reorganization.

Rapid growth happens when a relatively unexploited resource is abundant (usually after a release/reorganization) and there isn't much competition. This is like a bare field - weeds move in quickly to take up unused space.

Gradually, competition heats up as more things get in on the action. This is where optimization starts to become important, because the ones that do a better job with what's available start to come out on top, at the expense of the less-competitive ones. Late in the process, you're in the conservation phase, where it's hard to change the entrenched winners because they have all of the resources locked up (can be money, nutrients, or whatever). In the field analogy, the bigger weeds shade out the smaller ones, until eventually you have large trees taking up most of the available resources. This phase is the longest-lasting of the four in general.

The system at this point is highly optimized and dependent on the winners of the conservation phase. At some point, these few hub elements get hit by a disturbance that shakes them up a bit, and the repercussions spread throughout the whole system. If it's a big enough disturbance, the whole system can fall apart due to the loss of the hubs. This is the release (so-called because the stuff that was locked up by the entrenched players becomes available again). In the field example, this might be a blight that knocks out the climax tree species, opening tons of space for new competitors and releasing the accumulated biomass in the trees for reuse.

At this point, it's a free-for-all to see who can get a toehold first in the new rich environment, and a brief period of reorganization ensues while that's being sorted out. In the field example, this is where all of the dormant weed and tree seeds that have been waiting in the soil sprout at the same time. This quickly transitions into rapid growth again, and the whole thing repeats.

It's also an instance of self-organized criticality, where a system tends to evolve toward a critical state (i.e. local disturbances can have global implications).

There are ways to manage this cycle to some extent when you're aware of it - primary among them is to deliberately introduce disturbances to shake up the formation of the conservation phase and loosen the connections that have built up. Basically, you introduce more randomness into the works in order to keep things from becoming overly optimized (or highly correlated, in the language of self-organized criticality).

ethbro2• 12 years ago

"Basically, you introduce more randomness into the works in order to keep things from becoming overly optimized (or highly correlated, in the language of self-organized criticality)."

That sounds a lot like how I've heard AI guys talk about avoiding optimization pitfalls when training.

I guess the difference is we can run that on an accelerated timescale, whereas civilization marches to its own, slower beat.

sanxiyn• 12 years ago

Nobody knows how to make a pencil, and that is not a bug. Division of labour is how humanity improved its lot so far.

http://en.wikisource.org/wiki/I,_Pencil

alexmat• 12 years ago

"Improved" is a relative term and says nothing of the associated costs. Could we maybe reduce costs and still "improve" at a slower, more sustainable pace? It's worth exploring the notion without turning to the dogma of efficiency at any cost.

Retra4• 12 years ago

Exactly. "At any cost" should only apply to long-term full-scale survival threats. With every other problem, you should only ever except "at reasonable cost."

And in order to know what cost is reasonable, you have to do some kind of rational analysis. There are few things less efficient than an ideology without an understood purpose.

bildung• 12 years ago

You make a qualitative argument, while OP makes a quantitative argument. You rightly argue that division of labour boosted productivity tremendously, but that point doesn't affect OP's argumentation at all. Why not keep division of labour and try to make it less wasteful?

john_b• 12 years ago

Nothing in the OP's argument was quantitative. Both are qualitative arguments.

The OP raises some good points that deserve serious thought, but I find his argument sketchy and hand-wavy. The crux of the argument is this:

"Our mainstream economic system is oriented towards maximal production and growth. This effectively means that participants are forced to maximize their portions of the cake in order to stay in the game. It is therefore necessary to insert useless and even harmful "tumor material" in one's own economical portion in order to avoid losing one's position. This produces an ever-growing global parasite fungus that manifests as things like black boxes, planned obsolescence and artificial creation of needs."

Everything else in the post is either analogy or following the assumptions in that paragraph to their conclusions. But the sheer amount of reasonable objections that are ignored or dismissed in that paragraph is staggering.

If today's capitalist systems have systemic issues that result in gross misuse of resources, I don't think it's for any of the reasons that the OP suggested. For that reason, I am skeptical of his proposed solutions.

EGreg8• 12 years ago

And most likely exacerbated the negative externalities. Each pencil company wants to sell as many pencils as possible, just because. Most organizations try to grow not shrink.

Houshalter3• 12 years ago

How is that a bad thing? And how would it be improved by everyone making their own pencils?

akkartik• 12 years ago

Two links, one agreeing and one disagreeing:

http://slatestarcodex.com/2014/07/30/meditations-on-moloch

http://www.ribbonfarm.com/2012/08/23/waste-creativity-and-go...

zobzu• 12 years ago

tldr: civilization needs some goddamn refactoring.

jmnicolas2• 12 years ago

It is my opinion that unfortunately no amount of refactoring will save our civilization.

We need to restart it from scratch.

vetler• 12 years ago

Maybe we could start by doing a proof-of-concept before we rewrite the civilization?

bjz_• 12 years ago

If that is the case, then let's hope we don't suffer from second system syndrome...

sergiosgc3• 12 years ago

Young whipersnapers and their throw the old code away syndrome...

You don't restart from scratch and hope for the best. If the systemic conditions that make our civilization ill persist, they will affect version 2.0.

It is much much better to incrementally improve the current system, even if the end game is not clear yet. A full rewrite always throws out the good along with the bad, and it is clear our civilization has many good characteristics in it.

I, of course, disagree on the need to restart from scratch.I think that the resource problems we face are slow moving enough that we will avoid them. The real danger is in fast moving problems that we do not know about yet.

wlievens• 12 years ago

Parts of the post are interesting and challenging, and to some degree I agree that the cost of layering abstractions is often underestimated, but the final few paragraphs read like a naive manifesto...

gus_massa• 12 years ago

Going to the extreme, instead of using blogspot you can write your post in raw html and host them in your own server. But the blogspot abstraction layer is much easier (and it can handle the HN load), so you have more time to do your unique activities.

fit2rule2• 12 years ago

I maintain a small outfit of old, antiquated, 'un-useful' computers, the eldest 30+ years old. I have pretty much never thrown a computer out - there is always a use for it.

Computers do not die; their users do. ("s/Computers/ConsumerToy/g")&etc.

The leak can be plugged, and it starts like this: don't use technology you don't understand, or .. to put it likewise: use all technology with full understanding. The only reason we threw away our old computers was because we didn't think anyone could/should use them; they are nevertheless: still useful. The 'usefulness or not' is entirely a spiritual decision, however, because the machine does not care, and it is this fact - and ignorance of such spiritual components of so many facets of human culture in general - is the blind spot in the debate, imho. Things are only as they are, if we decide they are; the usefulness of a computing device is absolutely not dependent on its physics properties in any way other than: can you turn it on, does it work, can the user do something with it? It is the User who makes the decision, all along the way to actual, real, usefulness.

I'm still using my computer to send email, just like I did in the 80's and so on, albeit it now requires a much more sinister amount of maintenance, and general abstract stress it seems, emotionally, than the old hazeltine connected to a Vaxen we used to have to use to 'get things done'.

tl;dr: there is no fault in our machines but that which the user decides and identifies.

paulmd• 12 years ago

As a working definition, a piece of hardware really isn't useful if another piece of hardware will do the same task better, at a lower cost.

You can run a low-power PC and check your email like it's 1980 just fine, and these days they're VERY cheap. A $30 5W Raspberry Pi is more than capable of running 'mail' on the command line. For a bit more power, quad-core 1.6ghz ARM SOCs are about $60-70. For fileserver type stuff, a 15W Liva with a Baytrail-M can be had for $115, or you can build out a full quad-core mini-ITX for something like $175.

The reduced power usage very quickly amortizes the cost of the hardware. If you would save 200W by retiring a dinosaur - that's $17.28 per month of 24/7 usage. If you've got a Rpi on for 4 hours per day checking your email - it amortizes out within 10 months.

This totally ignores the additional headaches associated with old, failing, unsupported hardware. You yourself described it as a "sinister amount of maintenance and stress".

You can make a philosophical argument here that a piece of hardware is never truly dead if it'll still push bits, but be honest here - if you're running up your power bill to avoid spending $100 on new hardware, if you're putting up with a PITA system for the heck of it - it's not useful, it's your hobby. Hobbies are fun, I think it's fun to keep old gear going too, but be realistic about the merits of such activities.

fit2rule2• 12 years ago

> a PITA system

Entirely subject, utterly arbitrary. I can still use an 8-bit machine to write a good letter.

leoedin• 12 years ago

don't use technology you don't understand

What do you mean by this? I struggle to understand the workings of a linear voltage regulator. To even try and suggest that any one person could "understand" anything much more complex is deluded.

How deep does your understanding need to go to allow you to use it? You'd have to go back 40 years to find computers that any one person has a hope of "understanding". Even at that point there was all sorts of abstractions - did the logic designer fully understand the power supply? Did they understand the CRT screen?

Retra4• 12 years ago

If you don't understand the human brain, you shouldn't be trying to use it to answer questions like this. ;)

jonnydark• 12 years ago

From what I can tell the writer is arguing that abstractions and encapsulations are wasteful of resources. The author claims that this is an inherent problem of abstractions and a lack of understanding of their implementations. The justification for this is that time and mental energy are resources as well and although we are being wasteful of other resources we use up less time and mental energy. The author states "However, during recent years, I have become increasingly convinced that the portion of true tradeoff is quite marginal" however doesn't seem to give anything quantifiable to illustrate this marginality. Much of the argument is based on anecdote and personal ideas on general human nature.

I am inclined to think that the author has come to this conclusion because time and mental energy are less valuable to them than other resources and that's not an unreasonable stance to take. However I don't think this view is widely shared and that thought most people agree that premature optimisation is the root of all evil.

EGreg8• 12 years ago

I think as a developer I am attuned to this most acutely. I see garbage ok the street or read about Amazon's new warehouse and think, "yeah, we may have 500 years to store our garbage, but then what?" When gas prices go down I think "past peak oil, we are just accelerating running out". Or when I read about overfishing, etc I think how we could irreparably disrupt ecosystems with a cascading effect.

But then I think how big this planet is and how ingenuous humans are collectively, and wonder - am I overworrying? I see small waste accumulatung but in scandinavia they're discovering ways to recycle garbage into resources!

So, I don't know. Thoughts?

Houshalter3• 12 years ago

This entire thing is just a rant against specialization and abstraction. Both in economics and computer programming. This is just ridiculous. Without specialization the world wouldn't function at all. No single human being can build a car by himself from raw materials. Programmers shouldn't be expected to write their web apps in assembly, and can you imagine the mess if everyone tried that.

The author starts out with a legitimate complaint that things are suboptimal. But his cure is worse than the disease.

tjradcliffe• 12 years ago

The second comment on his blog says he's just repeating what Marx said, which is not quite correct but in the right ballpark. People who don't know anything about economics tend to do that.

The "problem" of exponential growth is the belief that it will result in using up all available resources rather than finding more efficient ways to use resources we didn't even know we had. Engines become more efficient, power generation becomes lower cost, food production moves from farms to factories... all of these are good things. We feed the world with far fewer resources today than we did a century ago when many more people were starving, even while we use more resources building smartphones and sending spacecraft out to explore the solar system. We solve problems our ancestors didn't have because they still hadn't solve the problems they faced.

Economically, exponential growth is required to keep the monetary system stable, but in human terms we live in a world where new problems always arise, and require new solutions.

Before Watt and Newcomb we didn't have many problems related to steam power. Before the Wrights getting to the airport was just not an issue for most people. Human invention creates new problems which human invention solves, and that is a treadmill we cannot get off.

Capital markets turn out to be a remarkably good method of funding new solutions, and private enterprise turns out to be a remarkably good way of exploring the landscape of new solutions. No one has ever been able to figure out how to incentive socialist managers to explore that landscape with anything like the same efficiency--particularly given the opportunities they have for corruption--and when you look at how inefficient capitalism is you'll realize that's saying something.

So we continue to press on, while a certain type of person who has been more-or-less loud in the past two hundred years shouts at us that it is all going to end badly Real Soon Now.

Somehow it never does. It could, certainly, but we're centuries away from that. Possibly millennia. In the short term, if the naysayers are so sure of their analysis, I recommend they apply it to the stock market and use their impressive prescience to get rich. It shouldn't be difficult for anyone who has the kind of deep insight into the workings of the world that they claim to have to manage that.

wmt• 12 years ago

Buggy civilization? I don't think so, the improvement through evolution favors those who aim to grow exponentially. The alternative, micromanaging extremely complex systems, will unfortunately fail.

Trying to force the global economic system into a software analogy just feels silly and wrong.

eropple• 12 years ago

Trying to force the global economic system into a software analogy just feels silly and wrong.

It's the same thing you see software developers do all the time. "I know this one thing, let me apply it to everything. No no, your job can't be hard, I'm a software developer, let me think for you."

Breadth matters.

api• 12 years ago

Meh. Describe an alternative to a growth-oriented society that isn't either feudalism, some kind of white agrarian fantasy, or an even more imaginary "sci-fi utopia where everyone wears togas." (Google The Venus Project and the Zeitgeist Project for an example of this stuff.)

crpatino4• 12 years ago

Don't discard feudalism just yet. If nothing is done (and there are reasons to believe that not enough will be done to count as "more than nothing") this is what we will be getting by default.

Also, do not conflate the social concept of feudalism (survival by pledging allegiance to a chain of command that competes with other chains of command for access to resources) with the particular customs, laws (or lack thereof) and religious & aesthetic sensitivities that occurred historically the last time this happened in Western civilization. It can be argued that global corporations are already modeled on those basis and beyond the control of outdated national democracies.

The resource leak bug of our civilization

Related Stories