Cracks in Reality: How our Systems Fool Themselves

http://ieet.org/index.php/IEET/more/eubanks20120611

by NotEvenNothing • 14 years ago

43 12 14 years ago

Third installment in the series of articles that ask if intelligence is self-limiting because motivation ultimately undermines survival.

marcusrobbins3• 14 years ago

This is the group of problems we need to solve in order to design effective government and stable markets.

jacques_chester• 14 years ago

I think his point is that they may be irresolvable.

To think about the world, you must first have a model of the world. Then you reason about the model, finally you take action.

Somewhere in there you have a motivation for following this observe-decide-act loop. Motivation provides a reference point towards which you want the observed system to evolve.

But there's a problem. The easiest way to satisfy the motivation component is to lie to it. Tell it things are just hunky-dory.

Any singularity-style intelligence will necessarily need to be built with some kind of anchoring motivations to try and stop it from getting out of control. But what's to stop it simply lying to itself and ignoring the outside world?

Trying to add a meta-motivation be as realistic as possible won't work, such a system will seize up with analysis paralysis.

One of the things that Hayekian economists argue is that knowledge about reality cannot be centralised; it is unevenly, lumpily distributed across the whole of humankind. No one actor does, and no one actor could, perceive the entire system. But it works, because no one actor has to.

marcusrobbins3• 14 years ago

I do not believe that it is irresolvable, I'm looking at biology for my counter example. In biology there exists a rigid framework inside which a "general" search of language/model is taking place, after a few billion years it seems to be doing ok. It is my hunch that there is a big something out there which will pull a lot of these issues together:

How do you build efficient markets? How do you build an effective AI? How do you design effective distributed systems? How do you build effective languages/models with which to compress the world?

Under what circumstances do such 'meta' searches fail and succeed? It's all beyond me but this is what my nose is saying...

anamax2• 14 years ago

> we need to solve in order to design effective government

Is there actually any demand for "effective govt"? (Yes, lots of people claim to want it, but do you really think that there's large scale agreement on what that means?)

sashahart2• 14 years ago

There is plentiful demand for 'effective government,' and a total lack of consensus on what it means.

In politics, 'effective' really means 'does what I want.' When there is a lot of conflict about what people want, the government is not 'effective' because (A) deadlock is frustrating and (B) yelling louder is a way to get more and intimidate the opposition.

A government is less like a "singular intelligent self-modifying system" and more like a war on controlled burn.

marcusrobbins3• 14 years ago

Isn't effective government a system which can maximise the following function?

G = SUM(Fi(W)) from i to WorldPopulation

Where W is the physical configuration of matter in the world. Fi is the function associated with citizen i that defines his notion of 'goodness'.

Bad government is one which tries to maximise an alternative function:

G = SUM(Fi(W) * Di) from i to WorldPopulation, where Di is some factor for each individual and where some Di are much greater than other Di. i.e. some individuals have much greater say over the configuration of the world which is chosen...

DanielBMarkham• 14 years ago

In extreme cases, state-adopted nominal realities may be purely driven by motivations, and have little or no connection to reality.

This essay was just a little too unfocused for my tastes. As an observer of many types of organizations of people, the predilection to spin is endemic. It's not just "in extreme cases"

I scanned the article and kept trying to find some meat, some kind of clear and forceful statement that the author was supporting. Instead it seemed like he was saying a lot of good things, but lacked the intestinal fortitude or insight necessary to actually get anywhere. I was left with a version of "it may be a property of monolithic systems that their motivations are at odds with an accurate perception of reality." That's almost useful, but not quite. There are a lot of interesting places to go from that statement, but we didn't go there. Hell we could have started deconstructing Aristotle and talking about how the tension between the usefulness of categorization versus all abstractions are leaky creates holes in what is knowable. Or we could have moved forward to sociology and discussed the role of the individual in large groups. Or we could have talked AI. Lots of cool places to take this train of thought.

That's a shame. It's obvious the author is well-educated and has thought this through. Sure wish he would have taken us somewhere a bit more interesting.

PaulHoule• 14 years ago

that's the trouble with discussion about A.I.

all of the exciting stuff isn't in the fundamentals and principles, it's in the exact details of how you train model X to do task Y, and few practitioners are going to give away information that's so commercially valuable.

lnanek2• 14 years ago

Not sure why you care about people or states in an AI article. They were just examples of the technique being discussed being put to use in a system. Seems like a complete article to me. Here is the technique that can be used in AI and here are some examples of how it works in other systems.

NotEvenNothing• 14 years ago

This is something I struggle with in writing what amounts to a long argument in discrete chunks. I was already 1000 words over my limit, and had to find an artificial stopping place. But partly where this is headed is the thesis that our systems of government are not very similar to ones that would be designed from scratch for survival as an individual intelligence apparatus, and that by comparing the theory to practice we can (possibly) improve them. The flip side of i is that if we aren't going to be serious about survival, we should create bureaucracies and economies so that they fail gracefully. Or else we should adopt an evolutionary, rather than singular, strategy: colonize other planets.

sashahart2• 14 years ago

What is the actual use case for a "self-modifying singular intelligence"? Define the problem you are trying to solve (as an actual task, not nebulous stuff about beliefs) and I reckon you will find more success.

There is a one nice way around the intractable problems of maintaining a huge model of the world: don't. Rodney Brooks' famous saying: "the world is its own best representation." And it isn't necessary or usually even helpful to translate all incoming data into some propositional model. Again: what is your use case? If you are just trying to satisfy some essentialist intuition of what it means to be intelligent and what 'must be' inside minds, then you will be lucky to ever get anything meaningful done.

If you want to write an agent to do something without ongoing guidance, and you have not thought out the task from the beginning to get a specific algorithm, do not start by making a monolithic program that makes lots of vague high-level decisions like "how much reality" based on abstruse constructions of data. Start with the data which is always available, processed minimally, and see how little you can do. Implement walking before you implement steering and implement steering before you implement path-finding.

Cracks in Reality: How our Systems Fool Themselves

Related Stories