stanislavzza

Straight out of Life Artificial...

This is pedantic, but I don't like the formulation of entropy as sum of p log(1/p). I think of log(p) as information of a single event, for which log base 1/2 gives the answer in bits. This makes the negative sign unnecessary, and technically all these formulas should specify the base of log > 1. Everything is cleaner with log base 1/2 (instead of e.g. using the equivalent negative log base 2). This comes up in log likelihood all the time too. I guess it's a prejudice against fractional bases.

It's not just crashing/halting. See Rice's Theorem. The machine can't predict its own future behavior in most ways that are important.

I'm not an expert in this stuff, and my point was that a serious treatment of super-intelligence should address such limitations to computations. In particular the "super" part seems to imply solving exponential-complexity problems in linear time. I remember looking in NB's book and not finding it.

(not OP) It's been a while, but if I recall correctly NB didn't address the technical limitations of Turing machines, e.g. the halting problem. How is a machine supposed to make itself smarter when it can't predict that it won't just crash after a code modification? Or just hack its motivation function (wireheading). The papers I've seen on the latter problem (years ago) start by assuming that the halting problem has been solved, essentially, by giving the agent non-deterministic computational powers. Biological intelligence evolved, so it's perhaps more realistic to imagine an ecosystem of computational agents competing in a trial-and-error race, but that makes the whole thing vulnerable to any threat to that superstructure--much more fragile than AI-go-FOOM.