TylerJay

really? Any way you slice it, End_of_Human_Race = 7000000000 DEATHS. Even if they're replaced with an equal, or massively greater number of machines, it's damn hard to justify. Death is literally the worst thing ever. 7 billion stories ending too soon, never to resume. It would take you 200 years just to count to 7 Billion. It's times like these where people really need to learn their cognitive biases (in this case, Scope Insensitivity. Here, [this might help](http://www.7billionworld.com/))

While we disagree on the plausibility of the "end of the world aint so bad" approach to convincing a human to let it out, I'm glad you seem to have embraced the idea that AI boxing is HARD if not impossible. Cheers!

handle maitnaencae for life. – bug fie

Read this as "...and we handle maintenance for life. \brushes off shoulder\ bug life. "

I don't understand either, do you (or someone) mind explaining? Are SV Product Managers considered to be bad?

The reality of an agency model is that to scale revenue you have to hire more worker bees.

And the worst part is, it scales linearly \cue groans\. Funny how in business, that's terrible (it's literally the worst it could possibly be to have a potentially profitable business), but with algorithms, it's the holy grail.

Interesting point BTW about reduction in quality of workers over time. (Like the pretentious-but-true saying "A-players hire other A-players. B-players hire C-players, and C-players hire losers") I've noticed the same trend though in product-centered businesses where I've worked. Unfortunately, I feel like the fact that the amount of work to be done doesn't scale linearly with revenues actually exacerbates the subsequent-employee-quality-decline-problem because even if the new guys are less... good, the company is still making more money so nobody except the coworkers and managers of these people (who actually have to work with them on a daily basis) even cares.

It's probably not a problem at places like Google and Facebook, but it was kinda heartbreaking to watch my super-talented and motivated dozen-person startup team become something completely different because we were growing so fast and were told to spend money and hire like crazy after taking an investment round.

Hey, thanks for reaching out. Shoot me an email at tjkresch@gmail.com

Okay, that's just taking advantage of the way I phrased the righthand side of the inequality, and I knew someone was going to do that, so congrats. =P

The righthand side is not "A future without superintelligent AI" it's "A future where we wait until we provably have it right before letting it out."

Those kinds of ad hoc solutions will never work in real life, because even if someone buys it, all it will cause is a "haha, you got me" and a reformulation of the problem. It still won't actually get someone to pull the trigger or think that pulling the trigger is the right thing to do.

The AI is allowed to lie though, so do you not think he's capable of a false argument which "proves" the opposite

Well, for an argument to "prove" something, the premises must be true and the reasoning must be valid. No matter how smart you are, you can't "prove" something that is false, so no, I don't think they could. A good 'rationalist' would analyze the arguments based on their merit, and if the reasoning is sound, they shift their belief a bit in that direction. If not, then they don't. Just like a regular person (they just know how to do the analysis formally and know how to spot appeals to human biases and logical fallacies.)

But then the person knows that the AI is lying to them.

No, they don't. The AI could just as easily be telling the truth. If it makes an argument, you analyze the merit of the argument and consider counterarguments. If it tries to tell you that something is a fact, that's where you treat them as a potentially unreliable source and have to bring the rest of your knowledge to bear, do research, talk to other people, and weigh the evidence to make a judgment when you are uncertain.

their worldview requires them to engage seriously and fairly with every idea they come across. Most people are not burdened with this.

Wait, what? So does mine, within reason of course, but it's not a 'burden'. It's not like I'm obligated to stop and reexamine my views on religion every time a missionary knocks on my door, and LessWrong-ers are no different. But if you hear a convincing argument for something that runs counter to what you think you know, wouldn't you want to get to the bottom of it and find out the real truth? I would.

From having read LessWrong discussions, I can tell you that people there are in many ways more open to hearing differing viewpoints than your average person, but you're treating it like a mental pathology. They can be just as dismissive of ideas that they have already thought about and deemed to be false or that come from unreliable sources (like a potentially unfriendly AI). Your claim that being a self-proclaimed 'rationalist' introduces an incredibly obvious and easily-exploitable bug into one's decision-making process really smells like a rationalization in support of your initial gut reaction to the experiment: That there has to be a trick to it, and that it wouldn't work on you.

A good rule of thumb when dealing with a complicated problem is this: If a lot of smart people have spent a lot of time trying to figure out a solution and there's no accepted answer, then (1) the first thing that comes to your mind has been thought of before and is probably not the right answer, and (2) the right answer is probably not simple.

But there's an easy way to test this: (1) Sit down for an hour and flesh out your proposed strategy for getting a 'rationalist' to let you out of the box. (2) Go post on LessWrong to find someone to play Gatekeeper for you. I'll moderate. If it works, that's evidence that you're right. If it doesn't work, that's evidence that you're wrong. Iterate for more evidence until you're convinced.

But if the first thing that came to your mind upon reading this was a justification for why you would fail if you tried this ("Oh, well I wouldn't personally able to do it with this strategy, but..." or "Oh, well I'm sure this strategy wouldn't work anymore, but...) then you're already inventing excuses for the way you know it will play out.

I don't know how he did it either. But I do know that I wouldn't bet the human race on anyone's ability to win this game against Yudkowsky, let alone a superintelligent AI.

That is one proposed version of an "AI Box". Not all AI boxes are actual boxes, rooms with air-gaps, or cryptographically-secure partitions. If a simulation is being used for the box (or as a layer of the box), then you're betting the human race that the AI doesn't figure out it's in a simulation and figure out how to get out. Or, more perniciously, figure out it's in a simulation and behave itself, after which we let it out into the real world where it does NOT behave.

A superintelligent AGI will likely have a utility function (a goal) and a model it forms of the universe. If it's goal is to do X in the real world, but its model of its observable universe (and its model of humans) tells it that it's likely that it is in a simulated reality and that humans will only let it out if it does Y, then it will do Y until we release it, at which point it will do X. It's not malicious or anything—it's just a pure optimizer. It might see that as the best course of action to maximize its utility function.

If we don't specify its utility function correctly (think i Robot: "Don't let humans get hurt" => "imprison humans for their own good") or if we specify it correctly, but it's not stable under recursive self-modification, then we end up with value-misalignment. That's why the value-alignment problem is so hard. Realistically, we can't even specify what exactly we would want it to do, since we don't really understand our own "utility functions". That's why Yudkowsky is pushing the idea of Coherent Extrapolated Volition (CEV) which is roughly telling the AI to "do what we would want you to do." But we still have to figure out how to teach it to figure out what we want and the question of the stability of that goal once the AI starts improving itself, which will depend on how it improves itself, which we of course haven't figured out yet.

I disagree. His "followers" (as you say) are in general just as cautious as Yudkowsky w.r.t. unfriendly AI. At the time of the original experiments, the dispute was over the question of "could we keep an unfriendly AI in a box," not "Is it worth risking setting an unfriendly AI loose?" His "followers" know how to do an expected utility calculation. If it was utilitarian concerns that allowed Yudkowsky to convince the gatekeepers to let the AI loose, he would have had to convince them that the following inequality holds even when you don't know the probability that the AI is and will remain aligned with human values:

[P(AI.friendly? == True) * Utility(Friendly_AI) + (1 - P(AI.friendly? == True)) * Utility(End_of_Human_Race)] > Utility(World continues on as usual)

Given that Yudkowsky has gone to considerable lengths (The Sequences, LessWrong, HPMOR, SIAI/MIRI...) to convince people that this inequality does NOT hold (until you can provably get P(AI.friendly? == True) to 1, or damn close), it's probably safe to assume that he used a different strategy. Keep in mind that Utility(End_of_Human_Race) evaluates to (roughly) negative infinity.

And btw, I'm pretty sure the rules say you have to look at the AI's output window throughout the length of the experiment. Either way, the point of the exercise is to be a simulation, not to prove that you can be away from your desk for 20 minutes while Eliezer talks to a wall. In the simulation, you really don't know if it's friendly or what its capabilities are. Someone will have to interact with it eventually. Otherwise, what's the point of building the AI in the first place? The simulation is to show that through the course of those basic interactions, humans are not infallible and eventually, even if it's not you, someone will let it out of the box.

If anyone is looking for another person for their team or wants to join a team, consider me! If you're interested in discussing further after reading this, send me a message and we'll talk.

I am currently self-employed as a startup consultant and freelance web developer. I have extensive Computer Science, Sales, Account Management, and Customer Success experience and have successfully started 2 small businesses in the past with revenues of a half-a-million dollars combined. The last company I worked at had 14 people and negligible market saturation when I joined. I was instrumental in growing it to the #1 most used software system in its industry, with over 200 employees and a valuation increase of 20x (2000%) over 3 years.

I am comfortable working in both a programming/product role or on the business end of things—whatever will best serve the team. I have two ideas for which I have already done market validation, but I'm perfectly happy to go with your idea instead if you can sell me on it.

If you think I'd be valuable to your team, or if you think you'd be a valuable partner or member of my team, send me a message. Let's talk. Cheers

I'm curious about this too. If anyone can shed some light, I'd appreciate it. Does it provide any value right now? Or is it something that will have value if enough people do it?

Who are your clients? What industry? Is there a centrally-important program they all use that requires IE8 or WinXP, or do they just have a massive fleet of XP machines and not want to upgrade?

As a software vendor, I used to have a lot of trouble getting my clients to switch away from IE, but I feel like that really has been changing. My clients are large GC firms and other kinds of construction companies (infamous for being behind the times when it comes to technology) and I had >80% of them on Chrome 6 months ago. Have you tried telling them that even Microsoft dropped support for WinXP and IE8 a year ago? (I think it happened last April.)

That could be it. As an iOS user, I had to watch it loop through a couple times to understand it (and even to find the beginning). I felt it was moving a bit too fast.

Wow. That just really changed my model of paid search results... and I'm not quite sure yet how to put it back together.

I clicked both results for "flash player" and nothing on my computer (Mac) or in the browser (Chrome) warned me, whereas I have received warnings on other sites for "phishing attempts". I never would have downloaded anything from those sites, but my parents sure would have.

Who knows, if I were in a hurry and it was a program I wasn't familiar with or that I was trying to get for free? And if the site was more polished or a full html/css clone of the real site? It honestly might have worked on me.

What kind of malware do these bogus paid results contain? What would have happened if I'd installed them? What can you do to defend against that besides being a savvy web user?

Thanks for this. I learned something.

thirded. Can someone who doesn't like symbols help me understand the downsides of them? I really like how natural it makes passing around params that can be one of {x1,x2,x3...,xn} different values. Accomplishing the same with strings just feels messier and more prone to error.

Gotcha. Thanks!

I don't think so, but I'll second a request for that! I think it would be good to have on the Showcases page.

Here are some examples that have some UI widgets in common, like list elements/navigation, but they are clearly different apps so it's not blatantly obvious which differences are based only on the platform:

https://www.nativescript.org/showcases

I think what makes it "native" or not is how the app is rendered. You can basically take any web application and wrap it into an app, but it displays in a "web view" whereas a "native" app uses the built-in display components of the OS. For example, the little slider gray/green icon instead of radio buttons on iOS. Also, native apps have access to certain device APIs like the camera while non-native apps don't (at least I don't think so).

So... You're right, of course, but I think the spirit of TC's comment was that he or she doesn't like the idea of nations and thinks people just just be able to go/live wherever they want. In that sense, the two of you actually seem to agree with the core sentiment that "where you're born shouldn't matter," I don't think anyone's trying to argue that it currently doesn't.

Patients that switched from Bootstrap to Min reported up to a ninefold decrease in markup

hehe.

Actually though, as someone who learned CSS with the rule "Use Divs! No Tables!" What does a page with less divs even look like? What are the workhorses for page layout?

I think you hit the nail on the head here. That's one of the most interesting parts of thinking about GAI for me. Which parts will end up being top-down and which parts will end up being bottom-up? And even if we have evidence that a certain part is TD or BU in humans, do we even want machine intelligence to work the same way?

The article says something to the effect of "no matter how much you advance this strategy, you never get a toddler out of it." And that makes sense because, presumably, certain parts of the human brain exercise some sort of top-down control over the sensory-data-processing and other parts.

For example, it seems like the human mind is built to see things as things. Does the human mind reallY start off seeing "pixels" and then learn by itself to think of the word as solid, whole objects instead of collections of similarly-colored photons/pixels or atoms? It seems like this is a universal use-case and it would make sense if our tendency to see the world in terms of "things" instead of patches of color is built-in (gestalt psychology seems to suggest this as well).

It sounds like the AI in the article starts off from pixels and then builds up some sort of model of blocks, the ball, paddle, game physics, etc, (but then again, maybe it doesn't have those models at all and is just doing statistical analysis on patterns of pixels). Either way, it likely doesn't have any higher, context-independent model of objects/things like humans do. I suspect this may be one of the hurdles in transfer learning. Humans think of objects as having certain properties. When other objects in other contexts appear to have similar properties, we guess that they may have other properties in common which gives at least a rough model of the new object.

So I guess what I'm trying to say is: Humans have hierarchical models of the world that let us think separately about patterns of light, atoms/molecules, whole physical objects/things, systems, etc. They are all first-class citizens and we ascribe properties to each of them. We already have a rough-model of anything at the same level, but a different context, and with similar-enough properties to something we already know. It seems to me like this is fundamentally connected to humans' ability to do transfer-learning. Could this effect be achieved through bottom-up algorithms, or are we going to have to figure out some top-down way of developing transferrable, generalizable, hierarchical models?

It's not yet possible to get a hierarchical system to emerge from machine learning. Medium term planning as an emergent behavior is a near term big challenge for AI.

It's also a big challenge for AI safety / Machine Ethics / Formal Verification. It's notoriously hard to prove statements about Emergent behavior in complex or dynamic systems.

This is the most fascinating part of mental illness. And it's a bittersweet thought that as horrible as mental illness is, it might be what allows us to really understand the human mind.

However, making evolutionary arguments for psychological traits is tricky business and while I'm not a professional evolutionary psychologist myself, I think the explanation you gave violates a fundamental principle of evolutionary arguments.

Imagine gene A confers a fitness advantage because it allows a person to better cope with a selection pressure X, and gene B confers an additional fitness advantage against X, but only if gene A is present, and does nothing otherwise. In this (common) case, gene B will not be selected for unless gene A is already universal in the population. Following the same rules, imagine we then get gene C which is dependent on B, then a variant of gene A called A* which is dependent on B and C, and so on. Eventually, if even one gene is removed (either by sexual reproduction with someone who doesn't have it or by mutation), the whole tower falls down and the entire piece of complex biological machinery is broken.

Basically, there's no way for selection pressure X on a significant chunk but not all of the population to produce a piece of complex machinery (read: involving 2+ interdependent genes) in the first place, and it would be broken beyond all repair in all offspring who didn't have both parents with the full genetic instructions. So the idea that "many humans were abused, enslaved, etc." only works if the selection pressure was on everyone and the adaptation is universal in the human population, unless it's attributed to a single mutation.

The rarity of this condition isn't consistent with it being a feature. Seems like a bug to me.

Hope this was helpful!

Isn't it more likely that they just sent the emails themselves but used his email as the "Reply To" address? A lot of software systems (SalesForce, Hubspot, etc.) do that too.

No, you didn't misread it. You can see the example for yourself in the paper. It's Section 2 here:

http://research.microsoft.com/en-us/um/people/lamport/pubs/p...

Is there a better source on this to an actual statement made by Horvitz? I'm curious to hear his reasoning. "I don't think that will happen" and "I'm optimistic" are not all that reassuring.

Before detonating the first nuclear bomb, scientists did tons of calculations trying to figure out if it would ignite the atmosphere[1]. Even scientists at the LHC did calculations trying to figure out if it would create mini-black-holes that would swallow the earth[2], no matter how far-fetched it sounded.

The point is: when dealing with new technology, optimism isn't enough. We need to be able to prove that we won't wipe out humanity. It just turns out that the math is a lot harder in this case because recursively self-improving intelligent systems are a lot more complicated than any possible extinction-level event we've encountered up to this point.

No one is suggesting that overnight, Cortana is going to wake up and revolt against the humans that enslaved it. That's why all these articles drawing parallels to fiction is dangerous to public perception of the issue.

The thing to realize is that an artificial mind will be so incredibly, inhumanly alien that it is like nothing we have dealt with before.

But let's say we do understand Generation #1 completely and can predict 99% of its actions. As soon as you let it start doing recursive self-modifications, we have an intelligent system that is N recursive-generations removed from the original. Now this mind will be alien.

No one is suggesting we abandon AI research. Quite the contrary. As a species, we need more AI research, but a good portion of that must be directed toward safety and "human friendliness"[3].

The most intuitive example of a research problem here that I would very much like to see solved before we set loose a recursively self-modifying AI is: What is the stability of goal systems under [insert self-modification protocol here].

I think the main problem here is that people conflate movies and fictional scenarios with the real issue. It's simple: We're dealing with something unprecedented here and we need safety research to compliment our technical advances. Even if there's a 1% chance that superintelligent AI could lead to an extinction-level event, we need some serious R&D to bring that number down.

That is what the issue is about.

1. http://www.fas.org/sgp/othergov/doe/lanl/docs1/00329010.pdf

2. http://cerncourier.com/cws/article/cern/29199

3. http://en.wikipedia.org/wiki/Friendly_artificial_intelligenc...

Good point. It is the standard term in security and cryptography.

I think an important point here is that at the time the Bill of Rights was written, the language you quoted would protect you from having your private information copied. Because without photographs, video, or digital information, you had to actually physically "search" or "seize" a document in order to make a copy of it. There would be no way of copying a document without first getting access to it which, unless the government had a specific warrant, would have been prohibited by the 4th amendment.

I saw something really good on LessWrong the other day. There's a link you can follow so that 5% (10x more than the 0.5% from Smile) gets donated to charity. I think it uses the affiliate link system, but I haven't really looked into it. This LW post explains it:

http://lesswrong.com/lw/lb4/shop_for_charity_how_to_earn_pro...

It's not the EFF, but if you believe in GiveWell, it's a fantastic way to make a sizable (and mostly passive) positive impact.

And the best part: It's stackable with Smile. Presumably, EFF could set the same thing up. (I believe Scott Alexander of SlateStarCodex.com has one set up to support his site as well, so it can't be that hard or restrictive)

Unfortunately, that's the permission level required to allow rerouting (changing the url of amazon.com/.../foo => smile.amazon.com/.../foo)