Skype Outage – I run a supernode and here’s what I think happened

http://blogs.voxeo.com/voxeotalks/2010/12/23/skype-outage-i-run-a-supernode-and-heres-what-i-think-happened/

by jdupree • 16 years ago

99 43 16 years ago

blogs.voxeo.com

Groxx• 16 years ago

A wonderful reason to have pre-release testers for every piece of software that communicates with other software, any time you plan to push updates. With pulls, your early updaters are your test beds; with push, you have to create your testing groups.

Daniel_Newby• 16 years ago

Push software is even better about testing: you can forcibly upgrade 1% of installs, monitor them intensely, and let the (non-upgraded) system executive roll back the version if something bad happens. The system executive can even automatically roll back if it loses contact with the vendor, in case the upgrade goes so horribly wrong that it brings down networking.

guelo• 16 years ago

The problem with a big adaptive p2p network is that it is close to impossible to test under real production loads.

jrockway• 16 years ago

My guess is that the average supernode is on something like 6M/1.5M DSL. This is easy to simulate.

Now, if most of your nodes are on machines directly connected to 10GbE, then that's a problem. But most Skype nodes aren't. (I do imagine there are a few connected to 100M+ connections. But only a few.)

pyre5• 16 years ago

They could have setup their 'mega-supernodes' running the new client in a production environment, so that if it crashed horribly, it wasn't a client machine that failed (or their entire network).

Upgrading (relatively) untested software network-wide on what is essentially their critical infrastructure is bad news. If this didn't happen now, it would happen sometime. It was just a question of when.

buro9• 16 years ago

Just checked my Windows event logs... as I also run a supernode and yup, one Application Error at 2010-12-22 15:51:10 from Skype.

Tech data shows version 5:

    Faulting application name: Skype.exe, version: 5.0.0.152, time stamp: 0x4cb31516
    Faulting module name: KERNELBASE.dll, version: 6.1.7600.16385, time stamp: 0x4a5bdbdf
    Exception code: 0xe0fafafa
    Fault offset: 0x0000b727
    Faulting process id: 0xd20
    Faulting application start time: 0x01cba1ea3b70b5ef
    Faulting application path: C:\Program Files (x86)\Skype\Phone\Skype.exe
    Faulting module path: C:\Windows\syswow64\KERNELBASE.dll
    Report Id: 4b2781d7-0de3-11e0-bbdc-005056c00008

I didn't upgrade voluntarily.

jallmann• 16 years ago

Nice to see the CEO of a large company that knows how to walk the talk. I've also seen him post on HN. Does Steve Ballmer know the difference between a micro and monolithic kernel?

ra• 16 years ago

I was thinking the same thing. What other big company CEO's would even think of investigating why their Skype was more jittery in some hotels, let alone actually know how to do it.

bl4k• 16 years ago

probably busy running big companies

js3309• 16 years ago

I am also surprised that a company as large as his, uses Skype instant messaging for their communication.

akalsey• 16 years ago

We're a very distributed company, with some projects being worked on by people in a dozen locations, so some form of IM is needed.

I'm a big XMPP proponent (and Voxeo acquired my xmpp-based company) and we do use XMPP for a lot of things inside Voxeo (like Phono, our jquery softphone: http://phono.com/).

Every voxeo employee has an XMPP ID, but they don't tend to get used. People gravitate to Skype naturally. It's the user experience on the client side of XMPP that keeps people from using it extensively for internal communications. At any given time I've got 40+ group chats going on and we create and destroy group chats many times a week for specific needs. The ability to make a voice call to one or more participants of the chat is also a winner.

The fact that everyone on the planet has a Skype ID also helps. We can easily pull partners and customers into chats as needed. They're already on Skype, so we don't need to teach them anything new.

It's a lesson in usability, certainly. Create a client that allows users, without any assistance, to create groups, move seamlessly between text, voice, and video, and has a foolproof signup and setup process, and folks will use it.

cookiecaper4• 16 years ago

Maybe this will cause Skype to take Linux and Mac more seriously. If you've used the Windows version, you know almost all development has occurred there, whereas the Mac and Linux clients have looked the same for the last four or five years. The Linux client did have a major version bump about a year ago iirc, and that brought some needed features, but it was mostly the same.

Also, hopefully this will teach Skype to do more with open-source. I really hope they open the client up. This bug may have been caught, and things would definitely have turned out differently if Skype ran freely on other platforms. Maybe someone could even factor out a "Skype server" instead of an exclusive policy of client supernodes. Even serious torrenters rent a server somewhere to host their torrents -- P2P doesn't have to be strictly consumer-level connection, and really shouldn't be.

michaelbuckbee2• 16 years ago

Agreed, I've always thought that Skype would one day roll out an enterprise type "Skype Server" that would explicitly make performance better within corporations.

eps2• 16 years ago

Why would they want to fragment the largest botnet in history? :)

dflock• 16 years ago

They've already said that they're going to open source the client (http://blogs.skype.com/linux/2009/11/skype_open_source.html)... they just haven't actually done anything about it yet.

They'd better get a move on though, otherwise someone else (http://www.enrupt.com/index.php) is going to do it for them: http://www.enrupt.com/index.php/2010/07/07/skype-biggest-sec...

The trouble is that Skype is so closed and has been seemingly uninterested in open source and Linux, it's hard to find open source dev's who are interested in developing Skype related stuff.

cookiecaper4• 16 years ago

Their opening of the client is more of an opening of the interface -- all of the real work will still be performed by a proprietary blob that the open interface interacts with. I don't think that will be very useful. I'd prefer if they released something that was _really_ open-source.

anon114• 16 years ago

Or, maybe this will cause users to take Jabber/XMPP more seriously and stop using proprietary technology for corporate IM.

I've worked at places where management is completely gaga over Skype and would push me to support it despite the fact that I had no ability to block spam, troubleshoot messaging problems or integrate our IM system into existing Asterisk, SSO, monitoring and collaboration solutions.

IMHO, Openfire is far more flexible, extensible, secure, reliable and most importantly - manageable as a service.

http://www.igniterealtime.org/projects/openfire/

disclaimer: I do not work for Ignite and have no vested interest in their business.

cookiecaper4• 16 years ago

When an XMPP solution that's as easy to use as Skype comes out I'll be all for this. As it stands, even Google's video chat is much more difficult for end-users to install and use.

I used to use Ekiga for video chat and it was much shoddier than Skype, constantly dropping calls, refusing to release the audio or video device so that we couldn't call back, and other serious bugs. Skype "just works".

I really hope that someone comes up with a decent free software competitor, but it doesn't really exist at this point. The fastest way to solve this problem would be for Skype to become free software.

billpg• 16 years ago

Are Skype relying on third parties with no contractual relationship for an essential component of the network?

ars• 16 years ago

Yup. But they do mention it when installing. You can also disable it.

If you are not firewalled or NATd you are a supernode automatically - on the plus side you get better sound quality for voice calls.

Normally the bandwidth used is pretty low since you are mostly forwarding text messages.

When they said mega-supernodes they meant machines controlled by them that do nothing else, and are on high bandwidth connections. (My bet is lots of amazon instances.)

yellowbkpk• 16 years ago

I've looked around for the ability to turn off supernode on Linux and Mac clients for a while but have come up empty. Do you know which section of the preferences/settings it's supposed to be in?

michaelbuckbee2• 16 years ago

Skype's EULA pretty clearly states that this is the case:

3.3 Utilization of Your Computer: Skype Software may utilize the processor and bandwidth of the computer (or other applicable device) You are utilizing, for the limited purpose of facilitating the communication between Skype Software users.

_stephan• 16 years ago

I had the same issue: Shortly before Skype went completely offline, my Windows Skype client suddenly started crashing every few minutes. The Mac client didn't have that problem.

Some bug in the new version is probably the likeliest explanation, but if someone deliberately attacked the Skype network, it would probably look similar, right?

jvagner• 16 years ago

...or vice versa. a bug could easily look like an attack.

Dylan16807• 16 years ago

You're not going to get full client crashes without a major bug. Some amount of attack is possible but I've been getting crashes for a couple weeks now and skype should have fixed things.

schtono• 16 years ago

tl,dr; a forced software upgrade kept crashing all skype windows clients, taking down supernodes as well.

hackermom2• 16 years ago

I, too, run a Skype supernode - completely unvoluntarily, barring the possibility of something hidden behind an asterisk in the EULA. Does anyone know if there is a way to "opt out" of this? Everytime I start skype up it makes a million connections to everywhere and without even having a call on the line my measly 2.5 mbps upstream is permanently choked.

citricsquid• 16 years ago

https://www.cs.uwaterloo.ca/twiki/view/CF/SkypeConfiguration...

Found this link lower down :-)

hackermom2• 16 years ago

Sorry, I should've mentioned that I'm an OS X user :)

With that said, I've had a look-see in Skype's .plist and its various Application Support files, but oddly there's nothing "grep'able" to be found anywhere...

st3fan• 16 years ago

I seriously doubt the Skype supernodes actually run in regular clients on people's desktops. I'm pretty sure these are special servers placed strategically on the net.

If you see thousands of connections on your Skype at home then there probably is some weird P2P problem going on.

zb• 16 years ago

I'm sorry to be the one who has to point this out, but you have no idea what you are talking about.

http://www.skype.com/intl/en-us/security/universities/

eps2• 16 years ago

Well, the big question is not if Skype went down becase lots of supernodes crashed. The question is why they crashed, and if the trigger was external. There are messages on p2p-hackers list arguing just that:

  I have evidence that suggests this is being done 
  by a ddos attack on the supernodes' object list 
  cmd parameter.

johndyer• 16 years ago

Nice write up, pretty interesting that Skype has not come out and said anything about WTF happened. Also ironic that their security blog's last posting it titled "The importance of updating" :) ( http://blogs.skype.com/garage/ )

-John

bsg75• 16 years ago

http://arstechnica.com/software/news/2010/12/skype-brought-d...

Skype Outage – I run a supernode and here’s what I think happened

Related Stories