Joyent Open Sources SmartOS: Zones, ZFS, DTrace and KVM

http://smartos.org/

by timf • 15 years ago

279 82 15 years ago

smartos.org

wmf3• 15 years ago

They're really going out of their way to not say "Solaris".

po2• 15 years ago

They're really going out of their way to not use Solaris.

SmartOS is comprised of the Illumos kernel (with ZFS, DTrace, OS-level virtualization and next-generation KVM) with BSD package management and a GNU toolchain.

I used Solaris for almost 5 years and when Sun was bought I thought for sure the cool parts of it were doomed. Illumos popped up and I thought that they didn't have a chance, it would wither and die. It's very nice to see that is not the case.

wmf3• 15 years ago

They're really going out of their way to not use Solaris.

Considering that the Illumos kernel is basically the OpenSolaris kernel, I don't understand what you mean. They are using Solaris, but for some reason they're trying to hide it.

killerswan• 15 years ago

Trademark disputes with hyper-litigious competitors would not be fun...

thirdhaf• 15 years ago

Well since they have a lot of former Sun engineers working for them I don't really think they have to SAY Solaris. As others have mentioned there are trademarks involved here too. Sun/Oracle have been hemorrhaging talent, particularly the former Fishworks team, for a while now and a lot of those people are ending up in places like Joyent.

jjm• 15 years ago

Cool. I see this really helping me when testing my application prior to deployment. Would like to see some more movement with No.de. Have been waiting for a bit to get access.

nyellin• 15 years ago

To clarify, Joyent open sourced SmartOS. Zones, ZFS, DTrace, and KVM are maintained by other companies and have been open source for ages.

Edit: That's not to imply Joyent is freeloading. There is another post on HN with an example of the awesome work they're doing: http://news.ycombinator.com/item?id=2887092

sciurus• 15 years ago

Can anyone either explain or link to material that explains the relationships between the different projects and distributions that have sprung up since Oracle discontinued OpenSolaris?

piotrSikora4• 15 years ago

After OpenSolaris OS/Net kernel development was closed, 3 forks were made: Illumos, SchilliX-ON and Stormix (sadly, they all seem quite inactive).

There are 3 distributions based on the first one: OpenIndiana, Nexenta and SmartOS. There is also SchilliX (based on SchilliX-ON) and StromOS (based on Stormix).

This is just based on my existing knowledge, so there might be more forks and/or distributions out there.

binarycrusader4• 15 years ago

Both IllumOS and OpenIndiana are quite active (which shouldn't be surprising since the latter relies on the former).

mrb2• 15 years ago

Wow this is huge. Having an OS be able to run both ZFS and KVM enables fantastic things. For example you can store each virtual machine disk template on a dedicated ZFS filesystem, and use "zfs clone" to rapidly deploy VMs (instead of using the KVM-level support for base disk images "qemu-img create -b ..."), as well as "zfs snapshot", "zfs revert", etc. The main advantage being that these clones and snapshots are possible while using the simple and fast "raw" KVM disk image format instead of the notoriously slower "qcow2" format that was, until today, the best format supporting base images and snapshots.

I, for one, have been wanting to use ZFS specifically like that for a while. This does not compare at all to running ZFS on, say, an NFS server or iSCSI SAN serving data to a server running KVM (ZFS data integrity is only verified remotely on the storage server, it is slower, etc).

Who ported KVM to Illumos? I know some old version of QEMU was running on Solaris at some point in the past, but I had no idea they had a full blown KVM port.

amazingman• 15 years ago

http://smartos.org/2011/08/15/kvm-on-illumos/

bcantrill6• 15 years ago

KVM was ported to illumos by Max Bruning, Robert Mustacchi and me. Details are here: http://t.co/knW1UIJ

rednaught6• 15 years ago

Can you provide more details of Crossbow? Who maintains it/status? I hadn't heard anything about it since OpenSolaris went dead.

djcapelis2• 15 years ago

Uhm, isn't that a violation of the GPL given that you're mixing the complete work with non-compatible (by design) CDDL code? Did you ask for KVM to be relicensed for the port or something?

slillibri• 15 years ago

Kind of an off topic rant, but can we at least avoid url shorteners that point to other url shorteners? Here is the original link http://www.slideshare.net/bcantrill/experiences-porting-kvm-...

emaste• 15 years ago

I really like the use of DTrace in #if 0'd out code for identifying the highest priorities on porting. Having faced a similar task that seemed impenetrable at the start I would have really liked to have this technique available.

Nice work and congrats on the port.

jlawer• 15 years ago

While ZFS adds some advantages, if you were going for snap shots and quick deploy with KVM your best bet was to use LVM as a raw device. A quick lvm snapshot + virt-clone + virsh start will let me have a cloned vm up within seconds.

Thats not saying ZFS doesn't add a lot (checksum backed store, heap of other features). But I would actually expect LVM to be faster, precisely because it is "crappier" (no checksumming, very basic address remapping, etc).

The real question will be support, if they can convince people that this will have enough life to build a user base large enough to support it moving forward. Especially if oracle stops publishing code under open source licenses.

mrb2• 15 years ago

The problem with LVM snapshots is that you have to manually reserve a portion of the logical volume to store them. And if you calculate your needs incorrectly, LVM runs out of space to preserve them, and they become corrupted (they are reported as "INVALID" in lvdisplay IIRC).

That's why I prefer KVM's built-in base image support (qemu-img create -b ...) to quickly provision VMs, instead of LVM. That's how I architected KVM hosts for my employer, running 100+ lightweight VMs each, with 1000+ disk images available on disk.

rednaught6• 15 years ago

I'm having a difficult time wrapping my head around this. If Joyent wanted to stick with a UNIX then why not FreeBSD and help implement KVM there? Wasn't XEN already available for Dom0? I just don't see much mindshare in Illumos/SmartOS for keeping up with current server hardware drivers. I really would have liked to see one of the existing BSDs benefit from the time they've put into this project.

Obviously Linux already has KVM, and there are new implmentations of DTrace and ZFS. Btrfs is not too many more kernel releases away from being considered stable(fsck being a glaring problem) and if you want container-based virtualization you've got LXC(some prefer OpenVZ).

With that said, who is their target audience?

hello_moto2• 15 years ago

They have a lot of investment around Solaris (let's not argue about Solaris, OpenSolaris, Illumos) since a few years ago. Brought a lot of ex-SUN engineers, hired open source activist around OpenSolaris, probably built their infrastructure around it as well.

rednaught6• 15 years ago

Thanks. But didn't they also have a lot of FreeBSD such as during the TextDrive days?

dap2• 15 years ago

You're suggesting Joyent should have finished porting DTrace, ZFS, and Zones to FreeBSD, and then port KVM to FreeBSD, instead of just porting KVM to Illumos?

rednaught6• 15 years ago

FreeBSD has already had ZFS and DTrace for years. They have Jails which would have benefited from Solaris' stronger Zones/Containers.

There are already a lot of BSD users. Where are the Illumos users? Are you suggesting that all BSD users go to SmartOS? I used HP-UX and AIX for many years...the one off UNIX systems just end up recreating each other due to fragmentation. They are not the way forward. The smaller mind share/usage of OpenSolaris/Illumos isn't helping. Wouldn't contribution to a BSD have strengthened the community?

grey• 15 years ago

The Linux implementations of ZFS are dog slow, and for the last few weeks I've been evaluating btrfs as a possibility for my home storage server, and I've ruled it out completely.

btrfs has been "Close to a stable release" for years now, but if you follow their mailing list there are still people reporting total loss of large filesystems once or twice a week (This week I saw filesystem corruption from an unexpected power outtage, and last week there was a data loss/corruption bug caused by a pool of drives with different sector sizes), I desperately want btrfs to be mature but it's more than a few releases away from being stable.

lsc• 15 years ago

last I looked, xen was culled from openindiana (then the only functional Illumos based distro) - this was back in the days when we thought xen was dying, back before it was pushed upstream in Linux.

mkup• 15 years ago

Better they had ported KVM to FreeBSD.

Who will add support for the new hardware to basically abandoned OpenSolaris kernel? Oracle? No. That small company? No. Leveraging FreeBSD community to grow the list of supported hardware would be much, much wiser.

nathanb3• 15 years ago

"ZFS makes SANs and other expensive, redundant storage systems obsolete"

Yes, because clearly what enterprise-level storage customers really want is to roll their own storage solutions using open source technology.

daeken• 15 years ago

Many, many companies do want this, yes. Back in 2005-2006, I worked at MP3tunes and among other things I helped design the storage system. After testing a bunch of storage solutions (many of them high-end systems), we ended up rolling our own using Linux + MogileFS. This was scaled up to a couple hundred terabytes before I left, and even higher afterwards. Sometimes the off-the-shelf solutions simply don't work, especially at scale.

mitchty• 15 years ago

Pretty much, the amount of times we've had issues with new storage vendor arrays from a 3 letter company that starts with an E, and ends with a C, is a bit too much to count.

Firmware bugs that affect anything on a fabric, that affect how it distributes its cache slot locks on writes, issues with the entire array acting funny, which are blamed on either the server hardware or os itself until proven to be an issue with the array (this is far too common to be honest, yay for "support" contracts), etc....

Yes you get "support", and I use the term lightly, with big vendors, but you also have to take a machete through their support organization to get to someone that can help you with a problem. At times rolling your own solution will end up being both cheaper and less problematic. The old adage of "Nobody ever got fired for choosing IBM" may soothe managers minds, but wait until you do end up buying those fancy pants high end arrays and find out how much snake oil turns out not to work on them.

acdha3• 15 years ago

In many cases, yes, because you can only afford gold-plated storage for your most important apps.

toddmorey2• 15 years ago

I'd also argue that you don't always need gold-plated storage. Think about user docs that are infrequently accessed , numerous, and must be retained for long periods. Deploying NetApp or EMC for that use case doesn't seem to make sense with options like this or OpenStack object storage.

toddmorey2• 15 years ago

If it can be an order of magnitude cheaper, why not? Enterprise storage has commanded a high premium for a long time and of course enterprises are storing more data than ever, not all of which needs the same performance characteristics. I welcome the shakeup in this space. The old days where you fretted endlessly over which data to preserve on expense SANs are fading behind us.

nathanb3• 15 years ago

I think in general the premium goes to pay for support contracts and service as much as for the atoms and bits. Of course, it can always be argued whether those contracts and services are worth the premium, but many enterprise customers (Google being the obvious high-profile exception) seem to think so.

strlen• 15 years ago

This is really great. Joyent has been running this internally for years. The combination is just great: ZFS, DTrace and KVM speak for themselves, but another great ingredient is their use of the NetBSD userland (in place of legacy Solaris one).

Linux is great, but a monoculture benefits no one.

kraemate• 15 years ago

This is simply amazing. Tracing KVM using Dtrace is going to throw up some pretty useful and surprising results. And add ZFS on top that as well - with all its features (sp. dedup+COW) useful for virtualized hosting.

Anyone know if linux has any variant/clone of dtrace yet? (and no, not systemtap)

otoburb• 15 years ago

Paul Fox seems to have been working on a Linux DTrace port since 2008 (based on the directory listing). He blogs about his progress on DTrace (and other things) at http://crtags.blogspot.com/

sogrady• 15 years ago

Here's Bryan Cantrill, one of the original creators of DTrace, on the status of the port:

https://plus.google.com/105843697186982227624/posts/5uk5SHrh...

darklajid• 15 years ago

The biggest yay for me is in the fine print:

"SmartOS is comprised of the Illumos kernel (with ZFS, DTrace, OS-level virtualization and next-generation KVM) with __BSD package management and a GNU toolchain__."

Oh god, this makes me happy. I gave Solaris a couple of tries, but I've never felt at home there. These two facts mean that I'll try again.

barrkel• 15 years ago

Try out nexenta too. Debian userland.

binarycrusader4• 15 years ago

Solaris 11 Express has the GNU toolchain and a modern package management system.

For those that want something like SmartOS but free, and with a modern package management system, I would suggest OpenIndiana. OpenIndiana will be integrating many of the SmartOS changes.

sgt• 15 years ago

Fantastic news. I've been excited about Illumos for a long time, and now we're finally getting a decent distro with commercial backing. My servers will be very happy when they hear about this.

ristretto• 15 years ago

I 'm happy for all the involvement of Joyent in opensource, but how about also dropping the prices in their accelerators? I 've been using the same servers with the same prices for 3 years with zero upgrades.

foobarbazoo• 15 years ago

Would be sweet to run SmartOS on Backblaze's storage hardware.

jkahn• 15 years ago

I'm struggling to understand why ZFS makes enterprise storage obsolete. One of the benefits of the SAN is that it is shared amongst a number of servers.

How does ZFS help with that? Assuming you install SmartOS as virtualisation host, you'd still need some kind of shared storage.

[Edit:] The reason I point this out is that if you lose a host, you lose all VMs running on that host. And then you won't easily be able to get them back online on another host if you've lost your storage as well.

wmf3• 15 years ago

You can use ZFS+Comstar to create a SAN controller.

organico• 15 years ago

But you were able to do this before SmartOS... I'm curious about this too, and how Joyent have architected their cloud arrangement. I feel like perhaps hypervisors have local ZFS storage, as opposed to large SAN-backed virtual machines, but I can't really find much useful information on how Joyent/SmartOS implement/architect storage...

kermitthehermit• 15 years ago

This looks very good.

How can packages be added? How can it be reconfigured? How is one supposed to add management tools (think libvirt, openstack) to it?

I took it for a spin and noticed it had kvm, a whole lot of regular Illumos binaries and that it had a BSD-ish userland.

I couldn't find anything like libvirtd, I didn't get to check if it has it, though.

RexRollman2• 15 years ago

As an OS geek, I always enjoy the annoucement of a new OS, even if it is one I personally have no use for. Choice is good.

Joyent Open Sources SmartOS: Zones, ZFS, DTrace and KVM

Related Stories