71 TiB DIY NAS Based on ZFS on Linux

http://louwrentius.com/74tb-diy-nas-based-on-zfs-on-linux.html

by ors • 12 years ago

164 128 12 years ago

KaiserPro• 12 years ago

I know I'm going to suffer hard for this comment. However I'd suggest reading the ZFS optimisation guide.

I know the allure of raid 7 is strong, however if a disk were to fail on a LUM thats 24 disks wide, the chances of a rebuild is pretty low. (I know you've since changed that) the rebuild time is around 60-90 hours with no load.

(we use 14 disk raid6 LUNs for the balance of performance vs safety, however we've come pretty close to loosing it all.)

You'll get better performance if you have many smaller vdevs. You'll also compartmentalize the risk of multiple disk failure.

a raid 0+6 of four LUNs of 5 disks will have much greater performance, and will rebuild much faster, without risking too much. However you will loose more space

rancor• 12 years ago

Nope, you're right on the money. My first thought was "this guy needs more VDEVs". Given the hardware and use case, 4x 6 disk RAIDZ2 VDEVs would make a ton more sense, and could also be cross-cabled to allow a controller failure.

storrgie4• 12 years ago

Smaller VDEVs make it a bit easier to increase size if you're rolling your own system. Buying 6 more drives instead of 8 or 10 is a smaller cost.

roncohen• 12 years ago

"The purpose of this machine is to store backups and media, primarily video"

o_O

junto• 12 years ago

The guy walks around with a video camera permanently on recording his life?

Demiurge• 12 years ago

Or he's supports video processing/FX production. RAW data takes up multiples more than compressed outputs.

tinco• 12 years ago

I know a great many people collect recordings of tv series and movies, it's a hobby that can get quite out of hand disk wise. Most don't go for such a professional solution though. I know one who had 20+ external USB drives connected to a few servers.

storrgie4• 12 years ago

Took a 10 day vacation with a single DSLR shooting RAW... 60GB of photos/videos. I've not gone through and deleted anything yet... but I don't really have to either.

Elhana• 12 years ago

I have a similar pack of raw images, which I never got to clean up since I got spare storage space and it is not even that big - looking at those new Panasonic GH4 cameras that can shoot raw video with some crazy bitrate, you will need loads of storage.

nilsimsa• 12 years ago

o_O

jacquesm• 12 years ago

That's 1000 such vacations!

rodgerd3• 12 years ago

I have kids. This translates into lots of home video.

I am also ripping all my DVDs onto my home server. That chews through a bit of storage, but it's also a lot more convenient (and, with kids, less likely to end in tears of sctrached or bejammed DVDs). BluRays will be next.

It's easy to use a vast amount of storage.

jspiros24• 12 years ago

I have a similar capacity solution running at home, though I have far more (fewer disks per) vdevs. I've been running it for a few years now, so some things aren't as optimized as they could be, but it still works well for my uses...

  # zpool list
  NAME      SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
  lanayru  73.3T  53.7T  19.6T    73%  1.00x  ONLINE  -

  # zpool status
    pool: lanayru
   state: ONLINE
  status: The pool is formatted using a legacy on-disk format.  The pool can
          still be used, but some features are unavailable.
  action: Upgrade the pool using 'zpool upgrade'.  Once this is done, the
          pool will no longer be accessible on software that does not support
          feature flags.
    scan: scrub repaired 1.58M in 201h47m with 0 errors on Fri Feb 21 13:37:24 2014
  config:
  
          NAME                                                  STATE     READ WRITE CKSUM
          lanayru                                               ONLINE       0     0     0
            raidz2-0                                            ONLINE       0     0     0
              ata-SAMSUNG_HD204UI_______________-part1          ONLINE       0     0     0
              ata-SAMSUNG_HD204UI_______________-part1          ONLINE       0     0     0
              ata-SAMSUNG_HD204UI_______________-part1          ONLINE       0     0     0
              ata-SAMSUNG_HD204UI_______________-part1          ONLINE       0     0     0
              ata-SAMSUNG_HD204UI_______________-part1          ONLINE       0     0     0
            raidz2-1                                            ONLINE       0     0     0
              ata-SAMSUNG_HD204UI_______________-part1          ONLINE       0     0     0
              ata-SAMSUNG_HD204UI_______________-part1          ONLINE       0     0     0
              ata-SAMSUNG_HD204UI_______________-part1          ONLINE       0     0     0
              ata-SAMSUNG_HD204UI_______________-part1          ONLINE       0     0     0
              ata-SAMSUNG_HD204UI_______________-part1          ONLINE       0     0     0
            raidz2-2                                            ONLINE       0     0     0
              ata-ST2000DL004_HD204UI_______________-part1      ONLINE       0     0     0
              ata-ST2000DL004_HD204UI_______________-part1      ONLINE       0     0     0
              ata-ST2000DL004_HD204UI_______________-part1      ONLINE       0     0     0
              ata-ST2000DL004_HD204UI_______________-part1      ONLINE       0     0     0
              ata-ST2000DL004_HD204UI_______________-part1      ONLINE       0     0     0
            raidz2-3                                            ONLINE       0     0     0
              ata-WDC_WD30EFRX-68AX9N0_WD-____________-part1    ONLINE       0     0     0
              ata-WDC_WD30EFRX-68AX9N0_WD-____________-part1    ONLINE       0     0     0
              ata-WDC_WD30EFRX-68AX9N0_WD-____________-part1    ONLINE       0     0     0
              ata-WDC_WD30EFRX-68AX9N0_WD-____________-part1    ONLINE       0     0     0
              ata-WDC_WD30EFRX-68AX9N0_WD-____________-part1    ONLINE       0     0     0
            raidz2-5                                            ONLINE       0     0     0
              ata-WDC_WD30EFRX-68EUZN0_WD-____________-part1    ONLINE       0     0     0
              ata-WDC_WD30EFRX-68EUZN0_WD-____________-part1    ONLINE       0     0     0
              ata-WDC_WD30EFRX-68EUZN0_WD-____________-part1    ONLINE       0     0     0
              ata-WDC_WD30EFRX-68EUZN0_WD-____________-part1    ONLINE       0     0     0
              ata-WDC_WD30EFRX-68EUZN0_WD-____________-part1    ONLINE       0     0     0
              ata-WDC_WD30EFRX-68EUZN0_WD-____________-part1    ONLINE       0     0     0
            raidz2-6                                            ONLINE       0     0     0
              ata-WDC_WD30EFRX-68EUZN0_WD-____________-part1    ONLINE       0     0     0
              ata-WDC_WD30EFRX-68EUZN0_WD-____________-part1    ONLINE       0     0     0
              ata-WDC_WD30EFRX-68EUZN0_WD-____________-part1    ONLINE       0     0     0
              ata-WDC_WD30EFRX-68EUZN0_WD-____________-part1    ONLINE       0     0     0
              ata-WDC_WD30EFRX-68EUZN0_WD-____________-part1    ONLINE       0     0     0
              ata-WDC_WD30EFRX-68EUZN0_WD-____________-part1    ONLINE       0     0     0
          logs
            mirror-4                                            ONLINE       0     0     0
              ata-KINGSTON_SV300S37A60G_________________-part1  ONLINE       0     0     0
              ata-KINGSTON_SV300S37A60G_________________-part1  ONLINE       0     0     0
          cache
            ata-KINGSTON_SV300S37A60G_________________-part2    ONLINE       0     0     0
            ata-KINGSTON_SV300S37A60G_________________-part2    ONLINE       0     0     0
  
  errors: No known data errors

If anyone has any questions, I'd be happy to try to answer.

apinnes• 12 years ago

Hey mate, I've got a few questions if you don't mind:

How much does it cost in terms of power to have something of that size running 24/7 (I'm assuming it is?)?

Now you've set it up, does it require much of your time to manage monthly?

What sorts of things are you using the space for?

Lastly, how much did it cost in parts/time to get it all up and running, does it work out cheaper rolling your own or would you have been cheaper using a cloud solution instead, or does that just not meet your use-case/needs?

jspiros24• 12 years ago

It is running 24/7; I don't feel comfortable powering it down regularly; that is something I would worry about from the OP's setup, I wouldn't want to subject all those mechanical drives to so many power cycles over time. I don't have figures for only that machine, but my entire rack, which includes a router machine, two ISP modems, the ZFS-running machine and two SAS expanders, averages around 330 watts.

After setting it up, I wouldn't say that it requires any time to manage. Getting it all set up just right, with SMART alerts and capacity warnings and backups and snapshots, all of which I roll myself with various shell scripts, took a long time. Besides that initial investment, the only "management" I have to do is respond to any SMART alerts, add more vdevs as the pool fills up, and manage my files as I would on any other filesystem.

I use the space for just about everything. Lots of backups. I use the local storage on all of my workstations as sort of "scratch" space and anything that matters in the long term is stored on the server. The highest density stuff is, of course, media: I have high definition video and photographs from my digital SLRs, I have tons of DV video obtained as part of an analog video digitization workflow, media rips, and downloaded videos. (I even have a couple of terabytes used up by a youtube-dl script that downloads all of my channel subscriptions for offline viewing, and that's something I doubt anyone would do unless they had so many terabytes free.) I keep copies of large datasets that seem important (old Usenet archives, IMDB, full Wikipedia dumps). I keep copies of old software to go with my old hardware collection. I have almost every file I've ever created on any computer in my entire life, with the exception of a handful of Zip and floppy disks that rotted away before I could preserve them, but that is only a few hundred gigabytes. I scan every paper document that I can (my largest format scanner is 12x18 inches, so anything larger than that is waiting for new hardware to arrive someday), so almost all of my mail and legal documents are on there too.

(I had a dream the other night that someone got access to this machine and deleted everything. Worst. Nightmare. Ever.)

A cloud solution would not have met my use case, since one of the primary needs I have is to be self-sufficient in terms of storing my own data, and I also want immediate local access to a lot of the things on there. I do use various cloud solutions, but only for backup, never as primary storage.

Rolling it myself was definitely cheaper than any out-of-the-box hardware solution I've seen. The computer itself is a Supermicro board with some Xeon middle-of-the-range chip and a ton of RAM, and an LSI SAS card. Connected to the SAS card are two 24-bay SAS expander chassis, which contain the drives, which are all SATA.

I'd say that building something like this would cost you maybe about 4000USD, not counting the cost of the drives. The drives were all between $90 and $120 when I bought them, but of course capacity eventually started going up for the same price over time, so let's say another 3500USD for the drives.

easytiger2• 12 years ago

Another question is what, if any, failures have you experienced

pkaye• 12 years ago

How much extra power do you consume to run such a system 24/7?

jspiros24• 12 years ago

~330 watts.

aftbit2• 12 years ago

How do you handle backups of that much data? I'm struggling to backup my ~5TB pool.

jspiros24• 12 years ago

I've got some external hard drives that I rotate in and store off-site but still nearby (down the street) for some data. I also have constant online backups running to various locations/services (Linode, AWS, CrashPlan, Dreamhost, some private services). I don't backup everything, only the irreplacable personal data (so, I'm not backing up Wikipedia dumps); at current count, at most 6TiB of the data is irreplacable. I also prioritize, so even out of that data, the absolute most important data is backed up more often, to more locations, and the less important data is backed up to fewer locations.

laumars10• 12 years ago

How often do you perform scrubs? The best practice for consumer drives is weekly, but at 200+ hours scrub time I can't see weekly being practical :p

Also, why haven't you upgraded your ZFS file system?

jspiros24• 12 years ago

As you can see, I do not perform scrubs as often as I should. Until I switched to SAS a year ago, I wasn't able to complete a scrub at all. The scrub you see is one of the few I've been able to complete. I need a week or two where I'm not using the filesystem that much, because the scrub really kills performance of the filesystem with the version of ZFS on Linux I'm running. I'm intending to do an upgrade to the latest version of ZoL, and then run a scrub, sometime in the next 3 months.

I haven't upgraded because, well, I haven't really seen a need to? The original reason is that this pool began under zfs-fuse, and when I switched to ZoL I kept the version at the last version supported by zfs-fuse so that I could switch back if needed. I doubt I'll ever switch back, but I do like the idea of maintaining compatibility with other ZFS implementations in case of any problems. I suppose when the OpenZFS unification stuff actually finishes, I'll be happy to upgrade to the latest version?

sean130132• 12 years ago

What does "scrub repaired 1.58M in 201h47m with 0 errors on Fri Feb 21 13:37:24 2014" mean?

I have a much smaller zfs setup at home, as well as a few at work, and I would be super concerned if the scrubs were repairing data. Am I overreacting?

jspiros24• 12 years ago

It means I ran a scrub that completed on February 21, 2014 after almost 202 hours, and there were no errors, but there were some corrupt data blocks that were repaired from mirrors, adding up to 1.58MB of data that was repaired.

See http://docs.oracle.com/cd/E23823_01/html/819-5461/gbbwa.html... for more information.

maccam94• 12 years ago

Hey Jay, you should probably set up a cron job for scrubs :) (or, 6+ months is probably a bit long for an interval...)

-Cam

edit: never mind, just saw someone else asked about that and your response since I opened the page earlier today.

jspiros24• 12 years ago

Yes, I should, and someday I will. I'm playing it fast and loose right now. :-p

jimlei• 12 years ago

I used FreeNAS with ZFS before but was put off by their HW recommendations at around 1GB ECC ram for each TB storage space. They do say home users can soften this up, whatever that means.

Are they totally wrong, do they have some kind of special implementation that requires lots of ram not to fail, or are you just not worried about it?

ref: http://doc.freenas.org/index.php/Hardware_Recommendations

erkkie2• 12 years ago

It's not freenas specific but ZFS in general likes plenty of ram (and ECC is a must). It's questionable whether the GB for TB holds once you have "enough" in the 8+ GB range of ram

softkitty• 12 years ago

Yes they are wrong. This is a very soft recommendation. This recommendation comes from community users experience.

jl62• 12 years ago

Hi there. I'd love to hear a little more about what kind of methods of organisation you employ for managing such a large volume of data (particularly the parts which aren't downloaded from somewhere else). Not so much in terms of the storage infrastructure, but in terms of directory structures, links, indexes, etc..

Do you have millions++ of files or just a lot of very large files?

I'm asking because I feel that not enough is published on the subject of personal filing/archiving systems, whereas it's something we all do and there's a lot of best practice sitting out there uncaptured.

jspiros24• 12 years ago

I hate hierarchical filesystems.

A lot of my older files, sadly, are stored in "SORT/Sort Me/To be sorted/Old computer/Sort again/Miscellaneous..." and the like. My server has an mlocate index, so I'll use mlocate, and I'll use find sometimes. I make sure to preserve metadata like last-modified/created dates, so I can use that to narrow things down.

Newer stuff, I try to keep a bit more organized, but I still have lots of unmanaged stuff floating around. For big projects, or big files, that's easy enough; my photos are sorted into a Y/M/D hierarchy, my VHS digitization projects are fairly well organized, some other things have their own structure. For my scanned documents, I just dump them all into a mess of folders, but then have a custom Django app with a management command that indexes them and gives me a nice "document management" website, and then I just search based on OCR'd text or title or date.

I really hate hierarchical filesystems. After using computers for this long, I'm convinced that hierarchy-optional, metadata-driven stuff is the only future I'll be happy in. I long for the ability to save things without really having to say anything about where it's saved, and still be able to find it... So, sorry, I don't think I have a satisfactory answer for you, as I don't think there's a good solution to this problem as long as we have filesystems where the organizational primative is a hierarchy. Even with tag-based systems that build on top of that, it's usually clunky and you still fundamentally have to figure out where to save something "first", even if you plan to access it via tag/metadata later. Such a pain.

nrzuk• 12 years ago

Nice build, I have something similar but with 3TB disks. My disks are grouped into 8 disks as the thought of trying to recover the whole array if that fails I'd probably cry a little.

Also I invested in 10GbE, at first was only server to server using some SFP+ NICs and DAC leads I picked up off eBay for between £10 and £80. But have since bought a couple of Dell 5524 switches and a PCIe enclosure for my Mac and having 650MB/s read/write to my server from the desktop certainly reduces the waiting around :)

chubot• 12 years ago

Echoing one of the comments -- why Linux? I thought most open source ZFS configurations used FreeBSD. (I know there are the open source Solaris variants, but I would imagine most customers still pay for support.)

ZFS is production ready on Linux now?

snw3• 12 years ago

There is indeed a great and active open source ecosystem of Illumos (ex open-solaris) distributions.

Illumos is the de-facto upstream of open-zfs and has many features not found in oracle solaris. [1]

So it is well worth checking out at least one of those:

- SmartOS, which is a fantastic Cloud-Hypervisor [2]

- OmniOS, a classic server distribution, probably best for what OP is building. [3]

- NexentaStor, a storage appliance - but i have not tried it yet [4]

I can definitely recommend using ZFS on a Illumos system as it has much better integration with things like kstats and fault management architecture that I would miss on any other OS.

[1] http://open-zfs.org/wiki/Features

[2] http://smartos.org

[3] http://omnios.omniti.com

[4] http://www.nexentastor.org

_delirium3• 12 years ago

I notice you didn't mention OpenIndiana. Is OmniOS now recommended over OpenIndiana for the classic-server-install case?

blinkingled• 12 years ago

I used 0.6 series on Ubuntu LTS and it was stable. I have since then upgraded my Proliant uServer to pfSense 2.2 to do the routing/firewall duty and it seems equally stable at the least.

storrgie4• 12 years ago

I have a similar system at home (almost exactly the same except I don't have the quad network adapter) using 2T drives from Hitachi. I quite love being able to save everything, and the peace of mind that ZFS brings.

As a plug, I've been using CrashPlanPro for offsite backup, I've been very happy with their service.

illumen• 12 years ago

Do you also use linux?

storrgie4• 12 years ago

ZFS on Linux is a really good project with a wonderful community. I have happily been using Linux and ZFS together for a long time.

gonzo2• 12 years ago

Used Dell C2100, 2xXeon E5620 (quad core) 96GB ram, Chelsio 10GbE

Two 2.5" 160GB internal SSDs for ZIL and l2arc.

12x4TB drives.

$2000 + Chelsio NIC ($230) plus 12 x 4TB WD Red ($2000)

So for $4250. I get 48TB and a better performing system in 2U.

His system costs 6012 EUR, or about 7900 USD at today's exchange rate. For $8500 I can fit 96TB in the same space, have 192GB of RAM, 16 cores @ 2.4GHz, 740GB of SSD for ZIL and l2arc.

And redundant power.

ZenoArrow• 12 years ago

You're not comparing correctly. Your system would only have 96TB if you weren't using any form of RAID-like caching. The guy from the article is using ZFS caching to improve data reliability.

gonzo2• 12 years ago

He has 24 4TB drives in 4U, but only 16GB of RAM, and a single E3-1230 (quad core) at 3.30GHz, and 4 x 1Gbps networking.

I'm not sure what a "RAID-like cache" is. I know what redundancy is, and I know how ZFS works.

16GB is likely adequate for his storage needs. He doesn't have that many clients accessing the storage.

and I could keep two complete copies of the data on two machines, each with redundant power, etc.

kev0092• 12 years ago

ZFS SLOG (ZIL on a dedicated device) and L2ARC do nothing for data reliability, they are only performance enhancements. Reliability is guaranteed by the CoW and atomic nature of the FS.

Theodores• 12 years ago

How many Libraries of Congress is that?

Isn't there still a danger of typing something silly, an 'rsync --delete' type of command that bricks the whole thing? Rather than the one Borg Cube I think I would prefer to add disks/machines with disks on an as required basis, or keep the last system in the loft 'just in case'.

I am still amazed though at how much disk people 'need'. Even in uncompressed 4K resolution it would take a long time to watch 71TiB of video.

Erwin• 12 years ago

He does not mention it, bug ZFS has great support for snapshots (compared to e.g. LVM snapshots). You don't have to carve out disk space in advance where snapshots are stored and they are relatively high performance, so you can have an hourly, daily and weekly rotating set of snapshots (there are some scripts around for that retention). As the snapshots take up space, your available disk space just diminishes. And you can set it up so while in you are in dir "foo", your "foo/.snapshot/16:00/" has the contents of that directory when shapshotted hourly.

Another cool feature: you can do a send a diff between 2 snapshots as a file stream (over e.g. SSH) and replay it on another machine. So that's like using rsync except it's actually a stable snapshot (rsync can't fully handle files that changes while it's copying them) and you don't have to re-checksum every part of every changed file. So you can use that for DR.

Transparent compression is also great (as is deduping -- if you have tons of spare memory!)

BTRFS will be able to do the same thing... one day.

chrismonsanto• 12 years ago

BTRFS will be able to do the same thing... one day.

It already can. btrfs sub snap does snapshots (and they work the same way), btrfs send/receive do the diffs, supports lzo/zlib compression and deduplication

mrb• 12 years ago

This guy would have more predictible performance and higher reliability if he had organized the array as two 12-drive raidz2 vdevs (instead of 18-drive and 6-drive). The reasons are because (1) the chances of triple drive failure though low is higher in a 18-drive vdev than in a 12-drive one, and (2) performance (IOPS) is variable between two asymmetrical vdevs.

There is very rarely a good reason for doing asymmetrical vdevs in a single ZFS pool.

I approve of his choice of HBA (based on the SAS2008 chip). For those looking for even bigger setups, I maintain a list of controllers at http://blog.zorinaq.com/?e=10 where you can even find 16-, 24- and 32-port controllers. 8-port ones are the most cost effective as of today.

I have been a happy ZFS user for 9 years, both personally and professionally. Using snapshots, running periodical scrubs, having dealt with a dozen of drive failures over the years. Everything works beautifully - it's a pleasure to admin ZFS.

voltagex_• 12 years ago

Any hints on building a ~16TB array? Drive size, RAM size?

I've been eying this - http://www.supermicro.com/products/motherboard/Atom/X10/A1SA..., but the project is firmly in the "one day when I have a spare $5k" basket.

XERQ• 12 years ago

My zpool is now the appropriate number of disks (2^n + parity) in the VDEVs. So I have one 18 disk RAIDZ2 VDEV (2^4+2) and one 6 disk RAIDZ2 VDEV (2^2+2) for a total of twenty-four drives.

That's a catastrophe waiting to happen. EMC recommends no more than 8 disks in a parity array group (ZFS recommends no more than 9), specifically because larger arrays cause longer rebuilds, which are more likely to trigger a URE, in which case your SOL.

darklajid• 12 years ago

Can you expand the acronyms? I'm having the most trouble with URE.

dsr_• 12 years ago

unrecoverable read error.

13throwaway• 12 years ago

Can somebody give me some recommendations on how to do this with encryption? I am fine sshing into my server and putting in a password after reboot.

icebraining• 12 years ago

http://serverfault.com/a/588056

grizzles• 12 years ago

I'd like something like this but not such a big box. Are there any open source drobo-like projects out there? I'd like something like a drobo but one that's hacker friendly. Anyone know of a chassis like that?

rodgerd3• 12 years ago

Closest I can think of is the SilverStone DS380, which is a NAS-sized case that gives you 8 hot-swap SATA bays, 4 2.5" internal mounts, and supports mini-ITX motherboards.

kogir4• 12 years ago

ZFS is pretty great, but it's not perfect. Occasionally you have to dump and re-create a whole pool to fix corruption:

    [nsivo@news.ycombinator.com ~]$ zpool status -v
      pool: arc
     state: ONLINE
      scan: scrub repaired 0 in 0h43m with 0 errors on Wed Aug 13 20:14:53 2014
    config:
    
        NAME          STATE     READ WRITE CKSUM
        arc           ONLINE       0     0     0
          mirror-0    ONLINE       0     0     0
            gpt/arc0  ONLINE       0     0     0
            gpt/arc1  ONLINE       0     0     0
    
    errors: No known data errors
    
      pool: tank
     state: ONLINE
    status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
    action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
       see: http://illumos.org/msg/ZFS-8000-8A
      scan: scrub repaired 0 in 5h3m with 0 errors on Tue Aug 26 00:19:05 2014
    config:
    
        NAME           STATE     READ WRITE CKSUM
        tank           ONLINE       0     0     0
          raidz2-0     ONLINE       0     0     0
            gpt/tank0  ONLINE       0     0     0
            gpt/tank1  ONLINE       0     0     0
            gpt/tank2  ONLINE       0     0     0
            gpt/tank3  ONLINE       0     0     0
        logs
          mirror-1     ONLINE       0     0     0
            aacd5p1    ONLINE       0     0     0
            aacd6p1    ONLINE       0     0     0
    
    errors: Permanent errors have been detected in the following files:
    
            /usr/nginx/var/log/nginx-access-appengine.log.0

I can't delete that file or recover it from backup, and don't really trust the pool or our disk controller anymore. So we're moving to a new server, again. Good times.

I imagine this would be disastrous for someone who's filled a 24 disk array with data, and has to purchase a duplicate to safely restore to.

insaneirish2• 12 years ago

Do you have ECC memory on that host? That corruption is not normal and suggests a problem likely with your hardware.

Do you have snapshots? Can you read that file in a snapshot?

Filligree• 12 years ago

I've had the same thing happen on a similar system, with ECC. In that case I eventually tracked it down to a dodgy power supply, but the damage was done... it was time to dump and reload.

Corruption happens. A good filesystem needs to cope.

kogir4• 12 years ago

The memory is ECC. No data I care about was affected. I'm just surprised there's no "do whatever you need to make it consistent so I can move on with life" option and instead I have to bring up a whole new pool.

dsturnbull20493• 12 years ago

This is one of the key strengths of ZFS. Your log file would be silently corrupted without checksumming.

kogir4• 12 years ago

I agree completely. Much better to know than not. But having no option to knowingly just lose inconsistent data and move on is a bummer.

rodgerd3• 12 years ago

Regardless, it's still frustrating that so many failure modes for ZFS have an answer of "blow it all away and start again". And that's with Solaris/ZFS.

takeda2• 12 years ago

I got ZFS corruption myself. Although another issue (which actually brought my attention) was also common system crashes, whenever ARC was enabled. When inspecting cored dump it looked like random bit flips.

In my case the problem turned out to be a broken WiFi card (PCI). After I replaced it no issues for 2 years now.

laumars10• 12 years ago

You don't need to drop the pool to clear those error messages:

    zpool clear tank

Source: http://docs.oracle.com/cd/E19253-01/816-5166/6mbb1kqod/index... (unfortunately there's no tag ID's to hash to, so you'll have to [ctrl]+[F] to find the specific section on 'clear')

kogir4• 12 years ago

Sadly, that only clears device errors. Tried it with no change.

simplexion• 12 years ago

Reading this just makes me think of this http://www.smbitjournal.com/2014/05/the-cult-of-zfs/

kev0092• 12 years ago

This article is quite ignorant.

ZFS uses integrated software RAID (in the zpool layer) for technical reasons, not merely to "shift administration into the zfs command set". Resilvering, checksumming, scrubs are all a unified concept that are file aware, not merely block aware as nearly every other RAID. The implications of this are massive and if you don't understand, please don't write an article on file systems.

For various reasons, the snapshots and volume management are more usable due to the integrated design and CoW, and also as pillars for ZFS send/receive.

The "write hole" piece is bullshit. ZFS is an atomic file system. It has no vulnerability to a write hole.

The "file system check" piece is bullshit. Again, ZFS is an atomic file system. The ZIL is played back on a crash to catch up sync writes. A scrub is not necessary after a hard crash.

Quite frankly, for any modern RAID you probably should be using ZFS unless you are a skilled systems engineer and are balancing some kind of trade off (stacking on higher level FS like gluster/ceph, fail in place, object storage, etc). You should even use ZFS on single drive systems for checksumming and CoW, and the new possibilities for system management with concepts like boot environments that let you roll back failed upgrades.

Hardware RAID controller quality isn't spectacular, and the author clearly has not looked at the drivers to dish out such bad advice. You want FOSS kernel programmers handling as much of your data integrity as possible, not corporate firmware and driver developers that cycle entirely every 2 years (LSI/Avago). And there's effectively one vendor left, LSI/Avago, that makes the RAID controllers used in the enterprise.

ZFS is production grade on Linux. Btrfs will be ready in 2 years, said everyone since it's inception and every 2 years thereafter. It's a pretty risky option right now, but when it's ready it delivers the same features the author tries bizarrely to dismiss in his article. ZFS is the best and safest route for RAID storage in 2014 and will remain such for at least "2 years".

newman314• 12 years ago

For people interested in a ZFS build, there is zfsbuild.com

smilliken• 12 years ago

We've been running ZFS on Linux in production at MixRank for 6 months. We push it hard and haven't had any issues yet. We're using arrays of 18x Intel 530 240GB SSDs in raidz2. It's performing very well: 697k read, 260k write IOPS; 3768 MB/s read, 1614 MB/s write throughput.

I recommend it, but only after you've done enough research to feel comfortable with all of ways to configure it and understand the internals from a high level. For a personal setup though, you can just install it and forget it.

bruce_one• 12 years ago

Has anyone tried HAMMER (a DragonFly BSD creation) and have any thoughts on it compared to ZFS?

techdragon• 12 years ago

HAMMER has great potential... but its utterly devoid of the momentum required to be anything but a toy at this stage. I am a huge fan of where the developers want to take HAMMER, but I can't run it on any of my hardware to help so I'm just stuck waiting to see progress.

_delirium3• 12 years ago

HAMMER is currently in the middle of a major rewrite to HAMMER2. That's supposed to support multi-master cluster usage, in addition to ZFS/btrfs-like features, so is quite ambitious. The new version is the main focus of full-time development, but not yet considered production-ready. I think the old version is stable-ish but was considered a dead-end by the main developer, so it doesn't have much deployment or development. I would consider the project research-stage at this point, though it's quite interesting research.

rem0x4• 12 years ago

... if you're going to use ZFS, you should really think about using FreeBSD. Open Solaris would be the obvious choice, but with solaris's lack of support, FreeBSD is second to none in ZFS support...

71 TiB DIY NAS Based on ZFS on Linux

Related Stories