Intel's Storage Acceleration Library

https://github.com/01org/isa-l

by sharva • 10 years ago

30 6 10 years ago

github.com

jerrysievert• 10 years ago

it would be very helpful if they included a description as to what it is this library does.

byefruit• 10 years ago

https://01.org/intel%C2%AE-storage-acceleration-library-open... has more info

notmyname• 10 years ago

I've used ISA-L as the backend implementation for erasure codes in OpenStack Swift.

There are two versions of ISA-L. The open-source one includes some optimized erasure code libraries. The other version is free (as in beer) AFAICT but it requires signing a license agreement with Intel. This version also includes some other optimized libraries for hashing, crypto, and stuff like that.

https://github.com/openstack/liberasurecode is a plugable C library that you can use for erasure coding. It supports some simple EC algorithms itself and works with jerasure and ISA-L.

https://github.com/openstack/pyeclib is a Python wrapper for liberasurecode.

dragontamer2• 10 years ago

This appears to be an implementation of Reed-solomon error correction code... at the assembly level. (!! handwritten assembly? I don't see that too often nowadays)

https://en.wikipedia.org/wiki/Reed%E2%80%93Solomon_error_cor...

Which is basically the mathematics behind RAID-5, RAID-6, and even hard-drive / SSD sector-level error correction. Parchive is an older open-source project that implements the same concept:

https://en.wikipedia.org/wiki/Parchive

beagle3• 10 years ago

RAID-5 uses simply parity, RAID-6 uses something slightly more complicated, but they are not close in complexity (or capabilities) of Reed-Solomon codes -- although, if there was a Raid Level that had 1024 drives of which any 16 can fail, it would probably employ Reed Solomon codes.

dragontamer2• 10 years ago

You're right about Raid5.

My understanding is that Raid-6 is Reed-solomon however: https://en.wikipedia.org/wiki/Standard_RAID_levels#RAID_6

Several methods, including dual check data computations (parity and Reed-Solomon), orthogonal dual parity check data and diagonal parity, have been used to implement RAID Level 6
if there was a Raid Level that had 1024 drives of which any 16 can fail, it would probably employ Reed Solomon codes.

Reed-Solomon codes are generalized for any number of drives with any number of failures.

There's a bitwise-efficiency thing with powers-of-two as well. I forgot the details, but I did the math back in college. I'll have to find my notes on this subject... but yeah, the matrix works for virtually everything.

I forgot whether or not Reed-Solomon becomes just a simple xor-parity check in the case of 1-failure, but it wouldn't surprise me if it were related.

Intel's Storage Acceleration Library

Related Stories