bbjnicklin

Michaelg,

Regarding the network, the paths are different in a meaningful way, you might want to trace the paths for yourself to see how: https://upload.wikimedia.org/wikipedia/commons/3/37/Netfilte.... Specifically, with IPSec, you make another pass through the network layer, after hitting XFRM in the protocol layer. This is not the case for a normal flow. This “loop” is significant because of the frequency with which it happens. Also, as the post states, random memory accesses are the killer. This additional loop and the logic within exacerbate the problem.

Also, If you’ve been using IPsec in production for multiple years, it’s possible that you using non-AESNI optimized ciphers, so the speedup could in fact be MUCH greater. If you can share a bit about your specific deployment, we would be happy to provide guidance.

theonewolf, to the best our knowledge, we are the first to offer TLS protected iSCSI (a block storage transport). As we said in the post, SSL was considered back when the iSCSI protocol was developed, but IPsec won out. Many people use TLS today for object storage (ie. S3, Kinetic, etc.). That said, this post is less of a “pitch” and more of a “here’s what's possible with modern technology”. Regarding NASD, If I recall correctly, that was a file based research project.

BTW, regarding the drivers: none needed, you can do it from userland on any platform with a simple SSL proxy like stunnel. If you want, reach out and we can hook you up with software to play around with.

Feld, you are totally right. In fact, there has been discussion in the Linux community about improving IPsec performance. However, as the post states, the difficulty in doing so is the API contract in the kernel. That said, even with a perfectly optimized IPsec stack, having to process each packet individually will always be slower than processing a logical record that is a multiple of packet size.

Glastra. Maybe this data is a bit better for you, direct from fio.

  TLSv1.2: ecdhe-rsa-aes128-gcm-sha256
  
  QD1: (groupid=0, jobs=1): err= 0: pid=2140: Sat Mar  5 14:59:49 2016
    read : io=2308.5MB, bw=78791KB/s, iops=19697, runt= 30001msec
      slat (usec): min=2, max=21, avg= 2.42, stdev= 0.50
      clat (usec): min=39, max=56179, avg=47.66, stdev=126.45
       lat (usec): min=46, max=56182, avg=50.14, stdev=126.45
      clat percentiles (usec):
       |  1.00th=[   46],  5.00th=[   46], 10.00th=[   46], 20.00th=[   47],
       | 30.00th=[   47], 40.00th=[   47], 50.00th=[   47], 60.00th=[   47],
       | 70.00th=[   48], 80.00th=[   48], 90.00th=[   49], 95.00th=[   49],
       | 99.00th=[   51], 99.50th=[   51], 99.90th=[   53], 99.95th=[   56],
  
  
  IPsec: aes128-gcm96
  
  QD1: (groupid=0, jobs=1): err= 0: pid=2442: Sat Mar  5 15:27:04 2016
    read : io=1902.6MB, bw=64938KB/s, iops=16234, runt= 30001msec
      slat (usec): min=2, max=29, avg= 2.39, stdev= 0.52
      clat (usec): min=52, max=57367, avg=58.51, stdev=140.69
       lat (usec): min=57, max=57369, avg=60.97, stdev=140.69
      clat percentiles (usec):
       |  1.00th=[   56],  5.00th=[   56], 10.00th=[   57], 20.00th=[   57],
       | 30.00th=[   57], 40.00th=[   57], 50.00th=[   58], 60.00th=[   58],
       | 70.00th=[   58], 80.00th=[   59], 90.00th=[   60], 95.00th=[   61],
       | 99.00th=[   63], 99.50th=[   65], 99.90th=[   78], 99.95th=[   82],