Google Compute Engine MPI Latency

http://blog.rescale.com/mpi-latency-on-google-compute-engine/

by gpoort • 13 years ago

37 13 13 years ago

blog.rescale.com

montecarl5• 13 years ago

This isn't that interesting or surprising. 100-200 microseconds is the latency of all ethernet that I have ever seen. Infiniband or other high performance networks can achieve about 10-100x lower latency, but are very expensive. Infiniband switches and cards can be double the cost of a cluster.

sargun• 13 years ago

One of the things that Google is also working on is PCIe switching, and using that to get rid of having to the ethernet encapsulation and conversion. This allows for significantly lower latencies <5 μs.

montecarl5• 13 years ago

That sounds awesome. Do you have any links to describe this research? I would be interested in the technology if it stands to lower the cost of low latency networks.

gpoort• 13 years ago

AWS is still using ethernet (10 Gigabit on the highest end) and can achieve ~2x better performance

montecarl5• 13 years ago

I have no experience with 10 gigabit ethernet. However, it seems that there are some lower latency (down to sub millisecond) switching fabrics.

wmf2• 13 years ago

Just for comparison, on non-virtualized 10GbE I think we got ~30 us. I would guess the extra latency in the cloud is caused by the hypervisor.

Also, last I checked 16 Gbps Infiniband was cheaper than 10 Gbps Ethernet but people don't want to use verbs.

rb12345• 13 years ago

This is slightly off-topic, but...

There's plenty of references to physical Infiniband cards being allocated to a guest VM. What I haven't seen though is a hypervisor-level Infiniband vswitch. I wonder if that could help with some of the latency issues; for example, inter-VM memory access could be done via host-level shared memory. Ideally the vswitch could handle inter-host links too, saving the need for manual assignment of cards to VMs.

wmf2• 13 years ago

Mellanox did some experiments a few years ago that showed that DMAing to the NIC and back was actually faster than going through dom0 in software. Things may have changed since then, although it's probably irrelevant since the people who want to use RDMA and the people who want to use VMs don't overlap much.

codemac2• 13 years ago

To me this is where the "software defined networking" type of virtualization can really make an impact.

The network performance of a known cluster of virtualized instances could be extremely quick if you just lie, and say your packet went through a network, when really you just pass a pointer in the hypervisor..

I assume this has already been done, but at almost 200 microseconds, you know it hasn't been done in these experiments.

montecarl5• 13 years ago

That is only the case for networking inside of a single physical machine. Most HPC MPI use cases span many machines, for which this is typical latency of ethernet.

codemac2• 13 years ago

In cases where I've used OpenMPI it was spanning machines as well as being multiple processes on the same box. The goal was making that interchangeable (in my use case).

I imagine google's compute engine isn't all on separate machines, but utilizes VMs heavily.. although it's probably a bad idea to put one customer's VM's on all the same box.

That's a long way of saying "of course you're right, I guess my thought doesn't contribute as much as I thought" :)

ori_b• 13 years ago

That's an interesting hack for MPI, but I don't think it would be worth the complexity of implementing it. There are two possible use cases here:

One: You're testing multiple VMs on one box to simulate a network. This is a sane use case, but a bit of overhead from network latency isn't going to kill you. It definitely isn't worth the effort of messing around with faking your network.

Two: You're trying to wring whatever performance you can out of a single box. In this case, you can ditch the VMs, and just run your MPI nodes as threads or processes on the local machine, using far faster and more efficient transports than TCP.

Also, in this specific case, note that the instances are not necessarily on the same machine.

Google Compute Engine MPI Latency

Related Stories