Linux's fsync() woes are getting some attention

http://rhaas.blogspot.com/2014/03/linuxs-fsync-woes-are-getting-some.html

by r4um • 12 years ago

135 67 12 years ago

rhaas.blogspot.com

bodyfour6• 12 years ago

I think part of the problem is that fsync() is that it's an insufficient interface. Most of the time, you want two things: * write ordering ("write B must hit disk after write A") * notification ("let me know when write A is on disk") In particular, you often don't want to actually force I/O to happen immediately, since for performance reasons it's better for the kernel to buffer as much as it wants. In other words, what you want should be nearly free, but instead you have to do a very expensive operation.

For an example for notification: suppose I have a temporary file with data that is being journaled into a data store. The operation I want to do is: 1. Apply changes to the store 2. Wait until all of those writes hit disk 3. Delete the temporary file I don't care if step 2 takes 5 minutes, nor do I want the kernel to schedule my writes in any particular way. If you implement step 2 as a fsync() (or fdatasync()) you're having a potentially huge impact on I/O throughput. I've seen these frequent fsync()s cause 50x performance drops!

Qantourisc• 12 years ago

Not a kernel dev but to my knowledge "write B must hit disk after write A" would actually be rather easy to implement: if the storage device supports it: insert a write barrier. For the "let me know when write A is on disk" adding a callback to a write-barrier would do, not sure what the kernel queue looks like though.

Filligree2• 12 years ago

In principle, yes. I believe most modern filesystems use (kernel-level) write barriers to ensure filesystem consistency, and those do depend on having hardware barriers as well.

The kernel is not set up to export this interface, however, and there doesn't appear to be any serious work being done to fix that.

EDIT: http://pl.atyp.us/2013-11-fixing-fsync.html has more details.

deathanatos3• 12 years ago

While a "notify me when this write completes" (but don't feel the need to rush) might be useful, I think it'd be more useful to be able to push both write A and write B to the kernel, along with the condition of "A before B". You not only get the write ordering, but the kernel gets full knowledge of what's going on. Perhaps it could use this to write A and B together (but journaled at the FS level) to guarantee the order, whereas a simple write-complete notification implies at least two writes, and the latency of doing such.

mnarayan01• 12 years ago

Is that not the purpose of [aio_fsync](http://pubs.opengroup.org/onlinepubs/009696899/functions/aio...)?

Disclaimer: I've never used it, I just filed it in the back of my mind for if I wanted to do basically what you just described.

justincormack• 12 years ago

Doesn't exist on Linux as a kernel interface, although the API is there; if it works it is either a thread doing fsync or its synchronous.

jcampbell1• 12 years ago

I don't understand, but I am not very knowledgable about this any of this. My understanding is that fsync is an instruction to command the physical disk to flush its write buffer to the non-volatile (platters/flash) portion of storage, so it is safe to yank the cord from the wall.

In your scenario, it seems like you could have a situation where the temp file and the data are sitting on RAM inside the HD.

I honestly don't see how multiple processes that share a single physical disk can't get trashed by a processes that instructs the disk to constantly flush the buffer. It seems like asking for a network interface that doesn't slow down the network due other processes.

nknighthb• 12 years ago

There are multiple levels of caching. The cache in a disk is just one. Filesystem cache is another, which happens in RAM. There may be a layer of caching between filesystem and disk. Applications routinely cache data in their own memory allocations before writing to the filesystem.

fsync lives between applications and the physical disk. Its (intended) effect is to flush dirty data that an application has written to the filesystem all the way to the physical disk.

For maximum performance and minimum fragmentation, you want to wait as long as possible before data is flushed from the filesystem's cache. In a typical storage system, the filesystem is what decides the approximate ordering of writes and the allocation of disk space. Disks have no idea what's going on, they just respond to commands to write certain data to certain locations; their discretion is minimal, at most they will re-order a small batch of writes for optimum head movement.

Delaying a flush as long as possible allows the filesystem to coalesce multiple writes to the same location into one write, allocate disk space in the largest possible chunks in the best locations to avoid fragmentation and excessive seeking, order the writes for rough optimization and for data and metadata consistency.

Caching data in RAM is a good thing.

fsync is what you use when you've got important data that needs to be preserved immediately. Most data on a typical PC or server is just not that important. You're better off letting the filesystem do its job. And there's a whole continuum of data consistency in between, where you might want certain operations to happen in a certain order, or data to be written in a certain order if it's written, even if you don't necessarily need them to end up on the disk right now.

dap8• 12 years ago

In particular, you often don't want to actually force I/O to happen immediately, since for performance reasons it's better for the kernel to buffer as much as it wants.

If you don't issue an fsync(), the kernel never has to write anything. If you had a different function that did what you suggest, returning when the kernel had decided to write the data (without instructing the kernel to do so soonish), you could literally be waiting forever. If the function ever returned, it would only be because something else on the system asked for a sync (and it was easier to write everything), or because you're operating on a filesystem that chooses to write data sooner than required. Both of these are basically working by accident (i.e., may well not work on other POSIX systems).

I think what you really want is to instruct the kernel that this data should be written out, and you want to block until that happens. That's exactly what fsync() is supposed to do. The fact that some kernels hammer the I/O subsystem while doing so is a bug in those kernels, not the fsync() interface.

ryandrake• 12 years ago

Coming from a graphics background, this reminds me of a similar problem in graphics.

In the OpenGL API, you have two functions (hard core graphics pedants, please forgive my simplification): There's glFinish() which instructs the graphics system to process all graphics commands previously sent and to block until everything is done. Then there's glFlush() which returns immediately, and basically says, make sure all commands previously sent will finish "in a finite amount of time" (as opposed to "maybe never").

From the discussion here, it seems that fsync() is like glFinish(). Maybe what's needed would be something similar to glFlush()? Something that says, "kernel, now would be a good time to start writing out data, but I'm not going to wait."

SamReidHughes24• 12 years ago

What bodyfour is proposing is not just getting a write-notification -- it's in tandem with being able to specify write ordering. If you need to be able to specify write ordering, you also want to be able to have a write notification.

fsync does not work because it returns when everything (in the same thread? or whatever) has been written to disk, and doesn't let you wait for a particular block to have been written.

bodyfour6• 12 years ago

you could literally be waiting forever

Technically that's true, but it's not an issue in real life. Any sensible operating system will flush dirty buffers to disk eventually. Imagine if a sudden power outage lost writes you had made a month ago; it would be considered a pretty severe OS error.

So in reality, you might be waiting many seconds for the write to hit disk, but you won't be waiting hours.

That's exactly what fsync() is supposed to do.

No, fsync is "Make sure the last write made it to disk. IN fact, force it out immediately. Also, all of the other pending writes as well, even if I'm not concerned about them. Oh, and block my thread until you're finished." It's an expensive operation.

riffraff• 12 years ago

could you explain what the kernels that "do it right" do differently from those that "hammer the I/O subsystem"?

Not being an expert, it would seem to me there shouldn't be much difference but obviously there is (i.e. TFA also says "linux woes").

sanxiyn• 12 years ago

Featherstitch http://lwn.net/Articles/354861/ is an interface (and an implementation) for write ordering without blocking. As far as I can tell, it went nowhere.

rwmj• 12 years ago

Interesting couple of related articles / rants by Jeff Darcy:

http://pl.atyp.us/2013-08-local-filesystems-suck.html

http://pl.atyp.us/2013-11-fixing-fsync.html

mrottenkolber• 12 years ago

I think this is a very important area of improvements for linux. While we call it "multitasking" there are a lot of situations where one might doubt it deserves that title.

I have been experimenting with very low cost computing setups that optimize for robustness and that led me to pretty slow disk I/O. While thats not a typical scenario for desktop computing, it can and should be possible with the limited but sane resources I ended up with. In practice however, certain loads freeze the whole system until a single usually non-urgent write finishes. Basically the whole throuput is used for a big write and then X (and others) freeze because they are waiting for the filesystem (probably just a stat and similar).

There are differences between applications. Some "behave" worse than others. Some even manage to choke themselves (ever seen GIMP take over an hour to write 4MB to an NFS RAID with 128kb/s throughput?).

I guess this is a hard problem, but I would wish for an OS to never stall on load. It is even better to slow down exponentially than to halt other tasks. Ideally the sytem would be smart and deprioritize long-running tasks so that small, presumably urgent, tasks are impacted as little as possible.

haberman• 12 years ago

It always amazes me that after all these years, Linux still hasn't fixed this.

In my experience, any program that overloads I/O will make the system grind to a halt on Linux. Any notion of graceful degradation is gone and your system just thrashes for a while.

My theory about this has always been that any I/O related to page faults is starved, which means that every process spends its time slice just trying to swap in its program pages (and evicting other programs from the cache, ensuring that the thrashing will continue).

I've never gotten hard data to prove this, and part of me laments that SSDs are "fast enough" that this may never actually get fixed.

Can anyone who knows more about this comment? It seems like a good rule inside Linux would be never to evict pages that are mapped executable if you can help it.

Has anyone experimented with ionice or iotop? http://www.electricmonk.nl/log/2012/07/30/setting-io-priorit...

FigBug3• 12 years ago

Happens to Windows and Mac OS too. Every time I boot Dropbox thrashes my disk for 10 minutes while the system is almost completely unresponsive.

PhantomGremlin• 12 years ago

Yes, OS X isn't very advanced.

In its heyday, Solaris was outstanding in terms of being responsive while simultaneously doing large amounts of I/O. (Or at least that's my perhaps clouded recollection, I haven't used Solaris in over 5 years).

ksk4• 12 years ago

Every time I boot Dropbox thrashes my disk for 10 minutes while the system is almost completely unresponsive.

I seriously doubt that any usermode program could overwhelm the OS scheduler like that. What are your use case parameters?

est• 12 years ago

Hmm. maybe that's why they had so many issues with single instance Redis & Mongodb? As soon as fsync() the whole db became unresponsive.

fiatmoney• 12 years ago

MongoDB's storage engine is more or less a mmap'd linked list of documents. It has a lot of issues once you actually start doing reads or writes, whether it's because your working set exceeds RAM or you actually want durability. It's a nice term-paper DB implementation but there's a good reason why most of the big single-instance RDBMS's use their own I/O algos instead of delegating to the kernel. Fundamentally the kernel doesn't know your optimal access pattern or your desired tradeoffs.

Redis turned away mmap'd storage some time ago; you can snapshot the DB to disk, but that's done via a fork() and the writes happen in that other process.

cratermoon• 12 years ago

O_PONIES ride again?

peapicker• 12 years ago

So, what does Oracle Enterprise Linux do on an Exadata box running Linux?

caf• 12 years ago

ora uses O_DIRECT.

zurn2• 12 years ago

Good to see this summit involving the kernel developers, since their past situation sounds rather bleak interaction-wise: using kernel version from 2009 and haven't tested the improvements in the (2012) 3.2 kernel.

BTW, Linux provides the direct I/O O_DIRECT interface that allows apps to bypass the kernel caching business altogether. This is also discussed in Mel Gorman's message that this blog borrows from.

zvrba2• 12 years ago

Ditch Linux and port PgSQL to run on top of raw Xen interfaces. You get to control your own buffering, worker thread scheduling, and talk directly to the (virtual) disk driver. I believe it'd be a win.

ibotty• 12 years ago

here is a mail from mel gorman with many more details.

http://mid.gmane.org/%3C20140310101537.GC10663%40suse.de%3E

dschiptsov• 12 years ago

Oh yes, that annoying problem (especially fro MongoDB) that data should eventually be committed to a disk.)

Informix (and PostgreSQL) allows DBA to choice "checkpoint/vacuum intervals".

The rule of thumb, unless you are a Mongo fan, is that checkpoints should be performed often enough to not take too long, which depends only on the actual insert/update data flow.

But any real DBA could tell the same - sync quickly, sync often, so server will run smoothly, but not "at web scale" and the pain of recovery will be less severe.)

zvrba2• 12 years ago

Re Mel Gorman's details in http://article.gmane.org/gmane.linux.kernel/1663694

I don't understand why PostgreSQL people don't want to write their own IO scheduler and buffer management. It's not that hard to implement (even a MT IO+BM is not really complicated), and there are major advantages:

- you become truly platform-independent instead of relying on particulars of some kernel [the only thing you need from the OS is some form of O_DIRECT; it exists also on Win32]

- you have total control over buffer memory allocation and IO scheduling

- whatever scheduling and buffer management policy you're using, you can more easily adapt it to SSDs and other storage types, which are still in their infancy (e.g., memristors) [thus not depending on the kernel developers' goodwill]

I mean, really: these pepole have implemented a RDBMS with a bunch of extensions to standard SQL, and IO+buffer management layer is suddenly complicated, or [quote from the link]: "While some database vendors have this option, the Postgres community do not have the resources to implement something of this magnitude."

This smells more like politics than a technical issue.

Linux's fsync() woes are getting some attention

Related Stories