Most UI applications are broken real-time applications

by todsacerdoti • 3 years ago

178 148 3 years ago

smokel2• 3 years ago

I like this perspective.

In the demoscene days, we used to obsess over 60 Hz guaranteed framerates. Code was constantly being profiled using raster bars. Our effects might have been crappy, but all was buttery smooth.

Then someone decided it would be a good idea to create a common user interface that would work on many different framerates and resolutions, and all was lost. Or people would try for complex 3D graphics and sacrifice smoothness.

Some people tried to convince us that humans have a 150ms built-in visual processing delay, and all would be fine. This is not the case, and we are still stuck with mediocre animations. The content has improved a lot though :)

TacticalCoder• 3 years ago

Code was constantly being profiled using raster bars.

For those wondering: we'd change the background color from, say, black to blue for the "physics" part, to green for the "drawing" part, to purple for the "audio rendering" part... All in the same frame, while the frame was being drawn.

So you'd see on the border of your screen (usually outside of the drawing area you'd have access to) approximately which percentage of each frame was eaten by "physics", graphics, audio etc. and how close you were to missing a frame.

And it was quite "hectic" typically these color "raster" bars would jump around quite some from one frame to another.

But yeah there was indeed something very special about having a pixel-perfect scrolling for a 2D game running precisely at the refresh rate (50 Hz or 60 Hz back then). And it wasn't just scrolling: games running on fixed hardware often had characters movement set specifically in "pixels per frame". It felt so smooth.

Something has definitely been lost when the shift to 3D games happened: even playing, say, Counter-Strike at 99 fps (was it even doing that? I don't remember) wasn't the same.

People don't understand that their 144 Hz monitor running a 3D game still doesn't convey that smoothness that my vintage arcade cab does when running an old 2D game with pixel-perfect scrolling while never skipping a frame.

masfuerte2• 3 years ago

Not just that. You hit a key and the game could react in literally the next frame. There was an immediacy you don't get now.

mrob2• 3 years ago

A lot of that smoothness was the lack of sample-and-hold blur on CRTs. Gaming LCDs now have modes that strobe the backlight to simulate this effect. Additionally, the importance of sample-and-hold blur declines as the frame rate increases. A 360Hz LCD displaying 360fps motion can look reasonably sharp even without strobing.

Blur Busters has detailed explanations, e.g.:

https://blurbusters.com/blur-busters-law-amazing-journey-to-...

orbital-decay• 3 years ago

People don't understand that their 144 Hz monitor running a 3D game still doesn't convey that smoothness that my vintage arcade cab does when running an old 2D game with pixel-perfect scrolling while never skipping a frame.

The actual motion-to-photon delay (measurable with a high-speed camera) in a well optimized competitive FPS is somewhere around 20ish ms or less these days, so it's on par with it in regards to the input lag, and overall it's far smoother because of the higher frame rate, better displays, and way more thought put into tiny nuances competitive players complain about. A proper setup feels extremely smooth, responsive, and immediate.

proc02• 3 years ago

I think the difference is that 3D is inherently continuous (because it is simulating a 3d space) and requires another layer of abstraction on top of the controllers to work intuitively. 2D on the other hand is pixel based which is usually discrete (the world units are uniform in pixels), and also the character movement is 1:1 mapping on the controller. These two factors make 2D feel more responsive, and I'm guessing that once VR gets better it will close the abstraction gap and make the controller mapping (body/hand movement) 1:1.

squarefoot• 3 years ago

People don't understand that their 144 Hz monitor running a 3D game still doesn't convey that smoothness that my vintage arcade cab does when running an old 2D game with pixel-perfect scrolling while never skipping a frame.

There are great examples even outside of the gaming world. I wish there were videos around on how fluid were Scala Multimedia presentations on a bare Amiga 500 in the 90s. An 8MHz clocked machine (7.16 MHz here in the EU for PAL synchronization) that in that field would outperform gear costing at least an order of magnitude more. It was hugely successful back in the day in many local TV stations.

devwastaken• 3 years ago

At 14 I discovered that tiny response times are essential by observing terminal echos from a TCP client and server connection. I could clearly see a 50ms difference, down to about 15ms if I recall correctly. It wasn't until years later that I discvered high response+refresh monitors and from then on anything under 120FPS is just laggy if it's a game.

m463• 3 years ago

I believe I agree.

The iPhone UI must have either a realtime UI (or something pretty snappy). It responds to finger inputs in a bounded way.

valbaca3• 3 years ago

Dispatch queue. Android has the equivalent thing.

kridsdale3• 3 years ago

More than that even. Since iOS 1, the majority of compute-intense work after a touch event has happened in an entirely different process, which has the highest OS-scheduler priority. This is what CoreAnimation is based on. Unlike a game engine or other "realtime" app, the OS itself is doing everything possible including preempting your main-thread to ensure that anything animating takes priority.

TeMPOraL10• 3 years ago

Dispatch queue. Android has the equivalent thing.

Must be disabled on Samsung flagships, as these are the ones I have experience with, and I wouldn't say touch response is bounded on them, not for any useful definition of "response" and "bounded".

The last mobile device I used that had sane input response was a Sony Ericsson K800i feature phone. Input processing on it was fast, and most importantly, consistent. That means predictable - I could do most stuff on that phone while keeping it in my pocket, because I muscle-memorized the input sequences and how long every action took. Such a thing is impossible on smartphones.

senkora• 3 years ago

As a human, I would like to be able to treat user interfaces as predictable, physical objects.

Unfortunately programs are built out of math and so I can’t usually do this.

Apple iOS actually does do a pretty good job of it, and that is a huge differentiator that I think is a large part of why iPhones feel great to use.

valbaca3• 3 years ago

Thank the dispatch queue and Xcode will yell at you if you do anything other than UI and very trivial computes on the main UI thread

favorited• 3 years ago

Some of Apple's biggest perf wins in the last several years involved removing dispatch asynchrony from code paths that really don't need to be asynchronous.

Obviously if you're writing an event-loop web server, then your file IO needs to be non-blocking or you're hosed. On the other hand, if you're reading a small configuration file from disk after your app launches, the most responsive option may well be to just read the bytes on the current thread and continue.

https://web.archive.org/web/20190606075031/https://twitter.c...

https://web.archive.org/web/20211005132519/https://twitter.c...

tantalor2• 3 years ago

So correct UI applications cannot call any blocking function from their main threads.

Author is redefining "correct" to mean "has the property I care about, which in this case is performance".

There are many desirable distinct properties of a computer system: correctness, performance, security, etc.

"Correct" usually means something like "gives the right answer". It has nothing to do with performance.

cobbal2• 3 years ago

In my experience, "Correct" means "satisfies a specification". The specification is often about the final result, but it's not limited to it.

cobbal2• 3 years ago

As an example, I would call a function called "mergeSort" that takes O(n^2) time incorrect.

anonymoushn3• 3 years ago

For a game like Street Fighter, you can present to someone the time at which the first frame begins and all of the input events within the game and they can calculate the unique game outcome that results. It's a bit embarrassing that we cannot have the same property in applications that we interact with using our keyboards all day, and we must constantly look at them to see if they are behaving as if we have sent the inputs we have sent.

valbaca3• 3 years ago

when I realized that most mainstream desktop UI applications were fundamentally broken.

Oh boy

File system IO functions belong to a class of functions called blocking functions.

Hasn’t non-blocking IO been a major feature for about a decade now??

deepspace• 3 years ago

Someone has not told Microsoft | It is not as simple as that.

On some Windows machines with network-mounted drives, the File-Print-to-pdf dialog takes *minutes* to become responsive, even when all currently open files are on a local drive.

This is the kind of thing the author is talking about. The programmers of that dialog box probably just called a generic "open file dialog" library function, without researching its worst-case performance.

In turn the library writers probably blithely coded something like "check if all mounted drives are accessible", without stopping to consider the worst-case performance.

pdonis• 3 years ago

Someone has not told Microsoft

More like the part of Microsoft that implemented non-blocking I/O for Windows some time ago never bothered to tell the part of Microsoft that writes the generic Windows UI code for things like the open file dialog. Or for Microsoft Office applications, for that matter; I still see Word and Excel block the UI thread when opening a file from a network drive, even though Windows has perfectly good asynchronous file I/O API calls.

pradn2• 3 years ago

The file picker on Windows should be thought of as more like a process than a function call. It has to 1) read the metadata of the current directory’s files 2) look up which icon to use for each 3) load the icon 3) call apps that may modify the file icon overlay (like “synced” or “not synced” cloud icons for Cloud file systems like Google Drive) 4) read the registry for various settings 5) load which pinned files a user set up - etc etc. All this involves dozens of disk reads, calling into third party code, etc. A lot of this may be cached but who knows.

The alternative to this is to roll a barebones file picker - there might even be one available in the Windows API.

anonymoushn3• 3 years ago

Hasn’t non-blocking IO been a major feature for about a decade now??

This doesn't make people use it, and even the ones that try to use it might erroneously expect that open(2) will return in less than a second.

yetanotherloss• 3 years ago

More like three decades. But you'll need to send the memo to small time players like Microsoft, who continue doing all kinds of stupid blocking IO on every possible user interaction because who even fucking knows.

clumsysmurf• 3 years ago

If this article is true, I wonder if newer frameworks like Android's Compose are fundamentally flawed, at least when used in languages with garbage collection.

Essentially, composable can recompose with every frame, like for an animation. But, in certain circumstances, this will cause allocations for every frame.

For example, a modifier that is scoped to something like BoxScope. You can't hoist it out of the composable, it has to be in BoxScope. If that scope is animated on every frame, that modifier get re-allocated, every frame. That could be a lot of pressure on the GC.

Edit: Then again, its hard doing anything realtime in GC languages like Java / Kotlin, maybe its possible if doing 0 allocations per event.

rcme• 3 years ago

Yes, these frameworks are fundamentally broken. Even a framework with extensive usage like React doesn’t work for real-time applications. Or at least your only option is to manipulate the DOM directly for the performant real time parts.

PaulKeeble2• 3 years ago

You do not want real time systems design for a GUI, its a lot more work and it wastes the vast majority of processing power to guarantee performance. You just need to be a careful about the amount of work you do in event handlers and split off heavy or IO work and split big updates. The problem is all concurrency is a lot harder than serial and so it tends to be something that is delayed until its really necessary so a lot of "not really good enough" handlers end up causing minor annoying delays.

gpm• 3 years ago

You don't want to do the design work, but you do want the results.

Maybe some sort of magical framework will figure out how to get the latter without the work in the future.

seanalltogether• 3 years ago

Even if you rebuilt the the entire OS and libraries on a separation of sync vs async code, you still come to the inevitable problem of propagating delays to the user in a predictable manner.

So I've pressed a button to load a file that's usually really fast and this time nothing happens because the async call is taking it's time. Is that being represented to the user in some meaningful way? Is the button disabled until the operation completes? Is it stuck in the down position to show I can't press it again, or does it spring back up and show a loading dialog? Is that any more clear in a real time system then it would be in a blocking situation?

The problem is not blocking vs realtime, it's about programmers not understanding all the state their code can run in, and it's not clear that a realtime system would save the user when they fall into that unknown state.

_y5hn6• 3 years ago

UI was pretty much solved in the 90's including the problem of instant feedback. Then abandoned for shiny stuff.

Personally I set up i3 to open most windows asynchronously, so my flow isn't interrupted. It's great, but takes a bit getting used to windows not randomly stealing focus. It's not for everyone though.

dgan• 3 years ago

You are replying out of context: if your file is on a network drive it doesn't matter if you have shiny UI, or text based terminal, you gonna wait for the round trip with your UI being unresponsive

dagenleg• 3 years ago

Sounds great, how did you do that? What's the config option?

OliverGilan• 3 years ago

Can you be more specific about what was “solved” in the 90s? The platforms upon which apps are run now and the technologies as well as the expected capabilities of those apps have drastically changed. Not all UI development has changed just for being shiny.

I see little reason why building a UI today cannot be a superset of what was solved in the 90s so I’m curious to know what that solved subset looks like to you

bhdlr• 3 years ago

solved in the 90s

Ya, you know, unless you didn't speak English, needed accessibility, had a nonstandard screen size, had a touch screen, etc

DiggyJohnson• 3 years ago

My biggest issue with i3 (which I generally love and have used for 5+ years) is that if I switch to a workspace, launch an application with dmenu, and switch to a different workspace before the application loads, the application will load on the current workspace instead of the workspace I originally launched the application from. Anyone have a solution?

anonzzzies• 3 years ago

Anything stealing focus should be a crime. It's the reason I use i3 as well. When I use Mac OS X sometimes, things just popping to the front to offer updates or whatever makes me want to just drop the laptop in the thrashcan. Who comes up with crap like that?

4rt• 3 years ago

Exactly this, doing it right is possible but vastly more complicated and requires tons more UI work.

It's not impossible - tape drives jukeboxes with 5 minute delays aren't a new invention.

nicoburns2• 3 years ago

An asynchronous action should always show immediate feedback (button disabled + a loading spinner or similar). You should not be assuming that the potentially slow operation will be fast.

dzaima2• 3 years ago

The reality is that optimizing for the common scenario just gives such significant benefits that the worst-case scenario becomes practically infeasible to even consider.

And it's not just OSes that don't particularly care about real-time - modern processors don't either. e.g., in the common scenarios it's possible to load maybe eight 8-byte values per nanosecond (maybe eight 32-byte values if you're doing SIMD), but if they're out of cache, it could take hundreds of nanoseconds per byte. Many branches will be predicted and thus not cost affect latency or throughput at all, but some will be mispredicted & delay execution by a dozen or so nanoseconds. On older processors, a float add could take magnitudes more time if it hit subnormals.

If you managed to figure out the worst-case timings of everything on modern processors, you'd end up at pre-2000s speeds.

And virtual memory isn't even the worst OS-side thing - disk speed is, like, somewhat bounded. But the number of processes running in parallel to your app is not, so if there are a thousand processes that want 100% of all CPU cores, your UI app will necessarily be able to utilize only 0.1% of the CPU. [edit note: the original article has been edited to have a "Real-time Scheduling" section, but didn't originally]

So, to get real-time UIs you'd need to: 1. revert CPUs to pre-2000s speeds; 2. write the apps to target such; 3. disable ability to run multiple applications at the same time. Noone's gonna use that.

anonymoushn3• 3 years ago

I feel like this comment is from an alternate reality where the Windows lock screen doesn't just drop all your inputs for a random period of at least a few seconds when you try to enter your password.

dzaima2• 3 years ago

It is very much true that there's a very large amount of things that could be improved on the status quo with rather minimal amounts of effort, that could resolve a vast majority of the brokenness of responsiveness of UIs. But decreasing dropped frames/inputs by 10x or 100x is still just simply not gonna make it "real-time".

I remember the windows lock screen being particularly annoying (an animation had to finish..?) but on Linux Mint I've always been able to start typing away immediately, so much so that I start blindly typing in the password before my monitor has finished turning on. Properly queueing events (keyboard ones at least) should be pretty simple to do properly, but of course many things still get it wrong.

jonathanlydall• 3 years ago

I’m a Windows fan, but this bugs me so much, prior to Windows 8 I could go Ctrl Alt Del and instantly start typing my password, now I have to wait a couple of seconds before entering being able to enter my PIN.

dTP90pN• 3 years ago

To be fair, SDDM has the same issue, at least on multi monitor setups.

modeless• 3 years ago

Ooh I know how to fix this one! gpedit.msc > Computer Configuration > Administrative Templates > Control Panel > Personalization > Do not display the lock screen > Enabled

Yes, the screen still locks, it just doesn't eat your keystrokes anymore. If you have Windows Home edition without gpedit.msc there's a registry key you can set instead, you'll have to google it.

jacobsenscott• 3 years ago

It is well known that general purpose operating systems do not allow real time applications. The reason is people prefer an OS that can do a lot of things well enough, vs a system that can do one thing in real time. Real time OS's exist, and you would never want to build a desktop environment on them.

Also, the OS will never become the bottleneck for any general purpose software because you'll never have a team of programmers good enough to make it so. All the performance issues be due to mistakes in the application itself.

btilly• 3 years ago

The article fails to distinguish between hard real-time and soft real-time.

If you're controlling a bandsaw, you've got a hard real-time application. You can't miss your window for the next instruction.

Most user interfaces are soft real-time. Occasionally missing your window is fine. And so it is OK to do things like hash lookups whose average performance is O(1), and whose worst case performance is O(n). Ditto for dynamically resizing arrays. As long as you're back in time, most of the time, it can be OK.

The problem isn't that we code UIs as soft real-time. It is that the soft bit gets squishier with each layer of abstraction. And there is a feedback loop where people get more and more used to slow applications, so nobody is concerned if their application is unnecessarily slow as well.

Compounding this is the fact that we often measure performance in terms of throughput, instead of latency. Therefore, every single device and peripheral is willing to lose a bit of latency. It happens at all levels. The classic I like to quote is http://www.stuartcheshire.org/rants/latency.html.

And the result is that modern applications on modern hardware are less responsive than older applications on older hardware. Fixing it is a question of fixing a lot of little problems. And, as we used to joke about Microsoft, what's the point of developing fault-tolerant software when we've already developed fault-tolerant users?

Groxx2• 3 years ago

Agreed here. While I wish more frameworks did what Android did, and allow you to literally block file and network access on the main thread (opt-in strict mode that I wish was default enabled)... worrying about paged memory in an active GUI is so, so much further down the hard-real-time branch of things that the effort/reward ratio is literally insane in almost all cases. It may be a fun challenge for some, but it'll absolutely never be a major use case, nor should it be.

theragra• 3 years ago

Im happy even with JetBrains products. They display "working" popup on blocking operations, and do a lot of stuff in the background, which is not blocking my work.

rootw0rm2• 3 years ago

I've finally learned to love VS Code, but the new UI of Clion is pretty damn decent

ComputerGuru2• 3 years ago

I agree with the main gist of the article and a lot of the points, but disagree on a few specifics.

1) mlock isn’t meant to be called on all your memory, just something that needs it to operate correctly. The situation the author described where the system comes to a halt as memory contents are paged in and out of memory/disk is (subjectively) worse when the only option is for the OOM killer to begin reaping processes everywhere (which would happen if all apps took this advice).

2) the following excerpt is not how any sane modern OS scheduler works:

Imagine you have multiple background process running at 100% CPU, then a UI event comes in to the active UI application. The operating system may block for 100ms * N before allowing the UI application to process the event, where N is the number of competing background processes, potentially causing a delayed response to the user that violates the real-time constraint

Modern schedulers calculate priority based off whether the process/thread yielded its remaining execution time in the previous round or was forcibly evicted. Background processes are additionally run at a penalty. The foreground window (on OSes with internal knowledge of such a thing) or terminal owner in the current login session group gets a priority boost. Threads blocked waiting for input events get a massive priority boost.

(But the point stands and it might be a whole lot longer than n * 100ms if drivers or kernel modules are doing stuff.)

frostiness• 3 years ago

The situation the author described where the system comes to a halt as memory contents are paged in and out of memory/disk is (subjectively) worse when the only option is for the OOM killer to begin reaping processes everywhere.

This one's interesting since your outcome often depends on what hardware you have. On systems with slow IO, i.e. a slow HDD, it's possible for swapping to make a system entirely unusable for minutes, whereas if swap is disabled the OOM killer is able to kick in and solve the issue in less than a minute. That's the difference between being able to keep most of your work open and none of your work open (because the alternative is being forced to reboot).

ComputerGuru2• 3 years ago

But in the era when spinning rust startup disks were in use everywhere, no app would autosave. I can’t imagine the carnage if MS Word or Excel were just violently killed at the first sign of memory pressure back in the day.

kaba06• 3 years ago

The “funny” thing is that I can still regularly hit the “sweet” spot where Linux completely freezes with 8GB of rams (with zram) on a fast SSD.. I think the paging logic still has some assumptions that are only true for hard drives, and that put the system into a frenzy - only REISUB works at that point.

deepsun2• 3 years ago

I believe it's pretty easy to resolve -- declare one thread as "UI Thread", and forbid doing any IO on it.

Android (and I believe iOS does too) enforce that.

It's up to the developer to show a meaningful message/animation if an IO operation takes noticeable time.

kudokatz2• 3 years ago

Android (and I believe iOS does too) enforce that.

This is absolutely not true. Android simply detects that there has been no progress on the UI thread "for a few seconds" before force-closing the app [1]. By this time, the interaction has been janky/frozen for WAY too long. If you have seen bad iOS scrolling and lock-ups, you know this as well.

I have worked on mobile software for these apps that have billions of users. When I pointed out how much stuff ran on the UI thread, there was a collective "this is just the way it is" response and life went on.

It's super-depressing.

-----

[1] "Performing long operations in the UI thread, such as network access or database queries, blocks the whole UI. When the thread is blocked, no events can be dispatched, including drawing events.

From the user's perspective, the application appears to hang. Even worse, if the UI thread is blocked for more than a few seconds, the user is presented with the "application not responding" (ANR) dialog."

https://developer.android.com/guide/components/processes-and...

deepsun2• 3 years ago

Interesting, probably my IDE (Android Studio) only warned me about that, but I took the warning for an error and never even tried to do the wrong way.

pgs• 3 years ago

Attempting a network operation on the Android UI thread will throw the android.os.NetworkOnMainThreadException exception, and has done since API 11 (Honeycomb) in 2011.

https://developer.android.com/reference/android/os/NetworkOn...

Too• 3 years ago

Correct. Looking at the Windows side of things, most of the UWP APIs are designed to be async, meaning they will never block the UI thread.

The author is giving the OS a unfair rep here. All three major OS has solutions to this problem. It's more up to application makers to use the tools correctly.

_greim_• 3 years ago

Web browsers should add a DOM method called `addEventStream()` to supplement `addEventListener()`. It would still accept an event type—e.g. `button.addEventStream('click')`—but would not accept a handler function. It would just return an async iterator of event objects. Backpressure on the iterator would place the element into a `:waiting` state similar to `:disabled` in which the element would become non-interactive. All UI event handling becomes a data processing pipeline.

adroitboss• 3 years ago

So is the moral of the story is to build an entire OS like erlang's actor model? Doesn't that work by using lightweight threads and messaging passing between them. The supervisor gives each thread a certain amount of time to run and then puts the current thread on the back burner while the next thread runs for it's allotted time. I remember hearing it in an erlang talk by Joe Armstrong on Youtube. I can't remember which one though.

amelius2• 3 years ago

Sounds like you'd end up with a microservices architecture. Not a fan.

moritzwarhier• 3 years ago

I felt like by reading the headline alone, I know exactly what this is talking about

Edit: I was completely wrong, but the article is very interesting.

nitwit005• 3 years ago

This is a transparent process that is not under control of the application. Thus, if any given memory access can block on IO from a disk drive, that means the system is fundamentally not real-time, therefore UI applications on such a system are fundamentally broken.

Not really. It means they chose to degrade performance when too much memory is used, rather than crash.

All options available when "out of memory" are bad. They thought that was the least bad option.

brundolf7• 3 years ago

One of the fundamental problems is that many UI applications on Windows, Linux, and macOS call functions that are not specified to run in a bounded amount of time. Here’s a basic example: many applications don’t think twice about doing file IO in a UI event handler. That results in a tolerable amount of latency most of the time on standard disk drives but what if the file is stored on a network drive? It could take much longer than a second to service the file request. This will result in a temporarily hung application with the user not knowing what is happening. The network drive is operating correctly, the UI application isn’t.

Say what you will about JavaScript, but a great thing it's done for modern software is making async programming default and ergonomic. With most APIs, you couldn't block on IO if you tried. Which means web UIs never do

taeric4• 3 years ago

Huh? It used to be very common to find that a web page was stuck processing some script. Still isn't that uncommon, all told.

You are mostly right that you couldn't block on IO, necessarily, but you could still hose up the event thread quite heavily.

brundolf7• 3 years ago

If you're actually doing busy logic then it'll block the UI thread, but you have to work pretty hard to do that, and IO won't block the UI thread

RecycledEle2• 3 years ago

Yes.

UI should be run in a separate processor in real time.

I am tired of clicking to see the screen change after I click and the click registers on the new screen, not what I clicked on.

ReactiveJelly• 3 years ago

Total overkill. If the screen loading is slower than a frame because of...

- I/O like file access or networks, you can punt to an async framework like Rust's Tokio

- CPU number crunching like cracking a password, you can punt to a worker thread (like Tokio's `spawn_blocking`)

The kernel scheduler is smart enough to give time slices to the UI thread even if the password-cracking thread is trying to eat as much CPU as possible, or, God forbid, the network thread is _sleeping_ until a packet arrives. (Networking is a waste of a thread. Most requests and responses are small enough that the CPU could process them on the UI thread, it's just that sleeping the whole thread until the message shows up is goofy)

It's not a lack of processors, it's the fact that good multi-threading and UI totally changes the architecture of a program, and most of us aren't trained for it or incentivized to implement it.

RecycledEle2• 3 years ago

I disagree that a dedicated I/O processor is overkill. We spend a lot of transistors and code to solve problems that occur very rarely. Obscure race conditions come to mind. Putting the UI in real time would save me hassle every day, so it's worth the effort.

Adding USB ports to a video card would achieve 99% of the hardware requirements. The other 1% is made up of things I have overlooked.

The software would be more complicated, but it would not require rewriting applications. It would mostly be recoding OSes.

Also, the security benefits of linking output the screen to input from the keyboard and mouse in real time would be substantial.

perryizgr8• 3 years ago

This doesn’t seem like a common problem but whole system “out of memory” conditions are not that uncommon. When the system is in this state, it starts rapidly paging memory onto the hard disk. UI applications will be affected and this will cause your system to hang without warning and with no way to intervene since keypresses cannot be processed. From a user standpoint, this is worse than a kernel panic. This type of failure has happened to me multiple times on Linux so I know it’s a problem there. Perhaps Windows and macOS engineers have already considered this issue but I doubt it.

In my experience this is only a problem on Linux. Windows and mac will simply inform you that the system is running out of memory and then you can kill things to set it right. No hung system.

zubairq• 3 years ago

Is this article another way of saying that the user doesn't like to wait more than 100ms when they perform an action?

Animats2• 3 years ago

I miss running Photon on QNX, which, being a hard real-time OS, did have upper bounds on timing for many operations. Photon was a GUI for near real time interfaces for hard real time programs. It was far smoother than what we see today.

* No paging. Entire program in memory at all times. Although it was possible to put a paging library inside an application and let it manage its own paging, which was done for gcc.

* A real time CPU dispatcher. Real time priorities were strictly preemptive. Unblock a higher priority task, it starts now, not when the dispatcher gets around to it.

* The usual test for a hard real time OS is that you have an interrupt routine that senses an external pin. It unblock a user level task when the pin goes high. The user level task turns on an output pin. You hook up a scope and a square wave generator, and watch the latency. If there are outlier values on the scope, something is broken. One implication is that stuff below the OS, such as anything running in system management mode after boot, has to be eliminated. It cannot be allowed to steal cycles.

* While real time work is going on, you can still run compiles and web browsers. They get preempted. I was impressed that that worked. Our real time application had a hardware stall timer, and if the commands coming out were late, a relay tripped and everything shut down.

So, it's absolutely possible to do this. Downsides:

* You're way out of the mainstream.

* You're going through the general case every time. Fast paths are unwanted.

* Everything is always on. Power consumption is constant. No CPU slowdowns, no sleep modes, no battery saving. This is just fine when you're running a rolling mill, and 99+% of the power is driving the heavy equipment. Not so good when you're running a credit card terminal. Battery powered devices cannot run well in this mode.

TeMPOraL10• 3 years ago

Everything is always on. Power consumption is constant. No CPU slowdowns, no sleep modes, no battery saving. This is just fine when you're running a rolling mill, and 99+% of the power is driving the heavy equipment. Not so good when you're running a credit card terminal. Battery powered devices cannot run well in this mode.

Is there no possibility for compromise here?

A credit card terminal sits idle 99%+ of the time, does little actual work, and whatever work it actually does, it's in a reaction to an input event (such as user pressing a button, or a command being sent from cash register). CPU power states are, to my understanding, something you can switch between many times in a fraction of a second. Outside of talking to the network, duration of which is by nature unpredictable, everything else falls into two modes - "slow mode" for listening to events (and perhaps doing whatever is needed to keep the network connection alive), and "fast mode" for processing those events. Sounds to me you could put a real-time guarantee on everything except the networking parts, and keep the CPU slow/sleeping for most of the time.

antipurist• 3 years ago

Everything is always on. Power consumption is constant.

That was the case in ye olden days of single-core CPUs, but is it still true today?

I can easily imagine a computer that has one always-on core for kernel chores and hardware management, and a dozen dormant cores that wake up when userland tasks need those cycles, all running in hard real time. Or even better, a mix of real-time for UI & audio on some cores + non-time-bound jobs on others.

Animats2• 3 years ago

There's the problem of what does the power management, shut down, and start up of dormant parts of the CPU. If that steals cycles from the OS, it can interfere with real time processing.

There's no reason that CPUs couldn't be designed to avoid such interference, but don't expect to find that on x86. Not sure about ARM.

rerdavies3• 3 years ago

Visual Studio Code has a huge realtime problem. I find myself typing ahead by ten or 15 characters most of the time before changes show up in the UI. At which point either my brain-typeahead buffer overflows, or keystrokes start getting dropped (I'm not sure which).

Not sure if this is a problem specifically on Linux or not. I like to think that on Windows, UI-thread/process priority boosts take care of this problem, but I suspect not entirely.

I've been meaning to turn off Intellisense to see if my productivity will increase. What I know for sure: my productivity in VSCode/C++ is hugely tanked compared to Android Studio/java.

pjmlp• 3 years ago

Or switch to something not built on top of Electron, like Sublime.

tiborsaas• 3 years ago

That's extremely slow it's in millisecs for me, how can you even work with that latency? I think it could be a graphics driver issue, I notice VSCode lagging sometimes when I have some WebGL app running next to it.

rerdavies3• 3 years ago

We have UI models that allow us to through long-running operations onto a background thread so that we can render perfectly animated hypno-toad cursors Or, more currently, perfectly animated cute-wait animations to keep people entertained while long-running operations complete. (Only a matter of time before wait-animations incorporate mini-games to play while while you're waiting, I think). But...

I'm not sure we have UI models for dealing with more complicated asynchronous operations on a document model though.

Old versions of Microsoft word used to run pagination and line-breaking on a background thread. The UI would keep a couple of lines around the edit cursor up-to-date, with precisely page layout for anything more than a couple of lines after the edit cursor running asynchronously. (It probably still does, but we probably notice it less these days).

And various elaborate schemes for keeping local and cloud object models synchronized asynchronously.

Oh. And I guess we have compilers and editors that collaborate to take snapshots of code when a compile starts, while allowing editing to continue, and to run Intellisense analysis in the background in the face of continually updating edit buffers. Incredibly complicated stuff, done mostly by brute force of intellect, I think.

Beyond that, I don't think we have a general theory of what to do in user interfaces when there isn't enough CPU to go around.

ponyous• 3 years ago

Windows Phone 7 fixed this in a way - it had a UI thread that could not be used for anything but UI updates. Your event handlers would be executed on a different thread. As long as you showed the loader, the rest of the app would work perfectly smooth.

It's been few years since then so I am light on details.

---

Also, as much as devs hate javascript, it mostly solves this by having 1 thread and offloading all system calls to another via callbacks? So the solution to your issue is: Use electron. (Just kidding, but there is some truth in it)

iamnotsure• 3 years ago

Yet another reason to ditch GC based languages by default.

Most UI applications are broken real-time applications

Related Stories