Falsehoods programmers believe about video

https://haasn.xyz/posts/2016-12-25-falsehoods-programmers-believe-about-%5Bvideo-stuff%5D.html

by pomfpomfpomf3 • 10 years ago

250 131 10 years ago

nhaehnle• 10 years ago

This is a good list, but it would be so much better with some (brief) pointers to counter-examples to the beliefs.

unscaled2• 10 years ago

This unfortunately follows the conventions of the genre called "Falsehood programmers believe about X": http://spaceninja.com/2015/12/08/falsehoods-programmers-beli...

I honestly think this genre is horrible and counterproductive, even though the writer's intentions are good. It gives no examples, no explanations, no guidelines for proper implementations - just a list of condescending gotchas, showing off the superior intellect and perception of the author.

kdeldycke2• 10 years ago

Managed to compile a list of these at: https://github.com/kdeldycke/awesome-falsehood#awesome-false...

inopinatus• 10 years ago

Perhaps there is scope for a list of Falsehoods Programmers Believe About Falsehoods Programmers Believe.

icebraining• 10 years ago

It's true that examples and explanations would be nice and make for a more helpful guide, but those can usually be found with a bit of legwork, whereas the gotchas themselves are often only discovered by trial and error. In essence, don't look a gift horse in the mouth.

A better approach would be to pick the list up and turn them into a collaborative work. Wiki, maybe?

TazeTSchnitzel• 10 years ago

Aren't the falsehoods inherently guidelines? They give you an idea of which assumptions aren't safe to make.

simias• 10 years ago

I agree, I think this format works when the subject matter is trivial enough that it's easy to construct counter examples yourself once the contradiction is pointed out.

The "Name" version is a good example of that, I can easily see how most of the examples on this list can be falsehoods.

On the other hand in TFA some of the affirmations leave me more perplexed. For instance, regarding color conversion: "converting from A to B is just the inverse of converting from B to A". I wonder what's meant here. Is it just a matter of rounding or is there more to it than that?

The catch 22 here is that if you understand this list then chances are you already knew about most of these gotchas.

So yeah, a pretty bad format. Now we just have to write "`Falsehood programmers believe about X` considered harmful".

tedunangst• 10 years ago

Just reimagine it as "implicit assumptions to check for". From my limited experience in the field, a lot of these are things I know (or knew) but could easily forget in the midst of trying to get code to work.

imsofuture• 10 years ago

I hate the smug attitude of things like this. I get they're trying to raise awareness of a thing, but maybe take a moment to educate, instead of just smugly dunking on people about how much more you know about a thing that they don't.

smallnamespace3• 10 years ago

This article would be infinitely better if it any provided counterexamples.

tuxidomasx• 10 years ago

This list makes me not want to program any [video stuff]

justinlaster2• 10 years ago

a H.264 hardware decoder can decode all H.264 files

and

video decoding is easily parallelizable

At a previous job, I don't know if it was just the field I was in or just bad luck, but having to explain this over and over again was kind of a personal nightmare.

That being said, this is an excellent list!

wstrange• 10 years ago

Curious - Why is this? Does this assume streaming video, and you can't look ahead in the stream?

If you can jump ahead, it would seem to be easy to have multiple threads, starting at key frames to decode the content. You'd have to splice them together, but this seems possible.

saurik2• 10 years ago

1) You are now assuming that "seeking to a position will produce the same output as decoding to a position"; even if the video is well-formed (and you don't end up with massive issues where the key frames just don't work correctly) you are likely going to end up with subtle discontinuities between every segment. 2) You are now going to have to be buffering a couple seconds worth of uncompressed video somewhere, probably not on the GPU, leading to a much higher I/O bandwidth requirement somewhere that isn't good at that, so this is only probably going to be sort of parallel (FWIW, I believe most people who try to do parallel video decoding are assuming that they can have different parts of the encoder concentrate on different sections of the screen, which sounds good until you see how non-local video decoding can be).

frozenport• 10 years ago

Decoding frames ahead of time gives no benifit to a user watching the video. The problem is how to decode a single frame in parallel. Contrary to the video expressed elsewhere, hardware decoders run a lot in parallel. As MultiCoreWare pointed out, one of the biggest challenges is latency.

jheriko6• 10 years ago

seems to be easy, but each frame depends on previous frames... so now you need to share lots of data between threads. its not as embarrassingly parallel as it looks from a naive perspective.

although i contend that most decoders are very threadable - just that the people trying to do it usually lack the time or the skill, more usually the former.

the state of video in programming is a total mess from my experiences.

justinlaster2• 10 years ago

it would seem to be easy to have multiple threads, starting at key frames to decode the content.

It's a resource issue (memory, cpu, etc; and meeting latency requirements between those constraints), versus the subtly different standards "H.264" hardware and software follow, as well as a few other intricacies with how the whole standard works anyways. Again, it's not that it can't be done, but as the article says it can't be done easily or at least in certain situations done consistently.

Key frames are a good anchor around anything you're doing with H264 (and other formats), but it's not the end all and be all -- and they may even cause you trouble if you "trust" them too much. It is perhaps a bit like date time programming. You can create something fairly easily that works for a decent amount of time, and even if it ends up being incorrect your clients may not even notice... or it may breakdown in a catastrophic manner in the future. But doing the latter is certainly not correct and it's certainly not professional. Quite honestly, I'd say date time programming looks like a dream compared to the inconsistent nightmare that is video programming. Date/time logic needs to be sound because many programs rely on consistent and sane output from a program perspective, where as video programming gets to slide as long as the output is generally correct from a human visual perspective.

It's been a few years since I've dived into this stuff, so some things may have changed/gotten cleaned up. But the article seems to indicate that the ecosystem hasn't really changed.

iopq• 10 years ago

my hardware contexts will survive the user’s coffee break

hell, they don't survive alt-tabbing into a game that has a different resolution than the monitor

pvdebbe• 10 years ago

Heh... for some reason youtube can't survive when I start a video on my monitor and then I switch outputs to TV using an xrandr script by closing one output and opening the other. I thought it was possible to continue the video that way but once I noticed it doesn't work, it made sense immediately.

Mplayer and co, on the other hand can cope with it but my window manager can mess it up so I don't bother.

donatj• 10 years ago

- "all subtitle files are UTF-8 encoded"

Hah, this strikes really close to home. I've had to work with so so many subtile files in Eastern European and Turkish Windows codepages mostly but not entirely compatible with Win-1252. There's no way to tell them apart programmatically, so you check that the extended characters make sense. It's a bit of a nightmare.

franciscop2• 10 years ago

The original one ( http://www.kalzumeus.com/2010/06/17/falsehoods-programmers-b... ) left me bafled. Then I realized you have to strike a balance; otherwise you cannot deal with names at all. The point where drawing the line depends on your industry/customers, but I'd safely say that it's too restrictive nowadays so these lists are useful somewhat and of course they are interesting.

agumonkey• 10 years ago

Wasn't there one about time and timezones ?

So many mundane things have the "how hard can this be" ..

progval• 10 years ago

http://infiniteundo.com/post/25509354022/more-falsehoods-pro...

kibwen5• 10 years ago

It's true that you have to draw a line somewhere based on technical and business constraints, but an important takeaway of the names article is that you almost certainly don't need to do anything with a name other than treat it as an opaque string that can be displayed back to the user. For example, I'm struggling to think of a good reason why user registration would require separate first name and last name fields, and yet this practice is overwhelmingly common. For that matter, why do you want my real name at all, considering that it can't be used as a unique ID anyway?

Frondo5• 10 years ago

That's easy, it's to make better marketing materials.

"Dear [first name]," flows better than "Dear [opaque string],".

smallnamespace3• 10 years ago

But sometimes your government expects names to be broken into given and family names (the US, Japan, and China all seem to make this assumption -- every government form I've seen from those countries wants your name fully broken out).

I've no direct experience with, say, Russian or Latin American governments, but cultures that use explicit patronymic or matronymic names might expect that broken out as well.

If you ever need to submit user data to the government (e.g. for tax reasons), and you don't ask your user to break the name apart, then you will necessarily be guessing, which seems strictly worse than just asking them how their name might split.

At the end of the day, if you operate in a given culture, then you need to address those cultural norms. Bending over backwards to support every possible edge case seems unwise if they also happen to disagree with those norms.

JoshTriplett3• 10 years ago

Yeah, the same applies to many of the falsehoods in this list. Some of them you have to get right; some of them you can probably ignore without causing practical problems.

artpepper2• 10 years ago

What I take from the names article is to make sure you understand your requirements. A blogging platform will have different requirements than, say, electronic medical records.

nradov• 10 years ago

For EMRs, the HL7 FHIR HumanName datatype covers most of what's needed. Patients can have 0 or more names, each of which can have a full string representation plus 0 or more separate family / given / prefix / suffix components. And each name can be tagged with a usage type (official / maiden / nickname / etc) and validity date range.

http://www.hl7.org/FHIR/datatypes.html#HumanName

innocentoldguy• 10 years ago

People's names is one area I think developers constantly bugger up. For example, my last name has a space in it, but half the websites I register on either:

1. Throw an error and won't let me enter my last name as it is supposed to be spelled.

2. Truncate the last part of my last name.

3. Try to be clever and end up shoving the first half of my last name into a middle-name field.

My preference for names, addresses, and other personal data is to stop trying to constrain people to preconceived "standards" and just let them enter their information the way they want it to be.

sdenton4• 10 years ago

I'm a fourth, and occasionally the IV suffix gets horribly mishandled. I recently booked a flight on Travelocity and had to change arrangements by looking up the reservation on the actual airline air, but it turned out they had dropped the space between my last name and the IV, making it impossible for me to log in until I figured out what had gone wrong...

franciscop2• 10 years ago

As a Spaniard I have two last names (no middle name, no one last name with space) but 0% of the non-spanish websites I have ever seen accommodate for this.

Normally I just use it with an space in the last name field, but then I get exactly the same problems you mention.

lolc• 10 years ago

And this is why I don't do video. (And have lots of respect for the people who write the libraries I use.)

AznHisoka• 10 years ago

can we have falsehoods programmers believe besides video that are more common? this list probably is relevant for 1% of programmers here.

greenyoda• 10 years ago

Just type "falsehoods programmers believe" into the search box at the bottom of the page and you'll get a ton of previous articles on falsehoods in various domains that have been posted here over the years:

https://hn.algolia.com/?query=falsehoods%20programmers%20bel...

And while this topic is not personally relevant to me since I don't work with video decoding, I do find learning about different technologies interesting. Reading this gives me an appreciation for how much effort goes into making video, something we all take for granted, work.

If people only posted articles that were relevant to a majority of readers, HN would be a much less interesting place.

fgandiya2• 10 years ago

Here's a whole list! https://github.com/kdeldycke/awesome-falsehood/blob/master/R...

derefr• 10 years ago

rendering subtitles at the output resolution is better than rendering them at the video resolution

I would like to know what's wrong with this approach. I watch a lot of commentated speed-run videos: that's often something like ~244p video, plus soft subtitles. The subtitles get rendered at the source resolution (presumably, into the video framebuffer) and then upscaled along with the image, forcing them to be a tiny blurry mess instead of the crisp, readable text they could be.

kazinator• 10 years ago

Maybe nothing is wrong; just that maybe it's not always strictly better. Suppose you are asked to form a plan for adding subtitle support to some unfamiliar video platform. It's probably best to start with an open mind about where in the pipeline subtitles will be composed with the video.

akiselev2• 10 years ago

I think that point should be amended to say "rendering subtitles at the output resolution is always better than rendering them at the video resolution." You don't want to upscale 244p soft subtitles to 1080p but you do want to default to giving video authors creative control over how the subtitles are displayed. The ASS subtitle format allows for some very complex styling that can be used as an artistic element in video (or just to make sure there's proper contrast, can be read by color blind people, character differentiation, etc.) so you generally don't want to assume anything. There's also the issue of coordinates for where the subtitles are supposed to be that all go to shit if you render them on a transformed (up/downscaled) frame.

haasn5• 10 years ago

This comment is pretty much what I was going for. I've reworded it to make it clearer.

The issue you can run into in practice is stuff like softsubbed signs, which can clash and look out of place with the native video if you render them at full res. There's also a related issue, which is that if you're using something like motion interpolation (e.g. “smoothmotion”, “fluidmotion” etc. or even stuff like MVTools/SVP), softsubbed signs will not match the video during pans etc., making them stutter and look very out-of-place - the only way to fix that is to render them on top of the video before applying the relevant motion interpolation algorithms.

Personally I've always wished for a world in which subtitles are split into two files, one for dialogue and for signs, with an ability to distinguish between the two. (Heck, I think softsubbed signs should just be separate transparent video streams that are overlayed on top of the native picture, allowing you to essentially hardsub signs while still being capable of disabling them)

Also, sometimes, rendering at full resolution is prohibitively expensive, e.g. watching heavily softsubbed 720p content on a 4K screen.

JoshTriplett3• 10 years ago

There's also the issue of coordinates for where the subtitles are supposed to be that all go to shit if you render them on a transformed (up/downscaled) frame.

Sure, you have to transform the coordinates to the output. But still, better to render fonts at the final resolution; they'll always look better than if scaled after rendering.

jheriko6• 10 years ago

the point is precisely that it is more complicated than this obvious interpretation.

whilst i don't necessarily agree... i do agree that if you want to conform to specs then you can't go thinking this way.

CoolGuySteve• 10 years ago

It's also missing the most common error I see: conflating subtitles with closed captions.

Closed captions are positioned on the screen to indicate who's talking, have descriptive audio for sound effects, and should be in a high contrast easy to read font (most people with hearing deficiencies also have problems seeing, ie: out of date prescriptions for both hearing aids and eye glasses).

As far as I know, QuickTime does it right but the Apple TV, Netflix, and YouTube fuck it up, but that's because I helped write the QuickTime one way back.

deadmutex5• 10 years ago

AFAIK, The YouTube implementation does all of those.

Here is a demo: https://www.youtube.com/watch?v=BbqPe-IceP4

Please do not spread falsehoods.

Disclamer: I work at YouTube.

tantalor• 10 years ago

Okay, so how are subtitles different?

the84726• 10 years ago

In fact, rendering subtitles at the display resolution is one of the big selling points of the xy-subfilter + madvr renderer combination.

The only practical downside I have noticed is that accurate rendering of subs containing complex vector graphics or effects (ASS supports that) at > HD resolutions takes a lot of CPU time, sometimes more than a single core can handle in realtime.

There probably is a lot of potential for optimization, but those are hobby projects for their maintainers.

FranOntanaya• 10 years ago

Could write an entire page just on subtitles.

milansuk• 10 years ago

Nice one! Now I would like to see article like this, but about ciphers, hashes, digital signitures, etc.

jheriko6• 10 years ago

it is true, video is a nightmare mess littered with weird functionality nobody needs. (limited range only just disappeared in rec 2100, optionally??? really??? i'm not worried about my electron gun in my CRT from 1975 these days...nor do i want to know what a Y or a Cb or a Cr means because everything is RGB and B&W TV is long dead... and 4:2:2 is not exactly compression so much as computational overhead etc.. etc.)

its a nightmare, but the reason for these observations is precisely that it shouldn't be a nightmare. this area of programming is a wasteland ... nobody that good wants to solve these trivial problems :/

mrob2• 10 years ago

Chroma subsampling isn't going anywhere. You'll usually get subjectively better quality with 4:2:0 chroma compared to 4:4:4 at the same bitrate. And this means you can't have everything in RGB, so all the colorspace conversion complexity can't be ignored.

Try experimenting with chroma subsampling in JPGs, but note that not all image viewers have good chroma upscaling. MPV can display still images as well as video and you can choose the chroma scaling algorithm.

haasn5• 10 years ago

Chroma subsampling isn't going anywhere. You'll usually get subjectively better quality with 4:2:0 chroma compared to 4:4:4 at the same bitrate. And this means you can't have everything in RGB, so all the colorspace conversion complexity can't be ignored.

What's more, YCbCr is more efficiently compressed than RGB even if you don't subsample, for the same reason that a DCT saves bits even if you don't quantize: Linearly dependent or redundant information is moved into fewer components, in this case most of the information moves into the Y channel with the Cb and Cr both being very flat in comparison. (Just look at a typical YCbCr image reinterpreted as grayscale to see what I meant)

jheriko6• 10 years ago

you are right, there is a weird grainy sharpness kind of 'feeling' when chroma is not subsampled...

but you get the exact same effect from higher resolutions, e.g. going from SD->HD->2K->4K we see the same thing... and we are still doing it, so i would question highly that it is subjectively better in a long-term sense given this continuing trend.

i remember hearing people discuss this sort of thing when HD was new, and they stopped after while - i suspect because they got used to it, and they now realise how low the quality of the SD image was. i noticed this in myself as well...

edit: incidentally there is a discussion about this here (first google thing i found): http://www.neogaf.com/forum/showthread.php?t=1308591

its seems either nobody or very few are taking the perspective that 4:2:0/4:2:2 looks better, and there are even a few descriptions of precisely what they notice as being worse.

kierank• 10 years ago

Please let me know what you do when you have filter overshoots or undershoots in full range. Limited range is there for good reason.

jheriko6• 10 years ago

er... what? can you explain that a bit i think i must misunderstand?

what i think of as undershooting or overshooting is relative to the range... and besides that, what is wrong with clamping? its how computer graphics has always had to deal with these things... limited range simply doesn't exist in that context, and it doesn't harm anything.

when computer games are forced into limited range for consoles you don't get these unless your tv is applying one of those god awful filters that ruins everything anyway... (i'm still not sure why so many tvs have these - reference monitors never do anything this insane) ... but i can tell you what you do get, a subjectively /and/ measurably worse quality of image than from a monitor.

(i don't think i'm alone in this based on the contents of the ITU-R BT.2100 either... which defines a full range as well as a 'narrow' one)

imaginenore2• 10 years ago

interlaced video files no longer exist

Interlaced video files should no longer exist.

Seriously, fk interlaced video.

upscaling algorithms can invent information that doesn’t exist in the image

That's not a falsehood. Upscaling does invent information that doesn't exist in the image.

mrob2• 10 years ago

"Information" in the information theory sense. The output of a deterministic upscaling algorithm can be exactly described by the input and the algorithm. There's no added information, only a different way of presenting the original information.

emcq• 10 years ago

Perhaps that author was being pendantic, but from an information theroetic perspective it is correct that you cannot invent information with upscaling.

The upscaled image does not have more information than what was in the original image; you can reconstruct the upscaled image given only the information available in the original image, the output resolution dimensions, and upscaling algorithm.

imaginenore2• 10 years ago

That's like saying fractal images are not information. Just because something is generated by a formula, doesn't mean it's not new information.

jeff_tyrrill• 10 years ago

Interlaced video files should no longer exist.

Yes, they should, as should silent movies, black and white movies, old game consoles with exotic output formats like vector graphics, and the like.

It is a worthy endeavor to create and maintain video playback software that lets people consume beloved content that was made to the technology of its day, including home videos, sports games, TV shows with special effects edited in 60i, and video games.

ryanmarsh• 10 years ago

Well video programming just sounds delightful.

/sarcasm

scottlamb• 10 years ago

From the article:

I can exclusively use the video clock for timing

Heh. I just finished writing up a design doc to address problems I had with this, and I referenced "Falsehoods programmers believe about time". Then I opened Hacker News and saw this article. So this is very timely for me.

(My doc: https://github.com/scottlamb/moonfire-nvr/blob/new-schema/de...)

microcolonel• 10 years ago

I don't think programmers believe any of the video decoding falsehoods; not because they know any better, but because they know they don't know.

Also, none of these unfounded preconceptions make intuitive sense, so I don't see why people would believe them.

the_duke• 10 years ago

An explanation for each 'falsehood' would have been nice

antirez• 10 years ago

There is a lot of potential information in such a list. But in this form is quite a "trust me" thing that does not really add to the reader knowledge.

Falsehoods programmers believe about video

Related Stories