You Too Could Have Made Curl

https://daniel.haxx.se/blog/2024/02/06/fosdem-2024-you-too-could-have-made-curl/

by sebg • 2 years ago

143 76 2 years ago

daniel.haxx.se

Several people eventually got involved, things were rebooted multiple times, cables were yanked and replugged in again. First after I installed arandr and forced-updated the resolution of my HDMI output to 1920×1080 the projector would suddenly show my presentation. (Later on I was told that people had the same problem in this room the day before…)

That's why it is a good idea if you're a presenter to always have a Decimator [1] or its cheaper and less-capable alternative from BMD [2] in your pocket. No matter what, both of them will make sure you can get at least a signal to any sink with an HDMI or SDI port. Even 20 years after EDID, negotiation issues reign supreme...

Personally, when I'm responsible for inhouse conferences, I always have the venue place a Decimator at the presenter desk and run the cable to the projector using SDI - particularly MacBooks and their USB-C dongles tend to have issues with cable lengths above 5-10 meters, I'd guess too low voltage swing.

I literally never ran into incompatibility issues with this setup, only issue I had was some dumbass thinking they could unplug the Decimator to charge their phone. Additionally, SDI makes it very easy to install a split-off run to a central recording site, even if it's 100 meters of cable run away, or a split screen at the presenter's feet. No negotiation issues, everything Just F...ing Works.

[1] https://decimator.com/Products/MiniConverters/12G-CROSS/12G-...

[2] https://www.blackmagicdesign.com/de/products/miniconverters/...

lqet• 2 years ago

Interesting, what exactly does the Decimator do?

(Side note: I don't think it is a particularly clever idea of Decimator Design to fill their logo and homepage with images of large spiders. They are effectively eliminating around 10% of the population with arachnophobia as customers, and are making the experience of visiting their website slightly uncomfortable for large parts of the remaining population.)

mschuster915• 2 years ago

Interesting, what exactly does the Decimator do?

A multitude of things, depending on how you configure it:

- signal boosting of low-voltage or otherwise problematic HDMI or SDI signals, including reclocking. That's imposing virtually no latency and if you're running in a TV broadcast environment with centralized clocking it makes work easier for the downstream mixer. Primarily, it avoids issues with mis-designed sources such as cheap-ass adapter dongles that have signal integrity issues - as long as it's "good enough" for the Decimator to recognize it will output a clean and high-strength signal.

- format conversion from HDMI/DVI to SDI and vice versa. That imposes a tiny bit of latency and allows the usage of "consumer" equipment with broadcast equipment, or if you use two Decimators linked by SDI, you can "just" go for far longer cable runs than you could ever do with HDMI - up to 100m, in practice way more.

- up and downscaling as well as framerate conversions. This is where things really get interesting - no matter what kind of source and sink you use or what their limitations are, the scaler engine can make them talk. That includes color space conversions and bandwidth as well, and it's particularly useful if you, say, have a 12G SDI link from the source to the sink, and you want to use an older display for additional monitoring. Splice a Decimator in there and it will output the full quality signal on one output while providing a scaled-down signal on the second output.

wruza• 2 years ago

Yeah, that’s the perfect case of a good product overrun by a questionable design idea. And its name sounds 13yo too, tbh.

Scaevolus• 2 years ago

Decimator Design excluding 10% of the population is a historically accurate decimation.

anthonyeden• 2 years ago

I agree. I personally leave my Decimator MD-HX’s in ‘Free Run’ scaling mode at my preferred resolution. The in-built ‘no signal colour’ of my choosing helps me troubleshoot the problem in a signal chain quickly.

Worth mentioning for the novices that SDI does not do HDCP, so you can’t pass copy-protected content via this setup without workarounds.

Of course, it really should be up to the conference AV suppliers to organise all this, not the presenters themselves.

mschuster915• 2 years ago

Worth mentioning for the novices that SDI does not do HDCP, so you can’t pass copy-protected content via this setup without workarounds.

Well, the workaround is a 10$ HDCP stripper from ebay. Easy enough. (And I think that both Decimators and BMD converters also strip HDCP, but don't quote me on that)

Of course, it really should be up to the conference AV suppliers to organise all this, not the presenters themselves.

Well, conference setups tend to be a hit and miss IME. Some where the venue and the conference is run by dedicated professionals, you don't even have to tell them to place Decimators and SDI cable runs, they'll come in with these from the start in their offer... but a lot of smaller venues and conferences (especially those ran by volunteers / NGOs) don't have that knowledge and just place some random home cinema projector together with a 20 meter HDMI cable. Corporate offices are just as bad IME, I've never seen a corp conference room with anything but HDMI or, in the worst cases, VGA. Corporate just doesn't give a flying fuck unless they got a dedicated team and a proper budget working on conference room setups...

stephen_g• 2 years ago

Yes, SDI was just designed for this kind of thing (moving video long distances between cameras in studios, tape decks and patch bays in technical rooms, vision switches and capture equipment in editing bays, etc.), whereas HDMI was only ever designed for connections of a few metres (and then active optical cables had to be made to try and fix the mess)…

Arelius• 2 years ago

It looks like the MD-HX is what you want for this use case?

I do a bunch of international travel, and the HD-LX seems ideal sized, but it seems like it doesn't do most of the useful functions?

mschuster915• 2 years ago

It's "only" 3G-SDI so FullHD only.

trustno22• 2 years ago

It seems like getting projectors to run is really the hardest problem in computer science.

Maken• 2 years ago

Harder than getting printers to print?

mschuster915• 2 years ago

At least printers give you some sort of error message.

EDID negotiation or signal integrity issues in HDMI, good luck diagnosing actual causes without highly specialized (and expensive...) equipment.

trustno22• 2 years ago

Hey that's what we get CUPS auto-discovery for.

high_na_euv• 2 years ago

It is unreal how tech as mature as printers is so fucking unreliable

eka1• 2 years ago

Typically, you aren't trying to print in front of a large audience

netsharc• 2 years ago

Meanwhile, "this scratchy audio is fine!".

I guess it's something one can throw AI at. It probably doesn't need GPUs, just some audio filters.

jimbokun• 2 years ago

I've been in Computer Science graduate courses and lectures in one of the top programs in the US where it seemed like every week the first 5 to 10 minutes was spent trying to get the projector to work with the lecturers laptop.

mlyle2• 2 years ago

I just have a ChromeCast in each room that I lecture in. Easy to bounce a tab over to the big screen, and it turns on the TV/projector. It also helps prevent notification or other screen sharing related mishaps.

My biggest complaints about this approach are that I can't easily see speaker notes (I have a fiddly workaround that I can use if I need it, but it would be nice if Google Slides would support the ChromeCast use case a little better..) and that the TVs/projectors tend to screw around for 10-15 seconds before automatically choosing the right source.

zikduruqe• 2 years ago

What amazes me on conference calls is how no one can explain audio issues. Really, really smart people on a call that can explain registers, cryptography, kernel tuning.

Sounds like you are having network issues/bit errors? You are breaking up.

Sounds like your bluetooth is having issues? You are breaking up.

Sounds like they are mobile, and in a weak service area? You are breaking up.

troupo• 2 years ago

I used to joke it was ironic how people in management would always have their presentations up and running almost immediately while programmers seemed to always struggle with projectors.

These days however nothing seems to work for anyone.

RcouF1uZ4gsC• 2 years ago

Management is typically running a stock Windows Dell/HP/Lenovo or a Mac.

Programmers are running some custom Linux distro with a custom window manager and other customizations.

rqtwteye• 2 years ago

Right after getting the screen layout right when you move between home office and work office :-)

a1o• 2 years ago

Seriously should be possible to save those in a layouts pane to force adjust the monitors

AndyMcConachie• 2 years ago

I once helped Andrew Tannenbaum get his laptop to work with the projector in the room. His presentation was only slightly delayed because of it, but the irony ...

jdboyd• 2 years ago

I've had good luck slapping an Extron IN-1606 in front of a projector to make all the compatibility problems go away.

thefaux• 2 years ago

This feels like such a fixable problem. Presenters should be required to show up early and test the av or send the conference their slides in advance.

ryandrake• 2 years ago

The burden shouldn't be on users. Why are computer manufacturers putting out a hardware/OS combination that can't work reliably with an external display? Don't they even test this common use case?

Even my Mac (which, to Apple's credit, does work 100% reliably with projectors) still struggles when I plug in multiple external monitors, blanking the screen, turning on one external display, blanking them both, turning on the other one, and so on. This is a manufacturer who normally accepts only a smooth, polished user experience, and they still can't get it perfect. What chance do Lenovo or similar garbage-tier plastic box manufacturers have of getting it right?

strunz• 2 years ago

Say what you want about Apple but I've never had a problem printing or connecting a monitor/project (as long as I have the correct dongle :P )

mbonnet• 2 years ago

A/V is the final frontier, not P ?= NP

hodgesrm• 2 years ago

I saw this talk in person. It was delightful. My favorite part was the stories of helping random people who contacted him to help fix things like car electronics. They found his email from the OSS license acknowledgments, which apparently are accessible to ordinary users.

(As they should be.)

oefrha• 2 years ago

Well he also got "I will slaughter you" through the exposure of his email in OSS license acknowledgements, so it's not always fun after all.

https://daniel.haxx.se/blog/2021/02/19/i-will-slaughter-you/

tivert• 2 years ago

After reading that I thought the guy who sent that email must have been crazy, and it turns out he was: https://daniel.haxx.se/blog/2021/08/09/nocais-apology/:

I had to retire from my career do to schizophrenia. Again, I should have not let my delusions go to the point they did nor should I have acted the way it does. My illness doesn’t detract from the rashness of my actions.

There a lot of technical word salad there, like "When I say hacked, I lost every device. They rooted my charge arbitrator, I was bios bonded and I basically lost every document and all my software."

silvestrov• 2 years ago

The problem is not to make a tool like curl.

The problem is the marketing and getting people to use it.

You can easily make the best product and still not being able to get the word out there.

tootie• 2 years ago

There was a bit of PR "war" between curl and wget a long time ago. At some point curl just became everyone's goto and even the reference implementation for http.

taneq• 2 years ago

I never did really figure out why curl became a thing when we’d been wget to do that since 2001.

masto• 2 years ago

Many moons ago, I was in the small ISP business, and I discovered the aftermath of an attempted script kiddie hack on one of our servers. When I examined the logs I realized we were extremely vulnerable to the remote code execution exploit but had been completely saved by two things: they kept trying to use curl to install the payload, but we only had wget installed; and their scripts were extremely Linux-centric but we were using FreeBSD.

tannhaeuser• 2 years ago

wget (and htrack) is able to parse downloaded HTML and CSS for scanning additional URLs to fetch recursively and mirror entire sites that way, while curl is "just" a very very complete HTTP client.

Though wget isn't perfect for mirroring, as it will typically download large amounts of resources redundantly only differing in URL query params when those refer to the same content, such as on typical WeirdPress sites with comment links. Ideal would be an HTTP client with http/3, auth token, and keepalive support etc. based on libcurl that can be customized, such as via "href handlers" triggered by event-driven markup parsers for SGML (though CSS and JS imports need special treatment).

mijoharas• 2 years ago

Curl's success has more to do with it's quality (and ubiquity) as a C library. I think the curl cli somewhat got "taken along for a ride" with all the improvements that curl lib had by becoming the de-facto standard http library.

Because the library gets ported to every new platform anyways (it's what people use so will be the first to be ported to a new architecture or whatever) so the cli gets support for everything new "for free" so can outcompete wget (because it's always _there_ and works the same).

Now, there's also the fact that the curl cli is a fantastic piece of software, but in terms of features I don't think there's that much between curl and wget for simple cli usecases (but I still use curl).

bigstrat20032• 2 years ago

Curl certainly isn't my go-to, because it's so complicated. For most uses I can just "wget http://blahblah" and it just works. With curl I have to look up the flags to save the file, etc. Curl is certainly more powerful, but that power comes with a steep usability loss for the common case.

ww520• 2 years ago

Amen. Marketing is so important for software, even for OS software.

hnthrow289570• 2 years ago

Word of mouth can still help you, but you at least need to find the community where people have the shared problem, so their recommendations have weight

Overcoming the initial "why should I even look at this" to me is the hardest part

bityard3• 2 years ago

I don't think Daniel "marketed" curl?

I couldn't say when curl was released without looking it up, but I know it's been around for at least a couple of decades. There were not many command-line tools for dealing with web stuff, and curl got picked up by most Linux distributions, the BSDs, and various embedded things because it was the only tool that did what it did, as well as it did. And then it became famous essentially for being ubiquitous, like bash or vim.

"Marketing" seems to have had very little (if anything at all) to do with it.

0cf8612b2e1e• 2 years ago

Best-in-class vs First-in-class (not that I have any criticisms of curl). Sometimes just being first is all it takes to win.

ShanAIDev• 2 years ago

It seems like the first-mover advantage really plays a significant role here.

JeremyNT• 2 years ago

If curl was created today? Sure.

But curl was something new when it came out, and it truly filled a void both as a cli tool and as a library. It needed no marketing. We needed it.

It's hard now to imagine a world without a tool like curl, but quite a few of us are old enough to remember it still.

tempfile• 2 years ago

arandr mentioned! I use it all the time. Never had an issue.

password4321• 2 years ago

TIL: ARandR is a tool to control output positioning and properties of multiple monitors using XRandR.

https://christian.amsuess.com/tools/arandr/

Recently set up something similar for Windows as a win in the battle against time wasted thanks to auto-logout.

https://github.com/caoyue/WindowResizer

yjftsjthsd-h2• 2 years ago

... Having used arandr for ~forever, it suddenly occurs to me that people complaining about display issues on Linux maybe just don't know it exists? That would explain a bit

ketanmaheshwari• 2 years ago

"Very few things are an immediate success". Wise words.

jeanlucas2• 2 years ago

Daniel is a unique OSS builder. We asked him to give a talk to our community and he accepted, no questions asked, just showed up :-)

His commitment to spread the word of curl and OSS is really admirable.

shark1• 2 years ago

The man built a software used by billions and still gets nervous talking in public :)

It was a delightful presentation.

You Too Could Have Made Curl

Related Stories