MattJ100

You can do big data in SQLite. Concurrent writes, sure, I'd recommend something else.

If you think the majority of systems require massively concurrent writes, I think you need to look a bit harder. SQLite is, after all, the most widely deployed database system, ever.

It's a reference to Eternal September, the name applied to the cultural shift of the internet as the general public started gaining access to it en masse in the 90s.

Sloptember is clearly a reference to this - the similarity being that masses of AI generated content, from social media posts to open source contributions are replacing the human internet. In a way this is related to the "dead internet theory", an idea I previously found hard to believe, but these days could easily be true.

If the history of the internet interests you, both these are worth looking up.

Each commit should build and pass tests, yes. When I say "partial commits", I don't mean that the commits are arbitrary - each commit should be as small as possible to implement a specific fix/feature. I've also heard it described as the smallest unit that you may want to revert.

For example, if you are working on something, but it requires adding an API to some module, then the first commit 1 is to add the new API (+ tests), and the second commit is the new code that uses that API.

Unfortunately many developers I have worked with would just combine these (and more) into a single commit (because they are part of the same work task). However this makes review, bisect, blame and revert harder (if you need to revert commit 2, you don't want to also revert the API you added if that was tested and bug-free).

Yes, Mercurial has a very advanced history editing system via "evolution": https://wiki.mercurial-scm.org/ChangesetEvolution

A good way of thinking about it is that every commit is itself version-controlled, allowing unlimited edits. This even allows two people in an evolve-enabled repo to make changes to history at the same time, and Mercurial will resolve any conflicts. It makes it trivial to commit (and even share) a "WIP" commit which you can later amend/split/whatever. It's different from git where you basically can't edit history after pushing (in Mercurial this only becomes true if you push to a non-evolvution or "publishing" repo, where everything then gets squashed for public consumption).

To be clear, Mercurial does not have a staging area, but it does have allow selective commits (and selective uncommits) via prompt-based or interactive UI selection of hunks. Disagreeing with the need for a staging area is not the same as saying selective commits are unnecessary (I use Mercurial more than git and I rarely commit everything in my working directory in a single commit - I like small commits).

We see this in our open-source community. We've had a community channel for over two decades, where community members help newcomers and each other solve problems and answer questions.

Increasingly we have people join who tell us they've been struggling with a problem "for days". Per routine, we ask for their configuration, and it turns out they've been asking ChatGPT, Claude or some other LLM for assistance and their configuration is a total mess.

Something about this feels really broken, when a channel full of domain experts are willing to lend a hand (within reason) for free. But instead, people increasingly turn to the machines which are well-known to hallucinate. They just don't think it will hallucinate for them.

In fact I see this pattern a lot. People use LLMs for stuff within their domain of expertise, or just ask them questions about washing cars, and they laugh at how incompetent and illogical they are. Then, hours later, they will happily query ChatGPT for mortgage advice, or whatever. If they don't have the knowledge to verify it themselves then they seem more willing to believe it is accurate, where in fact they should be even more careful.

URL parsing/normalisation/escaping/unescaping is a minefield. There are many edge cases where every implementation does things differently. This is a perfect example.

It gets worse if you are mapping URLs to a filesystem (e.g. for serving files). Even though they look similar, URL paths have different capabilities and rules than filesystems, and different filesystems also vary. This is also an example of that (I don't think most filesystems support empty directory names).

Yes, it is. The worst offenders hammer us (and others) with thousands upon thousands of requests, and each request uses unique IP addresses making all per-IP limits useless.

We implemented an anti-bot challenge and it helped for a while. Then our server collapsed again recently. The perf command showed that the actual TLS handshakes inside nginx were using over 50% of our server's CPU, starving other stuff on the machine.

It's a DDoS.

Yes, definitely. To me, the idea of a chat server that doesn't federate is as absurd as setting up an email server that doesn't federate. I understand that today people know more contacts with email addresses than XMPP addresses, but if we ever want to free ourselves from the current walled gardens, we need to stop treating chat as something that only happens in walled gardens.

Some people get worried about the idea of "federation", thinking that it somehow means their server is less private, and their data is being spread across a mesh of servers, and stuff like that. That's true in some decentralized/distributed chat protocols, but not in XMPP. Connections between servers only happen on-demand, similar how when you send email between different email providers, they will connect to each other to deliver the messages.

However we do have a feature which allows disabling federation access for specific accounts, for example to prevent kids from communicating with anyone outside their own Snikket server. This is a feature I want to expand on, so that you can permit communication with a limited number of approved contacts on other servers.

You're welcome!

I'm still experimenting with the messaging on the Snikket website. However my general approach with the site was to pitch Snikket to people who don't know what XMPP is, which is, frankly, the majority of people. Instead, I wanted to focus on explaining features it enables rather than protocol details. But I'm aware it has caused a lot of head-scratching among people who already know Snikket uses XMPP :)

I see Snikket as kind of a gateway into the XMPP ecosystem for people who are unfamiliar with it. After all, if you're already familiar with XMPP then the chances are you'll probably be happier with Prosody or ejabberd, and you'll already have opinions about which clients you want to use (e.g. the upstreams of Snikket).

Yeah, iOS is definitely a weaker spot and we're aware of it. Monal is currently working on an overhaul of their UI (they have a grant allocated for it: https://nlnet.nl/project/Monal-IM-UI/ ).

There is also a new app in the works between Cheogram and Snikket. There is a beta available, but it's still young (and we won't apply any Snikket branding until E2EE is complete).

Thanks for sharing your experience!

Be aware that this post has known issues that the author is not interested in fixing. In their own words (in response to clarifications by one of the OMEMO folk):

"I'll make an edit later about the protocol version thing, but I'm not interested in having questions answered. My entire horse in this race is for evangelists to f** off and leave me alone. That's it. That's all I want." [censorship of profanity mine]

You won't find this quote in the article with Ctrl+F, it's in the screenshot, where they omitted the original constructive comment by one of the OMEMO contributors that they chose to moderate, which you can find here: https://www.moparisthebest.com/tim-henkes-omemo-response.txt

So, by all means, read the blog post. But just be aware that its ultimate goal was not to be an unbiased accurate technical article.

I'm the founder of both the Prosody and Snikket projects. Sorry about triggering alarms :) I can try to explain...

Prosody is a popular choice of XMPP server software. It's used for all kinds of stuff, from self-hosted chat servers to powering Jitsi Meet, to Internet-of-Things applications.

Prosody is extremely flexible, and has a bunch of configuration options that allow you to adapt it and extend it however you want. For some people, this is ideal. Those people should continue using Prosody.

Snikket has a different scope. It is specifically an answer to a question like "How can I easily make a self-hosted WhatsApp/Signal for my family/friends using open-source software?"

- Snikket contains Prosody, for the core chat part. But it's Prosody with a very specific configuration, and the configuration is part of the project, it's not intended to be modified by the person deploying Snikket. They only need to provide the domain name.

- Snikket also includes additional components that a modern chat service needs. For example, it includes a STUN/TURN server to ensure that audio/video calls work reliably (again, preconfigured).

- Snikket provides its own apps, which are tested and developed in sync with each other and with the server. This avoids the common problem of incompatibilities that occur when you have an open ecosystem such as XMPP, where different open-source project developers may develop features at different paces, leaving users to figure out which ones support which feature. It also solves the discoverability and decision fatigue for users (searching "Snikket" on an app store will get you an app that you know is compatible with your Snikket server, you don't have to go through a list of XMPP clients and figure out which one is suitable).

- Snikket servers are not designed to be open public servers (these are an administrative nightmare). Instead, your server is closed and private by default. As the admin, you choose who signs up to your server by sending invitation links. The invitations also serve to simplify the account setup process - no need to prompt users to "choose a server", etc. They just need to provide a username.

Projects such as Conversations differ by running a single public server (conversations.im) and guiding people to sign up on that server, or choose one of a long list of free public XMPP providers. In some cases that's all what you want. But onboarding a group of people that way is not fun (for example, they all have to share their addresses with the group add each other to their contact lists one-by-one - Snikket makes discovery of contacts within the same server automatic).

Beyond these things, Snikket is all open-source and XMPP. But there is a focus on making a good polished and secure "product", if you like, rather than supporting the entire diverse XMPP ecosystem which includes a range of software of varying quality (weekend projects and more recently, 100% vibe-coded clients). For example, Snikket servers require certain security and authentication features which some older codebases that have fallen far behind modern XMPP standards (think Pidgin, etc.) simply don't support today.

it’s actually all based on prosody and conversations?

As mentioned, I develop Prosody. I also collaborate with the Conversations developer and other XMPP projects. There's nothing shady here. The goal is just to make a best-in-class XMPP project that solves one particular use case (and it was primarily my own use case to begin with of course - I wanted to move my family off WhatsApp).

As the founder of both projects, explaining the difference between the two projects is roughly 20% of my working day (okay, not quite 20% but sometimes it feels that way).

Your description is great :)

You're not wrong. PKI has better protections against MITM, dialback has better protections against certificate leaks/misissuance.

I think the ideal approach would be combining both (as mentioned, there have been some experiments with that), except when e.g. DANE can be used ( https://prosody.im/doc/modules/mod_s2s_auth_dane_in ). But if DANE can be used, the whole CA thing is irrelevant anyway :)

There is a lot of confusion caused by overlapping terminology in this issue.

By "client certificates" I mean (and generally take most others in this thread to mean) certificates which have been issues with the clientAuth key purpose defined in RFC 5280. This is the key purpose that Let's Encrypt will no longer be including in their certificates, and what this whole change is about.

However when one server connects to another server, all of TCP, TLS and the application code see the initiating party as a "client", which is distinct from say, an "XMPP client" which is an end-user application running on e.g. some laptop or phone.

The comment I was responding to clearly specified " I don't see how TLS-with-client-EKU [...]" which was more specific, however I used the more vague term "client certificates" to refer to the same thing in my response for brevity (thinking it would be clear from the context). Hope that clarifies things!

Yeah, the resistance is outside the XMPP community. However we have a long history of working with internet standards, and it's disappointing to now be in an era where "the internet" has become just a synonym for "the web", and so many interesting protocols and ideas get pushed aside because of the focus on browsers, the web and HTTPS.

They weren't "HTTPS certificates" originally, just certificates. They may be "HTTPS certificates" today if you listen to some people. However there was never a line drawn where one day they weren't "HTTPS certificates" and the next day they were. The ecosystem was just gradually pushed in that direction because of the dominance of the browser vendors and the popularity of the web.

I put "HTTPS certificates" in quotes in this comment because it is not a technical term defined anywhere, just a concept that "these certificates should only be used for HTTPS". The core specifications talk about "TLS servers" and "TLS clients".

Firstly, nobody is actually calling for authentication using client certificates. We use "normal" server certificates and validate the usual way, the only difference is that such a certificate may be presented on the "client" side of a connection when the connection is between two servers.

The statement that dialback is generally more susceptible to MITM is based on the premise that it is easier to MITM a single victim XMPP server (e.g. hijack its DNS queries or install an intercepting proxy somewhere on the path between the two servers) than it is to do the same attack to Let's Encrypt, which has various additional protections such as performing verification from multiple vantage points, always using DNSSEC, etc.

Of these, (1) and (2) are already implemented in XMPP.

(1) just isn't that widely deployed due to low DNSSEC adoption and setup complexity, but there is a push to get server operators to use it if they can.

(2) is defined in RFC 7711: https://www.rfc-editor.org/rfc/rfc7711 however it has more latency and complexity compared to just using a valid certificate directly in the XMPP connection's TLS handshake. Its main use is for XMPP hosting providers that don't have access to a domain's HTTPS.

Yes, definitely. Prosody supports DANE, but DNSSEC deployment continues to be an issue when talking about the public XMPP network at large. Ironically the .im TLD our own site is on still doesn't support it at all.

Sorry, it's late here and I guess I didn't word it well. Dialback (these days) always runs over a TLS-encrypted connection, as all servers enforce TLS.

The next question is how to authenticate the peer, and that can be done a few ways, usually either via the certificate PKI, via dialback, or something else (e.g. DNSSEC/DANE).

My comment about "combining dialback with TLS" was to say that we can use information from the TLS channel to help make the dialback authentication more secure (by adding extra constraints to the basic "present this magic string" that raw dialback authentication is based on).

Is there a reason why dialback isn't the answer?

There are some advantages to using TLS for authentication as well as encryption, which is already a standard across the internet.

For example, unlike an XMPP server, CAs typically perform checks from multiple vantage points ( https://letsencrypt.org/2020/02/19/multi-perspective-validat... ). There is also a lot of tooling around TLS, ACME, CT logs, and such, which we stand to gain from.

In comparison, dialback is a 20-year-old homegrown auth mechanism, which is more vulnerable to MITM.

Nevertheless, there are some experiments to combine dialback with TLS. For example, checking that you get the same cert (or at least public key) when connecting back. But this is not really standardized, and can pose problems for multi-server deployments.

It has never been secure to accept the clientAuth EKU when using the Mozilla root store.

Good job we haven't been doing this for a very long time by now :)

There might be some confusion here, as there is no refusal at all.

As stated in the blog post, we (Prosody) have been accepting (only) serverAuth certificates for a long time. However this is technically in violation of the relevant RFCs, and not the default behaviour of TLS libraries, so it's far from natural for software to be implementing this.

There was only one implementation discovered so far which was not accepting certificates unless they included the clientAuth purpose, and that was already updated 6+ months ago.

This blog post is intended to alert our users, and the broader XMPP community, about the issue that many were unaware of, and particularly to nudge server operators to upgrade their software if necessary, to avoid any federation issues on the network.

Thanks! I didn't intentionally write this for a broader audience (I didn't expect to see it while casually opening HN!). Our user base is quite diverse, so I try to find the balance between being too technical and over-explanatory. Glad it was helpful!

What logical fallacy, exactly? I think you're perhaps misunderstanding the conversation. This translates just fine to your proposed analogy.

In your analogy, the claim would be that some online account is tied to a laptop and whoever possesses the laptop has access to that account. The online service does not permit the account owner to revoke access from that laptop and move the account to a different laptop. I stand by my statement that this would be a serious security hazard. Because yes, laptops can and do get hacked or stolen, just like email addresses.

Where your analogy isn't quite as strong is that at least you can generally add additional anti-theft protections such as full-disk encryption to a laptop, while with an email account generally 2FA is the best you can do.

You're very wrong, because account takeover can still happen due to a compromised email account. People can and do permanently lose access to their email account to a third party.

Prosody dev here. Good to know this is useful :)

One of my favourite small features we added in the latest release is a 'prosodyctl check features' command which will validate that your configuration is up to date with current best practices in various ways.

Although we like to curate the default configuration with care, people tend to keep their existing configuration file when they upgrade (we generally recommend this, as we aim for backwards compatibility). The new command makes it easier for folk who upgrade to ensure they are still getting the expected experience as the recommendations evolve.

Obviously that has to be somewhat opinionated. One thing many people appreciate about Prosody is that you can customize it however you want to, even if it's not following the mainstream. E.g. if you really don't care about synchronizing to multiple devices, you can totally disable that message cache. Don't need audio/video calls? Fine!

Depends what you call "high", but the risks are far higher than most other drugs relative to availability.

"Paracetamol toxicity is one of the most common causes of poisoning worldwide." -- https://en.wikipedia.org/wiki/Paracetamol_poisoning#Epidemio...

I'm curious which aspect(s) became a time sink for you? I self-host a bunch of stuff myself. I can't say I never spend time on it, but it's measured in hours per year. Once stuff is set up, it just runs.