anze3db

Great point. I thought that `vacuum into` blocks writes, but it's only `vacuum` that blocks! I've updated the blog post, thanks!

I'm in a few Discord servers, but I have no idea how to find you there :)

I agree 100% with everything you wrote. It was very surprising to me that every transaction and every write operation blocks the whole database and not only the table it's performed upon (it makes sense since it's all a single file, but still).

The only workaround for this is splitting your main database into multiple databases, but this bleeds into your application logic and gets messy quickly. If you are in this position, it's best to switch to Postgres as you did!

Don't get me wrong, the PR review is still rigorous and can take weeks. But everybody I interacted with during the process was very supportive and helpful, so the overall experience was great!

Stephen has a whole series of blog posts on SQLite (in Rails) that I highly recommend. I've learned most of what I know about SQLite from him!

https://fractaledmind.github.io/2024/04/15/sqlite-on-rails-t...

Also, hello Lee! I miss being in a Slack with you!

From what I understand, you still need the experimental WAL2 mode[0] to not have this problem.

[0] https://www.sqlite.org/cgi/src/doc/wal2/doc/wal2.md

While this blog post was at the #2 spot on HN it was getting 0.138 rps, so yeah 14k RPS is A LOT!

That said it's still good to know when your tools will break! As far as I understand it's not really about RPS but more about constant write operations preventing the WAL file from being flushed. You can reproduce this issue with a lot fewer RPS.

From what I know, the journal_size_limit PRAGMA still affects WAL mode, but it doesn't solve the issue of the WAL file potentially growing uncontrollably. Am I missing something?

I've heard this is a potential issue but I've never encountered it. Do you know about the experimental WAL2 branch[0] that splits the WAL file into two to circumvent this problem? You'd have to compile SQLite yourself from the WAL2 branch to try it out, but it might be worth it at your scale.

Python is much slower than Go, so I don't think we could get 14k RPS as easily with Django, but I do have to see if I can reproduce the problem in Django. Topic for a future blog post!

Thanks for sharing this, I really appreciate knowing where the limits of SQLite are!

[0] https://www.sqlite.org/cgi/src/doc/wal2/doc/wal2.md

I can confirm. My experience contributing has been very positive!

That's rough. Is your code using transactions? If so, making sure you use `begin immediate` will help!

I'm not a Django core dev, but I have managed to get my changes merged into Django already (the transaction_mode setting in 5.1 was my contribution).

Carlton does seem to be onboard with my idea[0], so I'm optimistic that we can make it happen. Comments like yours will help me make my case so thank you for that!

[0] https://fosstodon.org/@carlton/112605212812578926

Hey HN! I'm the author of the blog post. I didn't mention this in the post, but I'll try to merge some of these settings into Django by making them the new default or the default for new projects created with the start project command.

Any feedback on all of this is greatly appreciated!

This joke makes some good points.

You've missed: pdm, uv, pip-tools, pipx, rye, and probably some others.

Only pdm and poetry generate cross-platform lock files by default as far as I know, but there are a lot of people trying to solve this problem right now.

It's not an easy problem to solve. Python's package management predates package managers from most other programming languages and Python itself predates Linux. There is a lot of baggage so change is very slow.

I'm not sure if you can compare a project that came out of a university and got adopted by Apple with a project developed by a VC backed company with no revenue. I'm sure Charlie has the best intentions with both ruff and uv, but we have no idea how this is going to play out.

If they stop development on Ruff today, Ruff won't be useful anymore after a few new Python releases. If the maintainers of the tools that Ruff is replacing stop maintaining them, the whole Python community will be in a really bad place. So, there is reason to be cautious. Astral being transparent on how they plan to make money would be very helpful.

What they are doing is already discouraging people from contributing to the projects their tools are replacing[0]. If they go out of business and stop supporting their tools, it might leave the Python community in a bad place.

[0] https://www.youtube.com/watch?v=XzW4-KEB664

I hope they answer this ASAP. People in the Python community are already concerned that Astral is following the Embrace, extend, and extinguish playbook.

I'd love to read your thesis! If it's possible please send a link :)

I personally wouldn't trust it with anything that involves submitting forms. What if it gets any data wrong? Even watching the demo video made me feel anxious!

I guess I'm finally on the other side of this Douglas Adams quote:

Anything that's invented between when you're fifteen and thirty-five is new and exciting and revolutionary and you can probably get a career in it. Anything invented after you're thirty-five is against the natural order of things.

I do think it's a cool project though!

That makes sense if you already have a pub/sub service running. But if you have a pub/sub service running, you probably also have a service that can count events so you won't need ratelimitapi.com.

In this case, I'm guessing the ability to configure the rate limits on your end would be useful. But again, I'm not your potential customer, so don't take this as customer feedback/feature request. I'm just thinking out loud :)

I like the simplicity of the landing page! For me personally, the tool would be more useful if I could adjust the duration and the rate limit count without redeploying my code, but I'd also sooner spin up my own Redis instance than use a 3rd party service - so I'm not sure if my feedback is useful :)

PS: You can simplify and make the Python example a bit more secure by using the params parameter[0] instead of building the query string manually:

``` requests.get(base_url, params=params) ```

[0] https://requests.readthedocs.io/en/latest/user/quickstart/#p...

Awesome, thank you!

Excellent, thank you! This is my project: https://github.com/anze3db/django-tui

Looks like a cool project! I really like the idea of seeing surging packages, but I probably won't remember to check your site every week. I'd be happy to sign up to a mailing list to receive the surge list in my inbox every week or two.

PS: As an author of a Django package my first question was where is the data coming from, and why isn't my package listed.

This bothered me as well and I've opened a PR to fix it: https://github.com/ofek/pyapp/pull/57

This looks really awesome! I'll try it out on some of my projects that I never tried to share outside of the Python community because setting up Python and/or pipx always felt too much of a hassle for end users.

Leave it to ofek to tackle one of the toughest problems in Python!

There are some configuration examples the docs[0] but since the project is still young there is probably room for improvements. Any particular that you'd like to see there?

[0] https://ofek.dev/pyapp/latest/examples/

I got confused by the buttons as well.

I'll try putting something more minimal in.

That would be great, do please add the boilerplate to all the weeks not just week one, because starting from a completely blank page is also challenging, especially for people who aren't familiar with Go all that much.