Richard Hipp Speaks Out on SQLite (2019) [pdf]

https://sigmodrecord.org/publications/sigmodRecord/1906/pdfs/06_Profiles_Hipp.pdf

by pncnmnp • 4 years ago

129 63 4 years ago

andrewl2• 4 years ago

This is interesting. I wonder if he’s actually working on it:

Q: If you magically had enough extra time to do one additional thing at work that you’re not doing now, what would it be?

A: Oh, I’ve got a long list. But my #1 thing right now I think would be a new version control system which the working name is Fit, it’s a combination of Fossil with Git. Uses the Fossil user interface but it uses the low-level file format of Git. So that then you can work with Fossil’s interface but push and pull to legacy Git users. And I think that would be huge.

vouaobrasil2• 4 years ago

That would be really nice. I think there is definitely room for version control for people who are less in the programming realm or who work on personal projects that have a more creative art flavour (writers, desingers, etc). The user interface of Git is too frustrating for these types. Even as a mathematician and somewhat-programmer I dislike Git a lot. Rebase, blame, stash...ugghhh. Although Git is nice I liked Subversion better.

tokamak-teapot• 4 years ago

Have you used Mercurial? I found it a lot more user friendly, especially with TortoiseHg for Windows. Not sure if that’s still around as I don’t use Windows any more but even though there are many similarities, git always feels like it’s designed to solve problems I don’t want to use source control to solve.

eternityforest• 4 years ago

Non-programmers aren't going to want any CLI version control, and git-cola plus meld is perfectly fine for simple use cases as long as nobody force pushes or uses a submodule or something like that.

Although usually what seems to happen is people only use one linear history and never branch anything. I'm not sure if that's because Git is hard, or just because nobody wants to learn about branches and stuff in any system if they don't have to.

In any case it works perfectly fine and is an amazing step up compared to no VCS.

stavros• 4 years ago

The user interface to git is too frustrating for everyone. Most of us just have Stockholm syndrome by now.

_a_a_a_• 4 years ago

That would seem to be just a new porcelain for git.

eesmith2• 4 years ago

Fossil also supports bug tracking, wiki, forum, email alerts, chat, and technotes.

I don't think it's right to call that "just a new porcelain".

I have no clue how Hipp thinks to implement it, much less in a way that allows push and pull to legacy Git users.

marmetio• 4 years ago

Isn't that the point of both projects?

Git is primarily a file format. There's also a canonical set of low-level command-line utilities. But you can create whatever interface you want. Most programmers and GUIs happen to use the canonical command-line tools, but that's hardly necessary. Fit would keep the necessary part of Git (the file format), and provide a different human interface.

mdaniel5• 4 years ago

Uses the Fossil user interface but it uses the low-level file format of Git. So that then you can work with Fossil’s interface but push and pull to legacy Git users.

I'm surprised that's the #1 magic-wand project for anyone. I'm not trying to yuck his yum, but am just surprised.

But the larger elephant in the room for any such effort is that Fossil's mental model is, by choice, way different from git. If they just want a new ui that isn't as "git sux," surely a new cli to kid-gloves git would be less work than trying to reinvent a whole new version control system, and that goes 1000x for trying to maintain wire compat with every git server in the world

ilyt3• 4 years ago

I don't think Git wire format changes all that often and old clients work with new servers just fine so I don't think that would be a problem.

But the larger elephant in the room for any such effort is that Fossil's mental model is, by choice, way different from git.

How? I've been glancing at docs but I don't see much difference, still a DAG of commits that are named after its hash like almost every other DVCS

From onlooker perspective the killer feature seems to be integration with project management tools vs "just" being a VCS

dahart3• 4 years ago

I’m surprised that’s the #1 magic-wand project for anyone.

It seems like Dr. Hipp is frustrated that Fossil hasn’t had mainstream adoption, and that git has, he’s talked about it quite a bit in Fossil docs and here on HN. Given that SQLite is very stable and adoption is an absolutely runaway success, I’m not too surprised his wish list now turns to Fossil.

Fossil’s mental model is, by choice, way different from git.

Is that accurate? It does come with a ton of extra features, but seems like the core version control model is quite similar conceptually, and differs more on implementation (SQLite backing instead of file system) and workflow dogma (Fossil’s moralizing of rebase and commit graph edits). The Fossil docs even start with “The feature sets of Fossil and Git overlap in many ways.” Fossil was released after but within a year of git being released, the timeline and docs and comments from the authors generally suggest it’s a reaction to git, taking the good parts and trying to, in the authors’ eyes, improve and fix their perceived problems with git.

Cyberdog• 4 years ago

Git’s UI is difficult, yes, but I’m sure far more people are familiar with it at this point than Fossil. It would be far more inefficient for me to learn Fossil’s commands for what I need Git to do than just continuing to use Git, at least in the short term (and with no guarantees it’ll improve in the long term). If anything, a tool which goes the other way around - allows me to interact with Fossil using familiar Git commands - seems more useful to me.

eptcyka• 4 years ago

This is the case for a lot of tools. There was a time that for a majority of people on the planet, handwriting a letter would be faster than typing it up too.

tiffanyh• 4 years ago

Super interesting given the purposeful differences between Fossil & Git.

https://www.fossil-scm.org/home/doc/trunk/www/fossil-v-git.w...

fbdab1032• 4 years ago

I think it is just being practical. Much to my chagrin, git has consumed the world. The Fossil tooling offers a reasonably slick (well, not the CSS) out of the box experience which would be nice to have elsewhere.

leeoniya• 4 years ago

the main reason i'm not all-in on Fossil is because you cannot squash commits; my WIP branch history is very messy, e.g. "looks to be working", "refactor", "typo", "optimize", "undo checkpoint". i don't need any of that junk on the `main` branch once the feature branch is merged.

it's really too bad, everything else in Fossil looks very compelling.

zie5• 4 years ago

You can hide those commits, you just can't delete/squash them. It's really not as big of a deal as everyone claims.

cryptonector• 4 years ago

Isn't there something like this already?

nigamanth• 4 years ago

It's been so long since Git has received a user interface update, that's why GitHub, GitLab and BitBucket had good ideas and made it big. Now I wonder if programmers will ditch Git and go to Fit.

adql5• 4 years ago

It's getting small usability updates from time to time.

Also GH/GL is popular because of everything else they do (issues, page rendering, now CI/CD) not because git CLI is too hard.

Improvements are welcome of course but the moment newbie hits merge conflict no amount of CLI niceness can get around them not knowing what to do with it and that's true regardless of DVCS. For "non-programmers" no CLI is good enough anyway.

qbasic_forever• 4 years ago

More recently he's been on the Changelog podcast in the last year and it's a great listen: https://changelog.com/podcast/454

clusterhacks• 4 years ago

There are great bits of discussion about SQLite being maintained and enhanced by a very small team - three people.

While I very much appreciate the safety of being a very small cog in a larger machine, I also wonder how my career might be more enjoyable and satisfying if I had chosen to limit my options to small teams where every single person on the team had a critical impact on the success of a project.

richardlblair• 4 years ago

There are organizations, even large ones, that prefer to work this week. Trying to set the maximum number of team members to 6 people including a PM and a designer. As you move through your career I highly recommend adding a question during the interview process to address this. It really does make a big difference on your quality of life, and you'll make some great friends along the way.

ilyt3• 4 years ago

That kinda entirely depends with what people you're stuck with. In 6-10 person team you can just avoid working with that one colleague you just can't gel with (even if they might be perfectly fine from competency standpoint).

I think medium-sized projects (say a team of max 10, or maybe 2 smaller ones) seems to be the most fun ones. You have enough manpower to do some bigger reorganizing if required but it isn't as big that rewrite of any part is years-long saga.

jerrygenser• 4 years ago

Is there a succession plan in place given the bus factor [1] with such a small team? Is anyone else concerned about maintenance if either a normal everyday medical issue came up or in the worst case something unimaginable?

[1] https://en.wikipedia.org/wiki/Bus_factor

xiphias2• 4 years ago

It’s not really a problem when it’s a 100% well tested software.

bob10292• 4 years ago

From my perspective, SQLite is stable enough to be considered "done". For our product, we could go back to builds from 2020 and none of our customers would be able to detect a difference.

SQLite is not something we wire up and expose bare-ass to the public internet. We don't worry about 0-day security issues in this context. It's more like a wrapper our product uses for structured disk access. Anything that would warrant urgent patching is in .NET vendor or in-house software piles.

If something horrible happened and the entire SQLite dev team was wiped off the face of the planet, I'd make a note to look at alternatives by ~2025.

rwbt• 4 years ago

Yes, Dr. Hipp mentioned it once that there is a foundation in place to take care of things and increase the bus factor. I think that was one of the conditions before Nokia signed their agreement to license SQLite for Symbian.

johnboiles5• 4 years ago

I've been trying to use SQLite3 in an embedded (ESP32 -- dual core 240MHz w/ qspi NOR flash) project and the performance has been surprisingly bad. Seems like the minimum amount of time to make a query (even for example, selecting from a table that does not exist, takes a minimum of 5ms). For a simple table with 10 short columns I'm getting 10s of milliseconds per record to SELECT. I can parse whole JSON files with the same data to RAM faster then I can query SQLite which seems wrong. SQLite is performing ~10x worse than I'd expected it would.

Anyone had experience running SQLite on low-power embedded platforms? Is this expected performance?

bob10292• 4 years ago

Could you share some source code around your implementation? Selecting from a non-existing table should be a very quick operation. I suspect you might be doing something like creating a new connection per command if you are seeing this behavior.

johnboiles5• 4 years ago

There aren't connections in SQLite like there are for network-connected DBs (eg MySQL). My simple code is like this:

  sqlite3_initialize();
  sqlite3 *db;
  sqlite3_open("/data/sqlite.db", &db);
  char *err = 0;
  sqlite3_exec(&db, "DROP TABLE table_name;", NULL, NULL, & err); // This takes ~5ms even when table_name does not exist

phonon• 4 years ago

Have you benchmarked the storage? That might be the bottleneck.

johnboiles5• 4 years ago

Yeah seems like it would _have_ to be storage related but my profiling tools are limited. I've run Doom on this same processor/flash combo and loading assets from the WAD files is on the order of 1ms and I can run Doom smoothly at 70fps. Not very scientific but that's part of the reason I was surprised by how slow SQLite is running.

I wonder if it could be doing multiple small writes on every operation -- I could definitely imagine small writes getting very costly, especially with wear-leveled file systems that might be moving around pages on even small writes.

fbdab1032• 4 years ago

If you have the RAM to load up a toy dataset, what happens if you utilize a :memory: database instead of disk? If the performance is the same, you know the problem exists outside of the storage.

johnboiles5• 4 years ago

Oh I didn't realize you could just tell it :memory:. That's a great tip

anitil2• 4 years ago

In the embedded world the specifics often outweigh the general. Specifics like - what storage are you using? What clock does it run at? What page size are you using (you'd want to pass this to sqlite for best results)? What locks are you using? Is there a 1mS undocumented sleep on startup to allow some peripheral to stabilise that accidentally got pulled in to the main code path (has happened to me - the initialize function didn't set the initialized flag)?

I'd expect closer to 5uS than 5mS, but I'm not that familiar with the ESP32 devices

johnboiles5• 4 years ago

Is your 5us number from experience? Have you run SQLite on embedded before? Just hoping to hear some real world performance numbers for any hardware remotely similar!

adql5• 4 years ago

I'm not getting 5us on gigahertz-clock CPUs. ESP32 isn't exactly demon of speed either compared to anything SQLite usually runs on

akiselev• 4 years ago

This depends entirely on how naive the sqlite implementation is - without exact implementation details and register values for the QSPI peripheral, it's hard to remote diagnose something like that.

Have you used a digital logic analyzer to figure out how long reading the page takes, how many pages SQLite reads for that query, and how much work SQLite has to do before it can fetch the next page? I haven't seen any ESP32 SQLite ports that can eagerly fetch pages, even though it's relatively simple to do with DMA (but would require some significant changes to SQLite because it's designed to be ignorant of OS preemption which is not the case in most RTOSes).

andrewl2• 4 years ago

In this conversation from July 2021 he said he’s going to write his own email server, which I would be very interested in:

https://corecursive.com/066-sqlite-with-richard-hipp/

I’m going to write my own mail server. I was making notes on that even as we were setting up this call. That’s a big problem, and that’s at least as difficult if not more difficult than writing a database engine, but I don’t want to be beholden to Gmail. I don’t want them controlling my destiny. I don’t want them controlling the record of all of my conversations. I want to control that myself, and so I’m going to go through a lot of pain and a lot of work and a lot of effort to come up with some solution that I can control myself. I can go out and lease a virtual machine out there in the cloud and run it myself and not depend on a third party to control my email.

mdaniel5• 4 years ago

I’m going to write my own mail server. I don’t want to be beholden to Gmail.

Wowzers, that's some RMS-"Web Over Email"-level "I have privacy concerns so I'm going to reinvent $foo" https://stallman.org/stallman-computing.html

eesmith2• 4 years ago

How do you go from "control" (used several times) to "privacy"?

Hipp not only wrote SQLite, which uses his own parser generator (Lemon) but he also wrote his own version control system (Fossil), and his own editor.

Perhaps it's more likely that he wants to be in control of the things which most closely affect him, rather than this being a privacy issue?

adql5• 4 years ago

Some kind of "all in one" where you have SQLite file with config (aliases/accounts etc) + maybe SQLite file per account for storage + some sane SMTP/IMAP config with DKIM and all the other crap spammers implement anyway but servers will spam you for if yu don't have it would be useful.

I've recently did a small server for private use (after google yeeted the free tier) and let's say it does require a bit of mail knowledge to set it up right.

That being said there is probably some container doing it, surely "easy mail selfhosting" isn't a new problem

Richard Hipp Speaks Out on SQLite (2019) [pdf]

Related Stories