SQLite's Automatic Indexes

https://misfra.me/2022/sqlite-automatic-indexes/

by preetamjinka • 4 years ago

216 40 4 years ago

misfra.me

chasil2• 4 years ago

I was not aware that SQLite creates temporary indexes in places where other databases would perform merge joins.

https://sqlite.org/tempfiles.html#transient_indices

SQLite also creates background indexes for ROWID, of which I was aware.

https://sqlite.org/rowidtable.html

Scaevolus• 4 years ago

rowid isn't really a background index, but rather the normal storage layout, with non-integer primary keys having a separate index structure.

yread• 4 years ago

would be cool to get a diagnostic message when that happens

chasil2• 4 years ago

For transient indexes, or permanent ones?

The transient indexes will be created in a separate, temporary file. They will be erased on a commit, rollback, or crash recovery. They will not persist.

masklinn6• 4 years ago

Per the linked mailing list post there’s no dust but there is a back channel through which you can be notified:

SQLite comes with instrumentation (specifically the SQLITE_STMTSTATUS_AUTOINDEX verb for sqlite3_stmt_status() - http://www.sqlite.org/c3ref/stmt_status.html) that can be used to detect when automatic indices are used and alert the developer through a back channel to this fact so that she can fix the problem with an appropriate CREATE INDEX. In other words, SQLite provides you with the tools to help you detect and eliminate the use of automatic indices.

SigmundA4• 4 years ago

SQLite uses a b-tree storage format for tables, so all tables are an index, similar to clustered indexs in MSSQL Server and Index Organized Tables in Oracle. The RowID is the true primary key unless you specify the table should use the actual primary key as the row identifier, this is legacy quirk of Sqlite.

Either way SQLite doesn't have heap tables like PG (which only has heap tables), all tables are clustered indexes.

winrid6• 4 years ago

This is not true in my experience at all. You have to create indexes.

Also, a btree is created for rowid, according to these docs.

https://www.sqlite.org/withoutrowid.html#:~:text=In%20an%20o....

Shorel• 4 years ago

Clustered indexes in MSSQL Server, Index Organized Tables in Oracle, and any table in MySQL using InnoDB as storage engine.

banana_giraffe• 4 years ago

There's an experimental module and CLI command in the normal shell to use this logic to dump out a suggested list of indexes to create for a given SQL command to speed it up.

parentheses• 4 years ago

I'd like to see this in Postgres and the like.

masklinn6• 4 years ago

Definitely would be nice in pg. I think it exists for sql server (likely also oracle), iirc the db can even be configured to optimistically create the index to try it out, and revert it if it’s bad.

One of the difficulties is indexes have a write cost so heuristically deciding on the trade off can be complicated.

jaggederest• 4 years ago

Cool details. Makes sense, since a database has to have index creation anyway, and if you're avoiding merge joins for code complexity reasons, why not use it to optimize.

An interesting project to manage indexes on postgres that seems tangentially related:

https://github.com/ankane/dexter

ayende• 4 years ago

That was interesting to read. RavenDB does something similar, if there isn't an index for the query, it will create one for you. The difference is that this is a persistent structure, which is reused across multiple queries and invocations.

winrid6• 4 years ago

RavenDB is interesting. Neat that it's in C#.

high_byte• 4 years ago

I think EXPLAIN is a little known feature that gives a lot of insight about database internals, especially in Sqlite.

Funny you posted this now, I just recently researched Sqlite internals for my own project. Maybe you will find it interesting: https://twitter.com/high_byte/status/1607853384123703296

Interesting blog, I love that you keep consistency writing for so long.

hinkley• 4 years ago

If EXPLAIN is little known then we have failed as educators, mentors and peers.

It does amaze me the gaps that people have, though.

haggy102• 4 years ago

I'm glad you said it because I was about to say the exact same thing.

krylon• 4 years ago

My gut feeling is that it's fairly well known, but few people use it. Including yours truly.

Then again, I have rarely run into performance issues with database engines. Either the ones I've had contact with (SQLite, MSSQL, Postgres) are very good at what they do (which I suspect they are), or my requirements were too puny for them to break a sweat (which I also suspect to be the case).

I did use MSSQL Studio's query analyzer once to fix a performance issue and was able to get the query in question from nearly an hour down to under a second. triumphant grin I don't remember the details of the query, but the solution was to use a hash join.

johndough• 4 years ago

EDIT: I misread. Nevermind!

dang• 4 years ago

"Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something."

https://news.ycombinator.com/newsguidelines.html

This could become a (very) good comment if you added enough explanation for people to understand what the issues are. But just posting a putdown isn't in the intended spirit of the site. If you know more than others, e.g. about nested loops in databases, that's great - but in that case please share some of what you know, so the rest of us can learn.

https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sor...

SCLeo• 4 years ago

Maybe I am missing something but I think O(N * N) is the correct complexity for a nested loop implementation. If you believe that is wrong, can you explain a bit more?

0x457• 4 years ago

Pretty sure nested loops are O(^2).

senderista2• 4 years ago

Well that assumes the same cardinality for both sides of the join, which may be far from the case, especially if selections can be pushed down to scans.

takeda• 4 years ago

I keep seeing posts like these, and I'm wondering, whether "lite" still applies. It feels like all it is missing is the network layer and it would have more features than original MySQL.

samkon2• 4 years ago

It’s not “lite”, it’s “ite”. SQLite, as if it were a mineral, like Azurite.

ckvjdonee• 4 years ago

That's not really it is it?

I thought they just dropped the extra L because it made more sense without it.

masklinn6• 4 years ago

The "lite" part is about being embedded, server-less, and single-file (at rest), it's not about being shitty, DBM did (and still does) that fine.

Furthermore this is exactly in furtherance of your stated goals: sqlite supports indexes, by using automatic indexes it avoids the need to implement merge joins and hash joins.

forinti• 4 years ago

It might make sense to flag in the metadata that an index might be needed; then again someone using SQLite will probably know very well which queries are being executed.

Also, most of the time joins are made on foreign keys and these should always be indexed. So I guess this should be a rare occurrence if the database is properly thought-out.

senderista2• 4 years ago

As others already noted, this is equivalent to asymmetric hash join, except less efficient. Constructing a hash table on the fly instead of a B-tree would almost certainly be faster to both build and query. You don’t need an ordered lookup structure for join indexes (unless it’s a merge join, which SQLite doesn’t do anyway).

SQLite's Automatic Indexes

Related Stories