Inserting 100k rows 66 times faster

https://xnacly.me/posts/2024/faster-inserts/

by todsacerdoti • 2 years ago

7 11 2 years ago

TLDR to get better performance, use bulk inserts instead of inserting one row per query. And (unrelated to DB perf) don't use the spread operator in JS for passing function args (the author says not to use the spread operator at all but you don't need to go that far)

ak39• 2 years ago

Thank you. Also, what database was in use?

throwaway1438292• 2 years ago

sqlite3, but I think bulk inserts would improve perf for other DBs too.

dopp0• 2 years ago

sqlite

vbezhenar• 2 years ago

JDBC has "batch updates" feature. You can use single inserts but they're sent as a batch of multiple statements to the database. Not sure about JS libraries, but this feature also significantly improves performance and probably easier to use and maintain.

For super fast inserts you can use postgres COPY feature. But it has to be implemented in the driver.

NathanaelRea• 2 years ago

I wonder how `.import` would compare vs creating a huge batch insert string in memory.

kikimora2• 2 years ago

I remember SQLite have been using inserts to implement import under the hood.

lmz• 2 years ago

I wonder if they tried prepared statements - parsing 100k identical insert statements surely is avoidable.

xnacly• 2 years ago

I didn’t even think of that, I‘m gonna have to try them out and compare against the two other implementations.

avinassh• 2 years ago

you might like my post - Towards Inserting One Billion Rows in SQLite Under A Minute [0]

[0] https://avi.im/blag/2021/fast-sqlite-inserts/

Inserting 100k rows 66 times faster

Related Stories