Show HN: Sqinn-Go is a Golang library for accessing SQLite databases in pure Go

by cvilsmeier • 3 years ago

113 61 3 years ago

eatonphil3• 3 years ago

If you really want to use SQLite without anything that isn't Go (since this project involves forking and communicating with a non-Go SQLite in a separate process over pipes), there's a Go translation of SQLite's C source. :)

https://gitlab.com/cznic/sqlite

https://datastation.multiprocess.io/blog/2022-05-12-sqlite-i...

scosman3• 3 years ago

What’s the case against cgo for SQLite? Just the usual cgo performance overhead?

It seems like a pretty good cgo use case: a decent amount of work, which is typically slow enough that cgo overhead isn’t perf critical (because DB usually means disk reads), a super robust and well tested C library with a super well maintained cgo wrapper (mattn).

llimllib2• 3 years ago

the other pain point I know of is that it's hard to cross-compile. You can do it with zig[1], but it's still not pleasant.

1: https://zig.news/kristoff/building-sqlite-with-cgo-for-every...

infogulch• 3 years ago

mattn's sqlite3 is probably the most ideal use case for cgo imaginable. But it can still be annoying to set up cgo to build across many platforms.

I think Go should just pull a Zig and just embed a full blown C compiler into go build.

parhamn2• 3 years ago

Just the usual cgo performance overhead?

No, the performance is certainly orders of magnitude faster than serializing over std streams on a subprocess (c ffi calls in cgo are 10s of nanoseconds).

But one of the big draws of golang is the write-once-compile-anywhere toolchain and calls cgo makes that harder.

cvilsmeier6• 3 years ago

Sqinn author here. Nothing against CGO, but I develop/deploy on Win/Linux, and cross-compiling CGO is very painful. Regarding performance: To my own surprise, Sqinn out-performs mattn (and others) for normal workloads, see https://github.com/cvilsmeier/sqinn-go-bench

hamandcheese• 3 years ago

Would it be possible to link a pure go codebase to pre-compiled sqlite binaries and not need to worry about cgo when cross compiling?

ncruces8• 3 years ago

The dynamic linking story in Go is complicated at best, particularly if you shun cgo, because you commonly need to cgo into ld.so to use dynamic linking.

But there's this if you wanna try: https://github.com/ebitengine/purego

kunley2• 3 years ago

Yep, confirming that cznic's pure-Go modernc.org/sqlite works great.

tedunangst3• 3 years ago

The logic of easy cross compiles doesn't really hold up for go translated SQLite. It depends on a huge pile of per platform support code, of varying quality. If you're only going to target known working platforms, may as well use cgo and a known working cross compiler.

ncruces8• 3 years ago

Does compiling to WASM, using a cgo free WASM runtime, and replacing the OS layer (VFS) with portable Go code count?

That's the elevator pitch (so far) for: https://github.com/ncruces/go-sqlite3/tree/main

eatonphil3• 3 years ago

I don't think cznic's argument is about cross compiling but just being able to avoid cgo. Since some people really don't like cgo.

JodieBenitez• 3 years ago

This is so cool... I see it has FTS5 support too. Now is the time to rewrite some of my apps into independent binaries

tgv2• 3 years ago

It’s not 100% Go, though. It forks a process that manages the sqlite file. Since the communication with it is written in go, it avoids using cgo.

guappa2• 3 years ago

Wow, that sounds incredibly inefficient!

Animats• 3 years ago

If you're going to run the database in a background process, you may as well run MySQL or one of its derivatives, or Postgres. More concurrency.

cvilsmeier6• 3 years ago

Wow, that sounds incredibly inefficient!

That's exactly what I thought, too. But, to my own surprise, it's as fast as CGO solutions, in most cases even faster. You may check here: https://github.com/cvilsmeier/sqinn-go-bench

twodave• 3 years ago

Nice work! It's fun to see other people hacking away at Golang projects. I've been contemplating how to write tests for my own project, and this has given me some ideas.

cvilsmeier6• 3 years ago

Thank you! Nice to hear :-) Use the code any way you want, it's unlicensed.

nighmi• 3 years ago

Unlicensed is the same as fully copyrighted. There is a presumotion of ownership. Licenses in this case serve to clarify allowable uses. Without a license, nothing is allowable as you maintain the right to do anything within a copyright holder's legal right.

https://www.synopsys.com/blogs/software-security/unlicensed-...

tw19842• 3 years ago

This is pretty much a os/exec + stdin/stdout redirection library.

I've been using cgo for accessing sqlite for ages, never ever had any trouble. cgo is not remotely as bad as what people might believe. give cgo a go, and you will realize that it saves your time, let you focus on your projects rather than non-senses like how to use stdin/stdout redirection to bypass cgo.

Show HN: Sqinn-Go is a Golang library for accessing SQLite databases in pure Go

Related Stories