SQLite builds for WASI since 3.41.0

https://wasmlabs.dev/articles/sqlite-wasi-support/

by rapnie • 3 years ago

111 47 3 years ago

wasmlabs.dev

aargh_aargh3• 3 years ago

What would you use this for? I don't get the whole picture and the article doesn't talk about it.

jraph6• 3 years ago

Apparently they use it for their tests.

We wanted to contribute upstream our patches for SQLite, as we were heavily relying on libsqlite for our tests of PHP.wasm with WordPress and Drupal.

rcarmo• 3 years ago

Well, you can use it to access SQLite files in WASM runners, of course. So you get a portable, platform-independent piece of code that can run anywhere.

Although it was a bit of a surprise to see that they started out with the toy CLI tool that my self and Nicholas hacked on for a weekend (I wanted to be able to run SQLite commands in a-Shell, which is an excellent iPad CLI environment).

samwillis• 3 years ago

So far the official WASM support from the SQLite project have only targeted WASM in the browser with a JS API. If I follow correctly this is about working with the SQLite team to upstream WASI support, which will eventually enable official support for it, and hence WASM on the server/cloud/edge and other WASI complement runners.

ridruejo• 3 years ago

Correct!

sgbeal2• 3 years ago

If I follow correctly this is about working with the SQLite team to upstream WASI support, which will eventually enable official support for it, and hence WASM on the server/cloud/edge and other WASI complement runners.

(sqlite team's Wasm/JS guy here)

That's the long and the short if it. We currently _actively_ target browsers only because our team has exceedingly limited bandwidth and has to be choosy with regards to where our dev time goes. Long-term, we intend to target all wasm platforms, but getting there is an ongoing process.

cfn• 3 years ago

The file of my desktop app is a SQLite db in disguise, I could use this to open these files in a web version of my app, I suppose. I know there is a javascript implementation of SQLite out there but maybe this has better performance.

fastball• 3 years ago

The goal of WASI is to add a standardized system API to WebAssembly, allowing you to access underlying systems (filesystem, networking, etc) in a homomorphic and secure way.

In practice, this means that you can compile your Rust/C++/etc code to WASM + WASI, and that code will run anywhere you have a WebAssembly VM. So the same code that accesses the "filesystem" in a browser context can also access the filesystem in a desktop/mobile/edge/etc environment.

You could also think of it as an attempt to turn WebAssembly into a fully-fledged JVM alternative. Compile once, run anywhere (including browser).

For this specifically, it means that SQLite can integrate with WASI APIs (instead of only Web APIs as before) so that an SQLite Wasm build can run in other contexts besides browsers.

aargh_aargh3• 3 years ago

What I don't get specifically is what's being implemented here with SQLite. Assuming WASI is an interface an application can call into for things like filesystem access, sockets, does this add SQLite as interface (analogously to sqlite.h) next to these as part of WASI? So SQLite will be "under" the interface, i.e. part of a standard runtime?

How much "batteries included" is this runtime (planned to be)? If I understood this correctly, then there would be a tradeoff with having more batteries included which would imply more functionality for applications but make porting to new targets more work.

rapnie2• 3 years ago

homomorphic and secure way

A lot of the sandboxing and security focus in Wasm makes sense. But the "capability-based security" mentioned at WASI spec [0] confuses me. For instance, when looking to what Cosmonic (an early adopter in Wasm fields) documents as capabilities [1] (e.g. 'messaging', 'http client', 'key-value store') it seems wholly different to concepts from the E programming language [2] by Mark S. Miller, that are finding their way in Cap'n Proto [3] and early work on the OCapN [4] unification effort. Is there any relationship to these, or are we looking at totally different approaches? Or is Cosmonic overloading the terminology of "capability"?

[0] https://github.com/WebAssembly/WASI#capability-based-securit...

[1] https://cosmonic.com/docs/category/capabilities

[2] https://en.wikipedia.org/wiki/E_(programming_language)

[3] https://capnproto.org

[4] https://github.com/ocapn/ocapn

angelmm4• 3 years ago

(Wasm Labs team member here) SQLite is a pretty popular database and it's a critical dependency for many different applications. By compiling it to Wasm32-wasi, you can add it to any WebAssembly module.

This enables a new set of possibilities for Wasm and SQLite. For example, now you can run a full WordPress application in the browser [1][2] / server [3] using the same Wasm module. Note that for the browser these projects use Emscripten [4], but in the future the same Wasm32-wasi module will work. A teammate gave a lightning talk about SQLite and WASI at KubeCon EU [5].

In general, any environment that includes a wasm runtime can potentially run applications that uses SQLite under the hood. Before, it wasn't possible.

- [1] https://wordpress.wasmlabs.dev/

- [2] https://developer.wordpress.org/playground/demo/

- [3] https://wasmlabs.dev/articles/running-wordpress-with-mod-was...

- [4] https://emscripten.org/

- [5] https://www.youtube.com/watch?v=E7tWtgf9V2s

i2cmaster• 3 years ago

I guess that's one way to prevent malicious plugins from destroying your site.

lioeters2• 3 years ago

these projects use Emscripten, but in the future the same Wasm32-wasi module will work

I'm curious to know more about this. Does it mean that when browsers support WASI, these projects will not need to use Emscripten? Or maybe use it for compiling to/with WASI but without the need for a runtime anymore.

Is there anywhere I can keep track of when/how this might happen?

raybb• 3 years ago

For legal reasons, we could not contribute directly (their patches)

What were the legal reasons?

foota3• 3 years ago

Looks like this: https://www.sqlite.org/copyright.html

Kind of weird, couldn't they use a CLA?

jraph6• 3 years ago

It's not enough to guarantee the code has not been copied from somewhere / can be released in the public domain. The only way to have this guarantee is to write it yourself.

est31• 3 years ago

A CLA only covers the things you wrote, and while you ensure in the CLA that you only contribute things you wrote and are allowed to license to them, it is possible that you lie, so there is the danger (although a small one), that contributors might upload proprietary code written by others. IANAL

jraph6• 3 years ago

Most likely what is documented at https://sqlite.org/copyright.html

SQLite is open-source, meaning that you can make as many copies of it as you want and do whatever you want with those copies, without limitation. But SQLite is not open-contribution. In order to keep SQLite in the public domain and ensure that the code does not become contaminated with proprietary or licensed content, the project does not accept patches from people who have not submitted an affidavit dedicating their contribution into the public domain.
All of the code in SQLite is original, having been written specifically for use by SQLite. No code has been copied from unknown sources on the internet.

They want to be freaking sure they are the original authors / they can release the code in the public domain.

Public domain is tricky and does not exist in the same way everywhere. Where I live, you just can't decide to put something in the public domain, you need to use something like CC0 [1]

[1] https://creativecommons.org/share-your-work/public-domain/cc...

x-complexity• 3 years ago

This is the one case where NIH syndrome is warranted: Entangled legal issues & legal dependencies are no fun for anyone.

stavros2• 3 years ago

Why can they not just license the thing under CC0? Won't someone contributing implicitly understand that their contribution will also be licensed under CC0? This feels like it's more complicated than it needs to be.

jraph6• 3 years ago

For legal reasons, we could not contribute directly, but we could discuss the required changes with someone from the SQLite team and then iterate with them by testing or commenting.

Ah, here's the catch! I knew SQLite didn't accept code contributions, I was suprised they managed to contribute anyway, and then this sentence clarified the situation. So they indeed did not manage to contribute their code itself, but the SQLite team agreed to write this code themselves. I wonder what was the SQLite team's motivation behind this cooperative work.

jillesvangurp• 3 years ago

https://www.sqlite.org/copyright.html

To summarize, instead of using one of the OSS licenses, the copyright holders simply declare the source to be in the public domain. In order to preserve that status they don't accept patches unless you submit some signed document that you agree with that.

To make things more complicated, they also use their a relatively niche version management system instead of git. Which would complicate making contributions (if they accepted them).

There's a popular fork that fixes all of these issues: https://github.com/libsql/libsql It is MIT licensed, on Github, and open for contributions.

Kind of a weird legal situation for a popular project like this that so many people depend on to have. Not judging; but it is odd. Seems like a lot of wasted efforts between users, would be contributors, and the people that forked this thing to address all that.

jraph6• 3 years ago

To summarize, instead of using one of the OSS licenses, the copyright holders simply declare the source to be in the public domain. In order to preserve that status they don't accept patches unless you submit some signed document that you agree with that.

Yep, see also this discussion about it in this thread https://news.ycombinator.com/item?id=36054521#36055014

mehdix• 3 years ago

Aren't all those potential limitations, in fact their strengths? They are eating their own dogfood, which works for them. That sounds great to me.

tarruda• 3 years ago

they also use their a relatively niche version management system instead of git

They use Fossil for source control: https://www2.fossil-scm.org/home/doc/trunk/www/index.wiki

It is a git-like system, but instead of storing objects on the filesystem, it uses a SQLite database.

Has some interesting features:

- Project Management

- Built-in Web Interface

- Single executable

hitekker• 3 years ago

Wait, when did libsql become popular? As I recall, the libsql startup burned some goodwill when they tried to publicly shame SQLite, after forking SQLite and barely changing its code[1]. Looking at the libsql repo today, it looks like the contributors are still on the payroll of that startup.

Beyond that, the insinuation that good OSS must conform to Github social norms is silly given SQLite's track record of success. The maintainers are smart to direct their limited time and resources into development and not into community management or clout-chasing.

[1]https://news.ycombinator.com/item?id=33099222 and https://news.ycombinator.com/item?id=33081159

KronisLV• 3 years ago

To make things more complicated, they also use their a relatively niche version management system instead of git. Which would complicate making contributions (if they accepted them).

The Fossil VCS actually has a page explaining why it was created, instead of just using Git: https://sqlite.org/whynotgit.html

Honestly, a lot of those points make sense, especially how Git is perhaps a little bit more complex and tricky to wrap one's head around than it should be, making you think more about the VCS than just what you want to do. To that end, I'd actually suggest that people have a brief look at Fossil, it even comes with a built in web interface and some common functionality out of the box (vs most folks needing to setup Gitea/GitLab or use GitHub/Bitbucket etc. separately): https://fossil-scm.org/home/doc/trunk/www/index.wiki

Of course, realistically, Fossil will be dead in the water for most, given that it's still niche and won't have integrations with any graphical software that some might want to use (e.g. SourceTree, GitKraken, Git Cola exist, but I'm not aware of a rich ecosystem of solutions for Fossil, something like Fuel seems dead https://fuel-scm.org/fossil/brlist), or even with any CI/CD server solutions.

gorgoiler• 3 years ago

From an administrative stance, I’m sure any project would happily consider a patch if you emailed it to them.

For local work, you can easily dump a codebase into your favourite version control system, work on it, then generate the final patch using your VCS’s tooling. Iterating towards a working change is as easy (or as hard) as exchanging patches between the feature developer and the committer of the upstream project.

It’s a bit old school, but it’s a good workflow to become comfortable with. I regularly patch my own private projects from my work account by emailing patches to my personal address. Once you compress and base64 a change it’s also humbling how little content goes into a piece of work!

(This isn’t a rebuttal to the parent post — more a call to arms that patch juggling isn’t as awful as others might think it is.)

sureglymop• 3 years ago

This is awesome. What I would really love to see is pandoc compiled to wasm and usable in a browser context. I tried before but since I'm unfamiliar with Haskell i didn't get very far.

gzurl2• 3 years ago

I didn't know about pandoc. Really useful!

ncruces• 3 years ago

I think this should come with the disclaimer that this completely eschews synchronization mechanisms used by SQLite.

Using this to read from a DB concurrently with other writers will produce garbage results. Using this to write to a DB concurrently with other writers will corrupt your DB.

WASI does not expose file locking primitives, and the unix VFS simply stubs them out in WASI builds: https://github.com/sqlite/sqlite/blob/4e8e33ba84e253878797d7...

SQLite builds for WASI since 3.41.0

Related Stories