UnQLite – An Embeddable NoSQL Database Engine

https://unqlite.org/

by rayascott • 8 years ago

85 56 8 years ago

unqlite.org

dfabulich2• 8 years ago

From the FAQ:

"7. Is UnQLite thread-safe

Threads are evil.[1] Avoid them.

UnQLite is threadsafe and full re-entrant. But in order to be thread-safe, UnQLite must be compiled with the UNQLITE_ENABLE_THREADS compile time directive defined. If you are unsure if the UnQLite library you are linking against is compiled to be threadsafe you can call the unqlite_lib_is_threadsafe() interface to find out."

[1] http://www.eecs.berkeley.edu/Pubs/TechRpts/2006/EECS-2006-1.... "The Problem with Threads"

"Threads are evil" is not a useful remark to make in response to the question "is this library thread-safe"

dominotw3• 8 years ago

sqllite has the same warning in their FAQ's :D

https://www.sqlite.org/faq.html#q6

blattimwind• 8 years ago

SQLite can actually do grouping and sorting (iirc) in parallel, though it's disabled practically everywhere (Linux distros and other environments shipping SQLite seem to pick compile flags kinda arbitrarily, e.g. json1 and fts4/5 are fairly widespread on Linux, but not included in e.g. Python for Windows).

dfabulich2• 8 years ago

I think the FAQ needs another question: when should I prefer UnQLite over SQLite? When should I prefer SQLite over UnQLite?

sephware2• 8 years ago

That one seems straight forward: when you want SQL use SQLite, when you want NoSQL use UnQLite

xellisx2• 8 years ago

From what I quickly gathered, it's just a Key-Value storage mechanism.

sitkack2• 8 years ago

Choices are often made across many dimensions.

bvinc• 8 years ago

The question I'm more interested in is: what advantages does this provide vs creating my own document style schema in SQLite. Using the JSON1 extension to SQLite, you can even query inside of JSON blobs.

dominotw3• 8 years ago

From reading the docs, I could spot the following differences

1. sqllite is 'sql' interface, unQlit is k/v.

2. Sqlite uses btree, unQlite is linear hash( doesn't support range queries).

otabdeveloper2• 8 years ago

An Embeddable NoSQL Database Engine

So, std::unordered_map?

gmueckl• 8 years ago

Looks more like an out of core unordered_map<string,string> with thread safety and persistence to me, if you really want to make that analogy.

coleifer5• 8 years ago

Not really. I mean, yes, but it also has a concept of document storage and a php-like scripting language they call jx9.

snissn• 8 years ago

this is great - in my opinion there's a shortage of key value type data storage engines and there needs to be more competition in this regard. There simply aren't enough free/open source data engines to meet the demand of today's software.

few quick thoughts - how does this compare to leveldb? How would it perform in something like geth or parity as an ethereum client back end data engine in compared to leveldb? do you have a python interface library?

UncleEntity3• 8 years ago

Found a python2/3 bindings library on the github -- https://github.com/coleifer/unqlite-python

It's based on cython apparently, might dust off pybindgen and see if I can do better...

coleifer5• 8 years ago

I'm the author of that library. I've worked to make sure all the unqlite features are available (transactions, cursors, jx9 vms, collections) and the interfaces are pythonic. Performance is good. Why not submit issues or patches instead?

manigandham• 8 years ago

Really? There seem to be a ton of key/value engines on the market, especially because it's such a simple model to implement.

dfox• 8 years ago

There is plethora of embedded K/V stores, but most of them only solve the simple to implement parts. That means either no concept of concurrency control or global RW-lock, zero resiliency to power failures, no support for atomic transactions...

If you want on-disk K/V store for critical data you are pretty much stuck with BerkeleyDB.

interfixus• 8 years ago

The licensing is not altogether obvious. There seems to be a "one time fee" for downloading. The name sort of implies a connection with SQLite. No such connection exists. There is a parent company, name of "Symisc Systems". Which appears to be just one guy somewhere in France or Tunisia who likes to call himself "we". The code is not too kindly treated by several who have actually tried to use it.

I think I'll pass.

tyingq• 8 years ago

https://github.com/symisc/unqlite/blob/master/LICENSE

It does use a fair amount of sqlite source.

MaxBarraclough• 8 years ago

Charging for a 'production build' is certainly unusual.

I'm reminded of when OpenBSD used to sell CDs.

UncleEntity3• 8 years ago

This production-ready package is available for a one-time fee of $20.

Don't fault them for trying to make a bit of cash but going to have to pass since I'm too poor to spend $20 to see if it's a viable replacement for a little project I built on top of tinydb that's getting pretty slow since the db has grown past what a pure python db can comfortably handle.

Or...I suppose they could provide a link from their download page to their github repo for us somewhat lazy folks -> https://github.com/symisc/unqlite

evil-olive• 8 years ago

Is there any documentation of the testing/verification process used in UnQLite? One of the things that gives me confidence in SQLite is https://www.sqlite.org/testing.html and it'd be interesting to see what approach UnQLite takes to the same problem.

symisc_devel• 8 years ago

Hey all. I worked on the UnQLite development team back into 2013 ~ 2014. It was designed to be the primary storage for a home targeting router manufactured by a French tech conglomerate.

The API is broken down into two independent layers plus one for various utility stuff like cross platform mmaped files, RC4 random numbers generation and so on. The first layer serve as a general purpose key/value store for the host application blobs (i.e binary or text data). You can store whole disk files inside a single database to generate a cross platform TAR archive for example.

The second one is the document store layer similar in concept to what MongoDB offer but without the networking layer since everything run in the same process of the host application. Both layers are transactional and are able to recover after some external failure thanks to the SQLite journaling mechanism from which UnQLite is based on.

Since its release in 2013, there was four major bugs including two critical that involves data loss under certain load. All known data corruption bugs were fixed in the last release and no major bugs were discovered a year or so.

The library enjoy particular success among Python and C/C++ developers especially Chinese that used to bombard the original developers with various internal aspect of the library.

ankrgyl3• 8 years ago

This is a good example of someone twisting the term "serverless" to capture attention:

Serverless Database Engine
Most NoSQL database engines (i.e. MongoDB, Redis, CouchDB) are implemented as a separate server process. Programs that want to access the database communicate with the server using some kind of interprocess communication (typically TCP/IP) to send requests to the server and to receive back results. UnQLite like SQLite does not work this way. With UnQLite, the process that wants to access the database reads and writes directly from the database files on disk. There is no intermediary server process.

The fact that it's embedded doesn't actually have anything to do with what popular "serverless" architectures are designed to solve, i.e. writing code that does not need to run in a specific server process to be invoked on demand. By their definition of serverless, pretty much any library is "serverless"...

Edit: This is referring to "Classic Serverless" (https://sqlite.org/serverless.html) not "Neo-Serverless" architecture (my definition is more like "Neo-Serverless"). Thanks parhamn for pointing out the distinction.

parhamn2• 8 years ago

That doesn't seem fair. The word you quoted isn't even in the section you provided. The use of "server" is interchangeable with daemon/isolated-process here. That section seems like a decent explanation of daemon vs embedded for the junior dev.

ankrgyl3• 8 years ago

Huh? It's literally in the "Serverless Database Engine" section. I can update the quote to include the header.

teraflop• 8 years ago

The UnQLite website has described it as a "serverless database engine" since at least 2013 (according to the Wayback Machine), long before it acquired its current buzzwordy definition.

If you ask me, using the word "serverless" to mean "a library that doesn't need a separate server process" makes a whole lot more sense than "a system that runs your code on a server that you don't control".

objektif2• 8 years ago

So what are the advantages of this over say python dicts? Genuinely asking thanks.

Buge2• 8 years ago

The data in a python dict disappears when you turn your computer off. The data in UnQLite doesn't disappear.

objektif2• 8 years ago

I could pickle the dict to a file right?

koenigdavidmj• 8 years ago

Sounds pretty similar in target usage to LMDB.

dominotw3• 8 years ago

thats log structured storage?

hyc_symas3• 8 years ago

No, LMDB uses a copy-on-write B+tree.

https://symas.com/lmdb/technical/

newscracker• 8 years ago

The name though — what is it supposed to mean? “Un” as in “One” in French with QL Lite or “Un” as in a prefix for an opposite in English with QL Lite (like “No QL Lite”)?

JulianMorrison• 8 years ago

Un SQL = no sql

And a pun on SQLite.

reelly• 8 years ago

a pun on SQLite.

And a bad one at that.

I normally don't chime in on HN, but this pun is a crime against humanity.

go_prodev• 8 years ago

Not sure, but my first thought was Un(structured) Query Language

teddyh• 8 years ago

UnQLite is a standard key/value store similar to BerkeleyDB, LevelDB, etc. but, with a rich feature set including support for transactions (ACID), concurrent reader, etc.

This is slightly interesting, since the big reason I personally prefer applications using SQLite over those using BerkeleyDB (which used to be very common, btw) is that I’ve had lots of problems in the past with corrupted BerkeleyDB files. Like, almost all the time; if it used BerkeleyDB, it would sooner or later get corrupted. If UnQLite can be a “non-corrupting BerkeleyDB” for those who don’t actually need SQLite, it can serve an (admittedly niche) purpose.

coleifer5• 8 years ago

Oh it corrupts... https://github.com/symisc/unqlite/issues/41

NonEUCitizen• 8 years ago

Is there an "amalgamation file" like SQLite has for easy inclusion into other projects?

rayascott• 8 years ago

Yes. unqlite.h/c in the root at https://github.com/symisc/unqlite

coleifer5• 8 years ago

I authored a python binding and have some experience that may be relevant.

The library seems poorly maintained in my opinion and might be abandoned. There have been critical bugs leading to data loss. They have a similar project vedis which is like an embedded redis that I think is completely abandoned.

For the love of God just use SQLite if you want an embedded database for structured data, even if it's just eav. If you just need key/value there are battle-tested, well maintained options.

https://github.com/symisc/unqlite/issues/41

oscargrouch• 8 years ago

For the love of God just use SQLite if you want an embedded database for structured data, even if it's just eav. If you just need key/value there are battle-tested, well maintained options.

You can also use the SQLite Disk Btree as a key-value data store. I've just implemented a solution that does so in C++ with a LevelDB-like api.

You should use the backend Btree api (the same the VDBE/SQL engine uses to store stuff) and create 'tables' the SQLite uses for indexes. The integer-key based is used for ordinary SQL table .. (1 PK = 1 Record) and the blob-key type, where its agnostic about whats on the key. (Thats the one you use to create the KV store)

There are a couple of details, to make it right, but its not that hard, and its a great solution.

hyc_symas3• 8 years ago

The Btree engine in SQLite is quite inefficient, and also unreliable on its own. (Needs the transaction log support to provide reliability.) LMDB blows it away.

https://github.com/LMDB/sqlightning

cwyers• 8 years ago

You've posted the same two-year-old closed issue twice on this thread. You're coming off as someone with an axe to grind.

sitkack2• 8 years ago

People often get passionate about data corruption.

coleifer5• 8 years ago

I just want to share my experience, which is extensive.

maxpert• 8 years ago

Agree here with the comment nothing compares to the solid codebase of SQLite. This project in my opinion trys to win your trust with QLite in name and does not deliver same quality.

dicroce• 8 years ago

Unqlite doesn't support in order traversal of data because the default (and only included storage engine) is based on hashing.

UnQLite – An Embeddable NoSQL Database Engine

Related Stories