CondensationDB: A general-purpose distributed database with end-to-end security

https://github.com/CondensationDB/Condensation

by Malexik • 5 years ago

94 33 5 years ago

Malexik15• 5 years ago

Author and contributors are here for your questions

indymike• 5 years ago

Interesting. Would love to try this out, but I generally try to avoid Java for personal side projects. Is the plan to make a Javascript client or a full Javascript port?

Malexik15• 5 years ago

Thanks, yes it may start this week from the enthusiasm of a contributor we just met, he begin to help to port it to Typescript/Javascript.

bsenftner• 5 years ago

I find this quite interesting, but do all my development in the C/C++ ecosystem. Is there anything similar there?

Malexik15• 5 years ago

Not yet planned but its definitely something we want to have. we onboard everyone who want to port the code, so I hope someone will come with this idea soon.

gritzko3• 5 years ago

http://doc.replicated.cc/%5EWiki/ron.sm

RON's reference implementation is C++. Op based sync, CRDT types inside. RON per se is the notation everything is built around.

ben509• 5 years ago

What's the story for deletion? Is that done through something like merge(record, delete_token) -> delete_token?

Malexik15• 5 years ago

Basically you have a document with all the references to objects, and if you remove the reference the object will be deleted after a certain timeout (you can set it for your specific case).

gritzko3• 5 years ago

May you introduce yourself, please? Cause the accounts look like throwaways. That's weird.

Malexik15• 5 years ago

Yes, basically here is Thomas the main author https://viereck.ch/thomas/ and I am working with him since 7 years https://www.linkedin.com/in/alex-mouradian/

We used to work without being connected, I got your point we will put a presentation of ourselves in this github page or the website.

cdbex• 5 years ago

Hi, I tried running the example from https://github.com/CondensationDB/Condensation-java/tree/mas... with the HTTP store from https://condensation.io/servers/php-files/. However, it seems that the later is outdated. Is there a functioning store available?

Malexik15• 5 years ago

Hey, can you send us an email? I can connect you to Thomas for the installation of the latest version. It will be a good exercise to guide you through.

Fiahil3• 5 years ago

Can you tell us a bit more about the "conflict free merge" ? Is it based on CRDTs ? How does it work ?

Malexik15• 5 years ago

Yes exactly, it's based on CRDTs and there is a strategy out there to mark the entries with a timestamp to figure out which one are the latest. An object may contain many entries and when they are read by the client they are just compared one by one to find the union, or the latest version.

The beauty of it is that the algorithm decide on how much entries to put in objects to ensure that only the data that is changed is sent on the network and compared on the other client. That's why we call it Condensation.

Fiahil3• 5 years ago

So, it's like a giant "grow only set" shared between as many nodes as you want and sending diffs on the network? I find it difficult to understand the possible applications* and the link with the article in the readme (https://www.inkandswitch.com/local-first.html).

*: I see that it could be used as backbone for an end-to-end encrypted messaging system, but what would it change for me if I were reading a remote API, running a multi-instance web app or parsing CSVs to train ML models on them ?

dfeiertab• 5 years ago

I'm not sure if I have fully gotten it correctly. It's a distributed database that syncs without conflict. So, cool for collaboration tools. What other things would you do with it?

Malexik15• 5 years ago

Yes, it excels at synchronization, you could just put a server synchronized and you have your backup, or let each user of an application have his own server which is synchronized with others (e.g., for a smart lock system). That's something pretty useful for privacy or if you have connectivity problems like in a mesh network with interruptions.

For the app itself, its really about getting end-to-end encryption be able to use the app while offline without loosing data.

jarym• 5 years ago

This looks really neat, cannot wait to read the paper to see the details.

The other research db I saw recently I liked was sirix (https://github.com/sirixdb) - I can certainly see some room for cross-over between these two.

Malexik15• 5 years ago

That's cool for querying, I will have a deeper look thanks

lifty2• 5 years ago

People that are interested in a similar feature set should check out https://github.com/attic-labs/noms and the SQL fork of Noms, https://github.com/dolthub/dolt

Malexik15• 5 years ago

Interesting one, I think the difference is that you dont really control where your data is, it's like IPFS? I am surprised they managed to have queries, I will dig deeper into that thanks.

lifty2• 5 years ago

It's just a db, so your data is where the db is running. You could have a central db storing your data and your local client could sync periodically, or you can have a p2p architecture if you want. I use Noms as an embedded db in my project, where syncing happens between p2p nodes.

expliced• 5 years ago

How does data synhchronization and conflict resolution work in CondensationDB? I expected to see something about OT or CRDT since it says it can be used for building collaborative applications such as Google docs (something like Docs is not possible without OT or CRDT).

Malexik15• 5 years ago

Exactly, look my reply to Fiahil a bit below.

LukeEF2• 5 years ago

Wow - great to see more work in this area. And interesting to see that the Swiss Government is supporting.

We’ve been working on a graph/document database with these kind of collaborative revision-control features over at TerminusDB (https://github.com/terminusdb/terminusdb). I think it is the wave of the future. And interestingly we also came out of a European Commission backed project.

Our approach to distribution is to use delta encoding and succinct data structures. We borrowed a fair few ideas from Git. Might be interested to read our storage layer white paper: https://github.com/terminusdb/terminusdb/blob/master/docs/wh...

Conflict free merges sound fantastic - not an easy road! Good luck.

Malexik15• 5 years ago

Thanks a lot, I heard about TerminusDB in the past, the capacity to do queries is interesting, that's something Condensation doesn't provide as it is on the client side and because at least now there are no tools to do it on the store.

I would be very happy to keep in touch and you have to give me the secret of these European funds.

LukeEF2• 5 years ago

I'm afraid the European monies are all spent! Was in a former life when we were a university progect called ALIGNED. But got us off the ground at least.

Very happy to stay in touch! We have a Discord server if you are interested.

abdul169• 5 years ago

Nice jobs

CondensationDB: A general-purpose distributed database with end-to-end security

Related Stories