Vanadium – Create distributed applications that can run anywhere

by chuhnk • 10 years ago

124 50 10 years ago

sethammons• 10 years ago

Quickly looked over this and the thing that caught my eye is that it says you can set up distributed systems that are available offline through device to device communication. At least, I think that is what it does. I could not quickly find more information on that. A diagram or something would be helpful to visualize this, with an explanation of the protocols that are used. Maybe I missed the examples or case study section.

oori• 10 years ago

this vs. firebase for sync ?

azinman28• 10 years ago

Firebase doesn't have offline p2p at all, let alone sync.

That said, right now I'd build using firebase unless you have very specific needs.

niftich2• 10 years ago

After 10 minutes, here's what I found out:

- The authors are either Google and/or heavily affiliated with Google

- On their website, the two highlighted projects/products are Syncbase and Vanadium Core.

- Syncbase is a distributed, works-offline datastore, with well-defined ways of doing reconciliation. Think, git merges, or collaborative docs, or y'know anything with vector clocks but approached from the side of the UX they want to target.

- Vanadium Core is "the discovery, RPC, and security layer underlying Syncbase", including its custom binary data serialization format 'VOM' and its IDL called 'VDL'.

the_duke2• 10 years ago

How did you find the association with Google?

But you are probably right: page is in material design, the repos are mostly Dart or Go.

sly010• 10 years ago

Their intro video is done by 2 Google engineers.

niftich2• 10 years ago

I'm so glad you asked!

- I read the Vanadium ToS [1]

- In the video embedded from the page on 'Syncbase' [2] everyone shown works at Google.

- At this point, the association with Google looks very likely. I searched for 'syncbase google' and found a ton of intriguing stuff, including the prospect that it's being used in production settings by Google.

- I searched for 'vanadium google' and found this Twitter thread [3]

[1] https://vanadium.github.io/tos.html

[2] https://vanadium.github.io/syncbase/

[3] https://twitter.com/chuhnk/status/685455321465237504

weitzj2• 10 years ago

I wonder if Vanadium is the project Jeff Nichols http://www.jeffreynichols.com is talking about.

  Quote:
  I am a Staff Research Scientist at Google managing a team of roughly seven people at the Mountain   View campus.
  I work on a project that you haven't heard of and that I can't talk about, but, needless to say, we're doing really cool stuff that you'll be excited about when it's finally released.

weitzj2• 10 years ago

Also something about Syncbase on Youtube.

https://www.youtube.com/watch?v=2cHzd8pBYmU

the_duke2• 10 years ago

Hilariously awkward video.

But the key points:

-) All data is encrypted both on the wire and "at rest" (presumably on disk)

-) All options are versioned to allow merging

-) Provides builtin policies for resolving conflicts (eg last one wins)

-) Handles structured data (nosql api) and binary data (blob api)

-) Fine grained ACLs for permissions

-) Apparently offers p2p syncing without a central server

inopinatus4• 10 years ago

Syncbase: "Offline-first and peer-to-peer, Syncbase provides local storage that opportunistically syncs data".

I have an application for that _right now_. Syncbase doesn't look ready though, does anyone know of an alternative data store that ticks all those boxes? e.g. I was looking at Cognito, but it lacks peer-to-peer.

aboodman• 10 years ago

You should take a look at Noms:

https://github.com/attic-labs/noms

Happy to answer questions here or in our Slack channel.

marknadal2• 10 years ago

ours https://github.com/amark/gun .

Pouch https://pouchdb.com/ .

You could even do http://webtorrent.io !

azinman28• 10 years ago

Nothing else does what vanadium does, especially at its sophistication level. I don't know if I'd use it in production yet unless you are willing to put in some engineering resources, but you're unlikely to find something else as well made elsewhere. This is the product of a very elite team... which means it's also very complicated to use.

HodGreeley3• 10 years ago

Couchbase Lite does all this, and is a mature, shipping product with many real-world deployments (including peer-to-peer as a primary use). It's a JSON document database, with full map/reduce indexing/query capabilities. It uses MVCC (similar to git) to handle offline first + conflict resolution.

Couchbase Lite can be used stand-alone, or as part of the Couchbase Mobile stack, which includes Sync Gateway and Couchbase Server. Everything is open source, too.

I'm happy to answer questions here, you can ping me on twitter, or through our forums, etc. Hod @ Couchbase

noahl2• 10 years ago

The core repository has an AUTHORS file, and the only entry is "Google, Inc."

Disclaimer: I work for Google, but I have no idea what this is.

solipsism• 10 years ago

You keep using that word ("disclaimer"), I do not think it means what you think it means.

Mao_Zedang2• 10 years ago

I didnt know io sold single letter domains, sexy.

inopinatus4• 10 years ago

So does .org, see http://www.project94.org/.

Mao_Zedang2• 10 years ago

I wonder how much they cost.

marknadal2• 10 years ago

Wonderful to hear about more P2P database sync systems!

I would love to hear how you compare to Deepstream and us ( http://gun.js.org/ ), and http://Horizon.io ?

Edit: Also how on earth did you snag `v.io`? That is amazing and I thought single-letter domains weren't allowed?

HodGreeley3• 10 years ago

I'm assuming Horizon is in limbo along with RethinkDB?

azinman28• 10 years ago

I ported vanadium and syncbase to iOS. Happy to answer questions. Yes indeed this is a google project but is now being taken on by its original lead, and former head of search infrastructure, now at Grail bio.

dunkelheit2• 10 years ago

The project looks interesting but something about it rubs me the wrong way. Which organization is behind vanadium? Is it google? Then how is grail bio involved? Or is it a side project of google employees who now work at grail bio? I'm confused.

Also, why are syncbase sources not on github?

azinman28• 10 years ago

Everything is open source and on GitHub. This is a rare project that was 100% open source from the beginning.

danielsju65• 10 years ago

It's all there. Just look under Go/Java/Swift https://github.com/vanadium

makmanalp• 10 years ago

Can anyone comment on the disadvantages of last-write-wins? Either that or how tricky it is to always have to define your own merge policies?

tgma• 10 years ago

The cool thing I really like about Vanadium is a unified worldview it provides with the ability to access every object in the Vanadium ecosystem through its powerful naming/mounting mechanism. In Vanadium, you can "mount" things on your namespace (think Unix VFS) on a hierarchical namespace and invoke RPC methods on the objects. The method invocation is mutually authenticated with ed25519 keys and end-to-end encrypted on the wire. I sometimes jokingly called it "Plan 10". It is certainly not just another "synchronizing database" though that is a subset of it.

Disclosure: I very briefly worked on it at Google and I can attest the team behind it is brilliant beyond imagination.

curiousmonkey90• 10 years ago

It seems like it is no longer under active development. I hope I am wrong, but does anyone happen to know?

It looks like a lot of the Googlers previously working on the project has now moved to Grail bio.

danielsju65• 10 years ago

I was on the team working on Vanadium @ Google and have since moved on to Firebase.

I can't speak to what level of commitment to the project those that have moved on to Grail have; you should direct questions to https://news.ycombinator.com/user?id=cosmosnicolaou or refer to his comments.

I actually recently gave a presentation / live demo of Vanadium at Drinks on Tap SF. It's not production ready, but a cool glimpse into a possible new paradigm ;)

leeeroyjenkins• 10 years ago

"a lot"? who else do you see besides Cosmos?

rixed• 10 years ago

Is this similar to Irmin (https://github.com/mirage/irmin) ?

allan_s• 10 years ago

In the documentation I see

  Note
  Requiring a cloud Syncbase is temporary. We are planning to allow the API to be used without a cloud Syncbase soon.

Does it mean that the following use case will be supported:

I have two devices in the same LAN with no internet access, and they will be able to synchronize ?

azinman28• 10 years ago

Correct.

botw• 10 years ago

besides wifi, what other type of p2p communication channels are supported in Vanadium? bluetooth? USB? radio?

danielsju65• 10 years ago

Prototypes also use BLE / Bluetooth Classic but the vision being that it's transport agnostic.

cosmosnicolaou• 10 years ago

Let me provide a brief history/overview and answer the questions that have been asked as best I can. I started vanadium about three years ago to build a secure rpc and naming system that would make it easier to build distributed applications, including those on mobile and embedded devices/IoT. Security was a central goal. Distributed, synchronzsed storage has long been a pain point, especially when devices are often disconnected and we built syncbase to address that need.

I left Google in May of this year and joined Grail, where we're developing an NGS-based blood test to detect cancer early, when it can still be cured. We're using Vanadium for the infrastructure we need to support our internal development and analysis pipelines. Since I left Google Vanadium was split into a 'core' component (security, rpc, naming) and syncbase separately. The core component is very stable, syncbase is likely stable but not quite complete. We're currently using only Vanadium core.

To answer some point questions: - yes, this is the project Jeff Nichol's is referring to - and here, https://arxiv.org/pdf/1604.06959v2.pdf

Vanadium – Create distributed applications that can run anywhere

Related Stories