Olric: Distributed in-memory data structures in Go

https://github.com/buraksezer/olric/releases/tag/v0.3.0-beta.1#=

by mastabadtomm • 6 years ago

217 27 6 years ago

github.com

hu3• 6 years ago

Perhaps a better link would be the README which explains what's the project about and its features.

https://github.com/buraksezer/olric/blob/master/README.md

whalesalad• 6 years ago

Very curious about the origins of this tool and how it is used by the creator. Looks very comprehensive, but some real world examples of use and performance would be great.

ddorian432• 6 years ago

Distributed in-process cache can make things very fast. See vimeo with group cache: https://medium.com/vimeo-engineering-blog/video-metadata-cac...

Or in-process distributed rate limiting: https://github.com/mailgun/gubernator

https://github.com/golang/groupcache: groupcache is in production use by dl.google.com (its original user), parts of Blogger, parts of Google Code, parts of Google Fiber, parts of Google production monitoring systems, etc.

There was also a paper by google that I can't find right now.

Edit: Found it https://blog.acolyer.org/2019/06/24/fast-key-value-stores/

rrdharan• 6 years ago

Maybe you’re thinking of the RAMCloud paper?

https://web.stanford.edu/~ouster/cgi-bin/papers/ramcloud.pdf

ddorian432• 6 years ago

Nope. It's "Fast key-value stores: An idea whose time has come and gone":

https://blog.acolyer.org/2019/06/24/fast-key-value-stores/

pojntfx• 6 years ago

Played around w/ olric a while back as a kv store for a routing graph, really enjoyed it even though I ended up home-brewing it later (the network topology meant that olric wasn’t the right tool for the job). Highly recommend it ;)

didip• 6 years ago

I have been curious about Olric for a while now.

* It is embeddable.

* Seems easy to install, even without Kubernetes.

I am just missing performance number to answer questions like: Is it significantly faster than etcd? Is it faster than tikv? And finally, is it faster than Redis Cluster?

lnsp• 6 years ago

I don't think it should be compared to etcd since it only offers best-effort consistency while etcd is strictly consistent.

shahsyed• 6 years ago

Great questions!

What's the underlying algo/impl used for this? According to the release notes it uses a DTopic data structure. While interesting, I'm not really sure if this is a unique/new structure by the author (buraksezer) or if it is an impl. from a paper I'm not aware of.

If anyone has more information about it, let me know!

phpjsnerd• 6 years ago

Thanks man!

mancini0• 6 years ago

So would this be similar to Apache Ignite?

hellofunk• 6 years ago

I’m curious how you actually would access this, would you need some kind of an RPC library to use in conjunction with it? The page doesn’t really seem to go far enough to answer actual usage information.

What would be the alternative to using a library like this? I’ve been looking for a good way to distribute workload to many machines and don’t want to invent it myself or jump into a very heavy system.

harikb• 6 years ago

This example gives a flavor of the client-server mode. Seems to be using some custom binary protocol - that seems like a surprising choice.

https://github.com/buraksezer/olric#golang-client

  var clientConfig = &client.Config{
    Addrs:       []string{"localhost:3320"},
    DialTimeout: 10 * time.Second,
    KeepAlive:   10 * time.Second,
    MaxConn:     100,
  }

  client, err := client.New(clientConfig)
  if err != nil {
     return err
  }

  dm := client.NewDMap("foobar")
  err := dm.Put("key", "value")
  // Handle this error

meddlepal• 6 years ago

Reminds me of a stripped down version of Hazelcast.

jwineinger• 6 years ago

Maybe just a personal thing, but going to a github repo and seeing "build failing" and low test coverage icons are usually turn offs for me to continue reading.

mholt2• 6 years ago

Why? It's normal for builds to fail between releases. And (warning: controversial opinion inbound) code coverage is hardly a useful metric -- one can write a single test case that gets 100% coverage, but doesn't test anything; what you want is assertion coverage but I don't know how possible that is.

sneak• 6 years ago

Is it normal for builds to fail on master between releases?

I would think a green build CI should be required for merge to master, even if perhaps the tests don’t all pass.

jnwatson2• 6 years ago

It strongly depends on the branch style. An older style makes the main/master branch to be the dev branch, allowing temporary deviations from working. If you want a working branch, you have to pull a release branch.

The modern trend is that master is sacred and should always pass.

reggieband• 6 years ago

It's normal for builds to fail between releases.

Why justify this with some categorical normative? In the last 7 or 8 years of my development life the master branch on every project I've worked on has always been clean. Breaking changes live in a branch and those branches are becoming shorter and shorter lived (previously lived for months, then weeks, now days).

That doesn't necessarily mean master is at all time ready to release (although some hold that extreme position). But it does mean that whenever you want to start work on some new feature you can be sure that branching from master is safe. And rebasing against master at any point should be safe.

In fact, almost all repos I've worked with recently explicitly deny merges into master unless a suite of tests pass, including basic builds, unit tests, and static code linting. The very idea that I could ever pull master and get a build failure makes me shudder.

eat_veggies• 6 years ago

Is 74% low? The 70-80% range feels about average, and decently acceptable for most projects.

eikenberry• 6 years ago

74% isn't low, that's right in the sweet spot. I generally shoot for 65-85%. Lower and you miss important stuff, higher and you start testing implementation instead of APIs and the tests become to brittle.

bellwether• 6 years ago

Love this idea, can’t wait to try it out!

Thaxll• 6 years ago

Looks like Hazelcast kind of solution.

dang• 6 years ago

Also discussed 6 months ago: https://news.ycombinator.com/item?id=22297507

Olric: Distributed in-memory data structures in Go

Related Stories