ComNik

Additional time axes allow you to separate transactions from different sources of input. E.g. in a collaborative system it could be helpful to ask questions while rewinding the actions of a particular user.

But a different question is whether the system can do something useful for you with first-class time co-ordinates, compared to just stuffing additional timestamps into your data. (something useful being clever indexing, compaction, maybe more?)

If model-theoretic semantics and the various ways to slice, dice, and extend Datalog are interesting to you, then almost any talk by Peter Alvaro might be as well.

In particular: https://www.youtube.com/watch?v=R2Aa4PivG0g

Check out the white paper at https://github.com/ipfs/papers/raw/master/ipfs-cap2pfs/ipfs-..., section 3.7. There is a name system called IPNS.

I think Finda (https://keminglabs.com/finda/) fills a similar need, runs offline, and indexes lots of other stuff as well. Although I think it is not possible to assign additional keywords.

(Not affiliated in any way, just a user)

Interesting point.

As I don't have much experience in this area: Can you elaborate on some use cases for layouts other than AoS and SoA?

Jonathan Blow's language "jai" allows for seamlessly switching between AoS and SoA and generally seems to value efficient data layout and "gradual" optimization over safety (in contrast to e.g. Rust).

Unfortunately, no public compiler seems to be available at this time.

For anyone interested in exploring the datomic model, there is a great ClojureScript in-memory implementation called datascript (https://github.com/tonsky/datascript) by Nikita Prokopov.

We use Datalog every day in a React-Native application, via the datascript db (https://github.com/tonsky/datascript). Has been a very positive experience so far.

Thinking in "facts" / binary relations is itself a refreshing approach.

Thank you for Murmur3! I have been using it in a consistent hashing implementation as part of my bachelor's thesis and it is performing very well.

Interesting note, thank you. Are you referring to "Sigma" https://code.facebook.com/posts/745068642270222/fighting-spa... ?

That was not the intension behind my comment at all. I rely on type systems a lot, myself.

I was trying to note that Clojure, as a dynamic language, is making a (to my eyes) very interesting choice, of doubling down on these dynamic methods of verification. For some uses, I can see this as the better choice, for others it isn't and won't be.

As a side note in his talk "Simple Made Easy" (https://www.infoq.com/presentations/Simple-Made-Easy, around minute 42) Rich Hickey mentions, that conditional statements are complex, because they spread (business-)logic throughout the program.

As a simpler (in the Hickey-sense) alternative, he lists rule systems and logic programming. For example, keeping parts of the business logic ("What do we consider an 'active' user?", "When do we notify a user?", etc...) as datalog expressions, maybe even storing them in a database, specifies them all in a single place. This helps to ensure consistency throughout the program. One could even give access to these specifications to a client, who can then customise the application directly in logic, instead of chasing throughout the whole code base.

Basically everyone involved agrees on a common language of predicates explicitly, instead of informally in database queries, UI, application code, etc...

But Hickey also notes that this thinking is pretty "cutting-edge" and probably not yet terribly practical.

Full, formal verification of complex systems (especially in the distributed case) requires a lot of modelling effort, beyond what is needed for "mere" implementation. A lot of important constraints cannot even be expressed in most type systems. And if one uses a specialised modelling language for the proof, the actual implementation might introduce bugs. Also relevant to real, shipping software: specifying a program in such that detail as required for automatic verification makes it even harder to change (which, given infinite time and money, is indeed a very very good property!).

This just goes to say, that a type system can be very helpful, but is ultimately just a part of regular testing.

So for most companies and most developers, anything that aides in keeping documentation up-to-date, writing or generating tests and helping developers understand what they are reading is probably a better ROI.

You make a very good point, the comparison was flawed there.

The thing I do see as being important is not how you write these things down, but whether they have an accessible representation at run/read/compile/whenever time.

This is a better way to put it. The second important factor for me is expressiveness. With spec or any other contracts-like system one gets the full power of the language to express constraints. Of course type systems are not artificially restricted in this regard, they simply make a different trade-off.

I hope my comment did not come off as a riff on static vs dynamic typing, and I don't think any contract system is meant to replace type systems. Until expressing all important program specifications formally becomes viable for everyone (maybe through this work https://www.math.ias.edu/vladimir/current_work?), a less-formal, dynamic approach seems very attractive.

Clojure, with its dabbling in schemas and rule systems / logic programming (as a substitute for conditional statements) and thanks to its great tools (like figwheel or devcards), could really be establishing a cheaper, "as-good" alternative to static typing and full formal verification.

Instead of encoding constraints as type signatures, the Clojure folks (true to character) encode them in data. In my eyes a very interesting, pragmatic trade-off between expressiveness and automatic verifiability.

There is a recording available as well: https://www.youtube.com/watch?v=98LdFA-_zfA

No, on a Core-2 :)

Can't comment on DarkBasic or Monkey, but:

BlitzBasic and BlitzMax are about the same language. What BlitzMax provided over BlitzBasic was a direct integration with DirectX and OpenGL, which allowed you to make 2D games with hardware supported rendering.

Your average BlitzBasic "Hello World" would usually take up 50% of the CPU.

I spent most of my early programming years with BlitzMax. The language itself probably has no future on it's own, but combined with the DX/OpenGL integration and overall straightforwardness, I'd heavily recommend it for interested children.

If you have one or know of one, consider this.

Just when I was halfway through explaining byzantine fault tolerance to my dad.

It means that operations don't get executed in the same order on every replica. If the systems state depends on the ordering of operations, then each replica might be in a different state for the same set of operations.

"Consistent replication" would be using a protocol like Paxos to have the replicas decide on a single order of operations.

This was the year I realized that one language does not fit all, and that it is okay to have more than one favorite language.

Same here. This, funnily enough, had the nice effect of taking a lot of stress out of talking about programming languages.

Maybe relevant: https://news.ycombinator.com/item?id=8203144

Every other week: http://blog.acolyer.org/ ("The Morning Paper")

Also, the "Weekend Reading" Mailing List (https://tinyletter.com/assaf) Interesting bits, lots of funny stuff, ideal for, well, weekend reading.

A bit more information can be found here: http://cdixon.org/2015/03/24/improbable-enabling-the-develop...

Too me it reads like MapReduce for highly place-dependent computations, whatever that looks like. Probably something along the lines of a distributed kd-tree with message passing at borders handled for you as well.

For all application-data events I consider postgres to be the ground-truth. That is somewhat unfortunate, because one can't easily place a queue in front of the database. For metrics and logs, the Kafka topic itself (which is persisted similiar to your flat files) would become the master. The use-case is pretty similiar.

Might it be feasible to have something like postgres work with an external WAL? That would solve the problem I guess, as well as leave us with a single "persistent" system.

Yes, I fully recognize the problem with double-writing. I will definitely try out Bottled Water. I was also thinking about replacing Kafka with a much simpler, lower-throughput system (because we are lightyears from LinkedIn's requirements).

Two reasons why I can't just use postgres (I'd love to): 1.) Kafka (or whatever queue we settle on) will be used for logs and metrics as well, data that doesnt flow through postgres.

2.) Postgres stores the data-model of my business-domain, at the lowest, normalized level. But derived data-stores are inherently denormalized and I want to be able to use them without talking back to my source-of-truth all the time. So currently I'm passing DTOs to Kafka, just like I would to any API request. This data is not easily available at the postgres-level.

I'm not yet sure on the right abstraction level for events. It seems very natural to have them contain information that I would send to clients directly.

Thank you for your detailed thoughts. You obviously have much more practical experience with this kind of system.

Many of the problems you mentioned I am aware of, and also have no workable solution yet (detecting lost messages being the biggest - Merkle-tees sounds like a very interesting approach, maybe even applied at the log-level?).

As mentioned in another reply, Kafka does support the kind of "pointer-to-log" setup you mention. Also Kafka is designed for lots of consumers, each with different characteristics. In principle, I should be able to sync something like memcache with the same information I need to sync Elasticsearch. The same holds for a websocket-server that reads from this stream and forwards new events to web-app clients. So I don't see the need for more than one "queue" yet, maybe that will show up in practice.

Also your setup would require a lot more coordination to handle updates from multiple postgres instances, if I understood correctly.

That being said, I'm still in the experimental phase with all of this, I will publish a writeup once I gain a bit more experience.

Just today I successfully finished an experiment: An existing web-app with Postgres as the source of truth pushing update events to a Kafka queue. From there, a screenful of Go forwards those events into Elasticsearch.

It doesn't solve every problem and it might be a lot of new parts if you're not going to use Kafka for anything else. I will be using it for other things like caching and push events. This kind of syncing problem seems to crop up in a lot of places.

A nice thing is that I don't have to care about Elasticsearch durability much, because I can simply rerun the ingester from the beginning of the log, as long as Kafka doesn't lose data.

I like this one: http://mochajs.org/ Gets the job done and stands out amongst todays usual landing pages.