ElastiQuill: A modern blog engine built on top of Elasticsearch

https://github.com/BigDataBoutique/ElastiQuill

by indigodaddy • 7 years ago

55 34 7 years ago

vore• 7 years ago

I've heard that Elasticsearch has had some durability issues in the past and the recommendation has been to have a more reliable source of truth (e.g. an SQL database) and periodically import data into Elasticsearch for indexing. Is this still a problem in practice, or is storing data directly in Elasticsearch like this still unsafe?

softwaredoug• 7 years ago

I definitely think it shouldn’t be your source of truth.

However, I think as a public facing content engine it’s fine. I know of one major e-commerce brand that does this for products. This works great for their use case, given how easily Elasticsearch lets you filter, sort, search pretty seemlessly. But they can always rebuild Elasticsearch from another system.

colechristensen• 7 years ago

It is, at least, a very much diminished issue in my experience going from 2.x to 7.x

Somewhere around 5 or 6 the tides turned. What was a big issue is how easy it was to overload the cluster and how data would be lost in that state.

To be safe you should have regular backups anyway. I wouldn't trust ES to handle very critical data which you wouldn't ever want to lose a bit of, but I have much less worry these days about the operational overhead of fixing an cluster because failure in experience is just much less likely.

It is still though, something which is less "set and forget" like many sql engines can be in less critical or intense workloads. You would want someone in your org to really know ES.

jillesvangurp• 7 years ago

They re-implemented the clustering algorithm for v7 and fixed numerous bugs in between v1/v2 and v7. Including essentially all of the ones reported in the infamous call me maybe article (https://aphyr.com/posts/323-jepsen-elasticsearch-1-5-0).

However, it's not a database. I've actually abused it as such and it's fine. You get optimistic locking but no real transactions. Search is eventually consistent unless you call _refresh (but you shouldn't), etc. Bearing in mind it is not a database, it is not intended to be used as a database, and probably will never be a database, it actually works fine as a database provided you don't do a lot of updates (write model is append only).

If safety is a big concern for you, obviously use something else. But for a blog it's completely fine.

sagichmal2• 7 years ago

Elasticsearch is not an appropriate choice for an authoritative store of data.

kbumsik• 7 years ago

Could you elaborate? I'm just started learning Elasticsearch so that sound quite interesting to me.

jaytaylor• 7 years ago

Elasticsearch should not be used as the sole source of truth. I've been using it for years professionally, and it's an amazing tool but not safe with regard to data integrity / durability.

While dramatic progress and improvements have been made over the past 9 years, sometimes things still go bad and indices get corrupted. When this happens, it's necessary to reindex the data. There are also additional situations where reindexing is required. So the safe advice is: Always have the authoritative data source elsewhere (in a reliable data store or database of some kind) and then load the data into Elasticsearch from there.

Postgres or even MySQL will be a safer bet when data integrity is key. Then it's only a matter of indexing the data from the DB into ES.

Using Elasticsearch for a blog is a fun you idea, but ultimately is likely to be overkill for a personal blog site.

hombre_fatal• 7 years ago

Surely the data store is the least interesting part of a blog. Or, rather, what competitive advantage does Elastic give to a blog that warrants having "Elastic" in the product name?

It reminds me of https://nodebb.org, the "forum built with Node.js" (though they've rebranded away from that) which always struck me as the opposite place to focus when selling software.

For example, none of my problems with Wordpress are related to its choice of database. Just like porting Wordpress to Elastic Search don't fix any of my issues with Wordpress. A blog is almost entirely defined by its UX.

On a better note, the platform seems pretty polished.

oefrha• 7 years ago

A blog is almost entirely defined by its UX.

A blog is almost entirely defined by its content. I read a lot of run-of-the-mill average to crappy Wordpress or Blogger blogs that neither load fast nor look flashy (or sometimes even pleasant). I don’t give a shit because I read them for the content, mostly from an RSS reader.

NewsAware• 7 years ago

Understood the GP to mean blogging-engine/CMS and UX to mean admin&editing ergonomics

badrabbit2• 7 years ago

Fuzzy search and kibana

whalesalad• 7 years ago

This will be less reliable than a one-click WordPress install running on a $5.99/year GoDaddy shared host.

stephbu• 7 years ago

hits tiny staple with enormous sledgehammer

While it is possible to use ES as your primary store, doesn’t mean it is a great use-case. Pushing ES to durability takes resources (people, time, compute) and rigor.

For a blog site this feels like overkill. Your priorities are durable low-freq writes, and more freq-reads. Hands-free day-to-day maintenance, and simple recovery mode such as restore. Caching (and optionally queuing) can absolve sins of many topologies to fulfill these requirements.

ES brings a much larger surface area than just store blobs and metadata. Big tax to gain free-text search - which while interesting most users will almost certainly never use (thank Google for that)

Focus on a cheaper, lighter, database engine that is durable, checkpoints regularly, and has simple, testable backup/restore processes. Ship the backups somewhere e.g. S3

sombremesa• 7 years ago

You don't even really need* a backup/restore these days depending on which provider you use.

* (They'll do it for you)

jimktrains2• 7 years ago

Postgres has a basic full text search built in.

https://www.postgresql.org/docs/12/textsearch.html

badrabbit2• 7 years ago

I disagree, both from a programming and admin perspective, I found ES to be the easiest option. With ES, I don't have to explicitly create tables (indexes) ,I find ES queries and results a lot easier to deal with. A single node ES cluster is easy to maintain, I mean, just give it enough ram and disk space and you shouldn't even have to think about it again. And if you want to visualize the data as an admin, you can slap kibana on it. You can even go a bit crazy here and feed ES your web server logs to have even better correlation for stats. ES also has a very simple testable backup proceee (as simple as a curl command!).

To your point, it does feel like "using a cannon to kill a mosquito". But I still had a better experience with it than even sqlite.

vikingcaffiene3• 7 years ago

Sorry but this is a terrible idea. ES is not meant as a durable store. Its designed as an ephemeral abstraction on top of a more durable storage mechanism like Postgres or the like. This is just asking for trouble to use it like this.

aprdm3• 7 years ago

How is something meant as a durable store or not? I've worked in companies that had only ES as a database for years and they didn't had complaints. I've seen it working mostly well in these orgs and because of my now experience with it would likely strongly consider it as a database just like mongo and friends with the benefits of having a very large ecosystem (elk, beats etc.) Which can make it a one stop shop for all things in an org

vikingcaffiene3• 7 years ago

ES is an eventually consistent data store meaning that it makes no guarantees about data availability when you put stuff in there. That alone should be a deal breaker for you.

If you use ES for what it's designed to be and make it a projection of your underlying durable data store, you can do things like rebuild your index or change your schema without fear of data loss.

Its a document store, not a database. MongoDB is too but they try to hide it by bolting on all the stuff traditional relational DB's have like ACID and transactions so you wont realize what a poor choice it is as a database when compared to say, Postgres.

alexnewman3• 7 years ago

By durable, you mean If I turn the power on and off, the data will be there. I'm pretty sure it has this quality. The question of whether or not it has ACID semantics seems a nonstarter when we are discussing a simple blog

big_chungus• 7 years ago

Why would you do this? PHP is awesome compared to this. Or go with a static site. This reminds me of the VS opening JSON meme: http://devhumor.com/content/uploads/images/September2019/vs_...

idclip• 7 years ago

bless you, why not. ever considered applying to logz.io ? they could use a guy like you.

inertiatic• 7 years ago

I don't see how ES could be an issue for a blog, it's not like Lucene indexes blow up randomly and keeping backups should be standard practice at scale even if you used a proper DB.

Meanwhile, if you want a small deployment that also does full text search with a bunch of "smarter" features and some analytics, it would make sense to only use a single data store.

All these in theory, as I haven't checked out the actual project.

NicoJuicy• 7 years ago

A blog engine build on top of a 1,5gb instance is a bit much.

PudgePacket• 7 years ago

I like the irony of being built on a search engine but not having search functionality :)

pensatoio• 7 years ago

This project is the epitome of, “how not to use Elasticsearch.” Comments in this thread can be divided perfectly into two groups: people that have experience with Elasticsearch and people that don’t. Only the latter will suppose this project could ever be a good solution.

ElastiQuill: A modern blog engine built on top of Elasticsearch

Related Stories