What I like using Grafana Loki for (and where I avoid it)

https://utcc.utoronto.ca/~cks/space/blog/sysadmin/GrafanaLokiWhatILikeItFor

by zdw • 3 years ago

30 14 3 years ago

utcc.utoronto.ca

A couple years ago we were hurting from having no easily accessible centralized logs. We also had some in-house timeseries database tech that we were looking to deploy more widely but needed first to stress test.

After stumbling on an overview of Loki, we whipped up a service that presented a Loki-compatible API on top of our timeseries database, and killed two birds with one stone. Because we have full control over the data ingestion and storage too we are able to keep the data in a structured form with full type information.

It's fantastic in that it ties in with the rest of our use of Grafana too.

lazypenguin• 3 years ago

Loki is great but PromQL is a doozy. I’m basically useless on grafana without the query builder. Tempo is also pretty cool if you’re emitting otel spans.

samspenc• 3 years ago

I'm not that familiar with Loki but I'm aware that PromQL is the Prometheus Query Language. Just curious if PromQL works with Loki? Or are you referring to Prometheus being the more powerful system due to PromQL queries?

trallnag• 3 years ago

The Loki query language is heavily based on PromQL

e12e3• 3 years ago

Would be interesting to hear if qryn (née cloki - clickhouse Loki) with vector might be a valid alternative to both Loki and traditional syslog.

AFAIK logshipping with journald is bespoke - but vector should be able to handle both syslog and journald - for relatively painless setup ...

Not sure if tailing is better with qryn though.

https://github.com/metrico/qryn

https://vector.dev/docs/reference/configuration/sources/sysl...

https://vector.dev/docs/reference/configuration/sources/jour...

https://www.digitalocean.com/community/tutorials/how-to-cent...

chrnola• 3 years ago

Oh interesting. I like that qryn exposes Grafana compatible query APIs. I’ve been playing with SigNoz (another Clickhouse-based, OTel compatible project for ingesting and analyzing logs, metrics, and traces) and I wish it would offer the same capability.

qxip2• 3 years ago

Thanks for the feedback! qryn developer here, and that's exactly our design goal: requiring no custom agents or clients in order to use our solution, through API emulation. There's a chance you could use qryn as a reader for data written by other clients, and that's exactly what we do with opentelemetry where we have a qryn-otel-collector allowing direct writes into ClickHouse at max speed, just like signoz, and reading through our Grafana compatible APIs using qryn.dev or qryn.cloud

qxip2• 3 years ago

qryn and vector get along very well! We use it all the time for testing and developing qryn and qryn.cloud and most of our users love it! But since qryn supports Logs, Metrics and Telemetry at once, we're just as compatible with any Agent supporting Loki/LogQL, Influx protocol for metrics and logs, Elastic Bulk, Prometheus for metrics, opentelemetry for everything... and more coming!

Feel free to open an issue on our repository if you end up trying it and/or would like us to help out!

https://qryn.dev https://qryn.cloud

e12e3• 3 years ago

Thank you. In this case,it was more a reference to:

Our central syslog server is a central source of accessible, text based truth. It requires almost no infrastructure to be working on any machine in order to get logs to it and accept logs (...)

From the linked post on central log server (and Loki):

https://utcc.utoronto.ca/~cks/space/blog/sysadmin/GrafanaLok...

Unfortunately, AFAIK journald and qryn don't speak rsyslog - so you'll need something "alien" - all though vector seems very straightforward.

Topgamer7• 3 years ago

We are hosting the grafana stack at my work. And if there are some comprehensive articles about trying to keep metrics counts low, and hosting on block storage with s3. I would appreciate peoples anecdata and links.

ironfizer• 3 years ago

Loki is the bomb-dizzle when it comes to real-time monitoring. First it’s designed to provide real-time monitoring of log data, allowing users to detect issues and troubleshoot problems as they occur. Gonna come in handy, particularly for mission-critical applications where downtime can have severe consequences.

elthor89• 3 years ago

i am looking into loki to use at work. Just wondering do most of you self-host or use the grafana cloud?

ncrmro• 3 years ago

Single node kube cluster running Loki and promtail and I’ve had great results until I started trying to push in some bare metal machine logs which cause 503 from nginx ingress. Then Loki stops working in Grafana.

But inside the cluster it’s great. Though my Loki helm chart is still on v3 rather than 4

What I like using Grafana Loki for (and where I avoid it)

Related Stories