GitHub CI/CD observability with OpenTelemetry step by step guide

https://signoz.io/blog/cicd-observability-with-opentelemetry/

by ankit01-oss • 1 year ago

152 59 1 year ago

reactordev2• 1 year ago

As someone who has some experience in observability at scale, the issue with SigNoz, Prom, etc is that they can only operate on the data that is exposed by the underlying infrastructure where the IaaS has all the information to provide a better experience. Hence CloudWatch.

That said, if you own your infrastructure, I’d build out a signoz cluster in a heartbeat. Otel is awesome but once you set down a path for your org, it’s going to be extremely painful to switch. Choose otel if you’re a hybrid cloud or you have on premises stuff. If you’re on AWS, CloudWatch is a better option simply because they have the data. Dead simple tracing.

6r172• 1 year ago

I did have some bad experiences with OTEL and have lot of freedom on deployment ; I never read of Signoz will definitely check it out ; SigNoz is working with OTEL I suppose ?

I wonder if there are any other adapters for trace injest instead of OTEL ?

darkstar_16• 1 year ago

Jaeger collector perhaps but then you'd have to use the Jaeger UI. Signoz has a much nicer UI that feels more integrated but last I checked had annoying bugs in the UI like not keeping the time selection when I navigated between screens.

elza_11113• 1 year ago

yep, SigNoz is OpenTelemetry native. You can instrument your application with OpenTelemetry and send telemetry data direclty to signoz.

bbkane• 1 year ago

There are a few: I've played with https://uptrace.dev and https://openobserve.ai/ . OpenObserve is a single binary, so easy to set up

FunnyLookinHat• 1 year ago

I think you're looking at OTel from a strictly infrastructure perspective - which Cloudwatch does effectively solve without any added effort. But OTel really begins to shine when you instrument your backends. Some languages (Node.js) have a whole slew of auto-instrumentation, giving you rich traces with spans detailing each step of the http request, every SQL query, and even usage of AWS services. Making those traces even more valuable is that they're linked across services.

We've frequently seen a slowdown or error at the top of our stack, and the teams are able to immediately pinpoint the problem as a downstream service. Not only that, they can see the specific issue in the downstream service almost immediately!

Once you get to that level of detail, having your infrastructure metrics pulled into your Otel provider does start to make some sense. If you observe a slowdown in a service, being able to see that the DB CPU is pegged at the same time is meaningful, etc.

[Edit - Typo!]

makeavish4• 1 year ago

Agree with you on this. OTel agents allows exporting all host/k8s metrics correlated with your logs and traces. Though exporting AWS service specific metrics with OTel is not easy. To solve this SigNoz has 1-Click AWS Integrations: https://signoz.io/blog/native-aws-integrations-with-autodisc...

Also SigNoz has native correlation between different signals out of the box.

PS: I am SigNoz Maintainer

elza_11113• 1 year ago

FYI for anyone reading, OTel does have great auto-instrumentation for Python, Java and .NET also

reactordev2• 1 year ago

Not confusing anything. Yes you can meter your own applications, generate your own metrics, but most organizations start their observability journey with the hardware and latency metrics.

Otel provides a means to sugar any metric with labels and attributes which is great (until you have high cardinality) but there are still things that are at the infrastructure level that only CloudWatch knows of (on AWS). If you’re running K8s on your own hardware - Otel would be my first choice.

elza_11113• 1 year ago

There are integrations that let you monitor your AWS resources also on SigNoz. That said, I personally think CloudWatch is painful in so many other ways as well,

Check this out, https://signoz.io/blog/6-silent-traps-inside-cloudWatch-that...

mdaniel4• 1 year ago

A child comment mentioned k8s but I also have been chomping at the bit to try out the eBPF hooks in https://github.com/pixie-io/pixie (or even https://github.com/coroot/coroot or https://github.com/parca-dev/parca ) all of which are Apache 2 licensed

The demo for https://github.com/draios/sysdig was also just amazing, but I don't have any idea what the storage requirements would be for leaving it running

bravesoul2• 1 year ago

That's a genius idea. So obvious in retrospect.

hrpnk• 1 year ago

Has anyone seen OTel being used well for long-running batch/async processes? Wonder how the suggestions stack up to monolith builds for Apps that take about an hour.

zdc1• 1 year ago

I've tried and failed at tracing transactions that span multiple queues (with different backends). At the end I just published some custom metrics for the transaction's success count / failure count / duration and moved on my with life.

makeavish4• 1 year ago

You can use SpanLinks to analyse your async processes. This guide might be helpful introduction: https://dev.to/clericcoder/mastering-trace-analysis-with-spa...

Also SigNoz supports rendering practically unlimited number of spans in trace detail UI and allows filtering them as well which has been really useful in analyzing batch processes: https://signoz.io/blog/traces-without-limits/

You can further run aggregation on spans to monitor failures and latency.

PS: I am SigNoz maintainer

ai-christianson• 1 year ago

Is this better than Honeycomb?

brunoqc• 1 year ago

Is SpanLinks supported by jaeger?

madduci• 1 year ago

I use Otel running in a GKE cluster and tracking Jenkins jobs, whose spans/traces can track long time running jobs pretty well

dboreham2• 1 year ago

It doesn't matter how long things take. The best way to understand this is to realize that OTel tracing (and all other similar things) are really "fancy logging systems". Some agent code emits a log message every time something happens (e.g. batch job begins, batch job ends). Something aggregates those log messages into some place they can be coherently scanned. Then something scans those messages generating some visualization you view. Everything could be done with text messages in text files and some awk script. A tracing system is just that with batteries included and a pretty UI. Understood this way it should now be clear why the duration of a monitored task is not relevant -- once the "begin task" message has been generated all that has to happen is the sampling agent remembers the span ID. Then when the "end task" message is emitted it has the same span ID. That way the two can be correlated and rendered as a task with some duration. There's always a way to propagate the span ID from place to place (e.g. in a http header so correlation can be done between processes/machines). This explains sibling comments about not being able to track tasks between workflows: the span ID wasn't propagated.

imiric• 1 year ago

That's a good way of looking at it, but it assumes that both start and end events will be emitted and will successfully reach the backend. What happens if one of them doesn't?

hinkley4• 1 year ago

Every time people talk about OTel I discover half the people are talking about spans rather that stats. For stats it’s not a ‘fancy logger’ because it’s condensing the data at various steps.

And if you’ve ever tried to trace a call tree using correlationIDs and Splunk queries and still say OTEL is ‘just a fancy’ then you’re in dangerous territory, even if it’s just by way of explanation. Don’t feed the masochists. When masochists derail attempts at pain reduction they become sadists.

sethammons2• 1 year ago

We had a hell of a time attempting to roll out OTel for that kind of work. Our scale was also billions of requests per day.

We ended up taking tracing out of these jobs, and only using on requests that finish in short order, like UI web requests. For our longer jobs and fanout work, we started passing a metadata object around that appended timing data related that specific job and then at egress, would capture the timing metadata and flag abnormalities.

atypeoferror• 1 year ago

We have, and I won’t lie, it’s kinda a pain. The reality we discovered is that async flows are second-class citizens in otel tracing.

We essentially propagate the context manually between APIs and lambdas, through HTTP headers, SNS/SQS and even storage. We also ended up pulling Tempo’s Parquet files (we’re self-hosting the grafana stack) into Redash to be able to do real analysis. We got a lot of great insights and were able to do a lot of tuning that would have been impossible otherwise, but it was quite an investment. Would love to know if there is anything out there that would have made this less painful.

drzaiusx112• 1 year ago

yes, the only tricky part for us were the short async jobs where the otel queue didn’t have time to process its events before the tasks exited. we just added a drain wait loop in our at_exit callbacks

sali02• 1 year ago

noob question, i'm currently adding telemetry to my backend.

I was at first implementing otel throughout my api, but ran into some minor headaches and a lot of boilerplate. I shopped a bit around and saw that Sentry has a lot of nice integrations everywhere, and seems to have all the same features (metrics, traces, error reporting). I'm considering just using Sentry for both backend and frontend and other pieces as well.

Curious if anyone has thoughts on this. Assuming Sentry can fulfill our requirements, the only thing taht really concerns me is vendor-lockin. But I'm wondering other people's thoughts

whatevermom• 1 year ago

Sentry isn’t really a full on observability platform. It’s for error reporting only (that is annotated with traces and logs). It turns out that for most projects, this is sufficient. Can’t comment on the vendor lock-in part.

srikanthccv• 1 year ago

I was at first implementing otel throughout my api, but ran into some minor headaches and a lot of boilerplate

OTeL also has numerous integrations https://opentelemetry.io/ecosystem/registry/. In contrast, Sentry lacks traditional metrics and other capabilities that OTeL offers. IIRC, Sentry experimented with "DDM" (Delightful Developer Metrics), but this feature was deprecated and removed while still in alpha/beta.

Sentry excels at error tracking and provides excellent browser integration. This might be sufficient for your needs, but if you're looking for the comprehensive observability features that OpenTelemetry provides, you'd likely need a full observability platform.

dboreham2• 1 year ago

You can run your own sentry server (or at least last time I worked with it you could). But as others have noted sentry is not going to provide the same functionality as OTel.

mdaniel4• 1 year ago

The word "can" is doing a lot of work in your comment, based on the now horrific number of moving parts[1] and I think David has even said the self-hosting story isn't a priority for them. Also, don't overlook the license, if your shop is sensitive to non-FOSS licensing terms

1: https://github.com/getsentry/self-hosted/blob/25.5.1/docker-...

vrosas• 1 year ago

Think of otel as just a standard data format for your logs/traces/metrics that your backend(s) emit, and some open source libraries for dealing with that data. You can pipe it straight to an observability vendor that accepts these formats (pretty much everyone does - datadog, stackdriver, etc) or you can simply write the data to a database and wire up your own dashboards on top of it (i.e. graphana).

Otel can take a little while to understand because, like many standards, it's designed by committee and the code/documentation will reflect that. LLMs can help but the last time I was asking them about otel they constantly gave me code that was out of date with the latest otel libraries.

stackskipton2• 1 year ago

Ops type here, Otel is great but if your metrics are not there, please fix that. In particular, consider just import prometheus_client and going from there.

Prometheus is bog easy to run, Grafana understands it and anything involving alerting/monitoring from logs is bad idea for future you, I PROMISE YOU, PLEASE DON'T!

avtar• 1 year ago

anything involving alerting/monitoring from logs is bad idea for future you

Why is issuing alerts for log events a bad idea?

sali02• 1 year ago

Thank you, this is where I'll likely start.

From other comments as well, seems it's still worth trying to integrate otel. Appreciate everyone's insights

vanschelven• 1 year ago

I'd say "track errors first" [0] and focus on APM later (if at all). If you're worried about Sentry's lock-in, know that there are API-compatible drop-in replacements[1][2] though they are less feature-complete on the APM/observability side.

[0] https://www.bugsink.com/blog/track-errors-first/

[1] https://www.bugsink.com/

[2] https://glitchtip.com/

totetsu• 1 year ago

I spent some time working on this. First I tried to make a GitHub action that was triggered on completion of your other actions and passed along the context of the triggering action in the environment, then used the GitHub api to call out extra details of the steps and tasks etc, and the logs and make that all into a process trace and send it via an otel connection to like jaeger or grafana, to get flamchart views of performance of steps. I thought maybe it would be better to do this directly from the runner hosts by watching log files, but the api has more detailed information.

candiddevmike2• 1 year ago

How does SigNoz compare to the other "all-in-one" OTel platforms? What part of the open-core bit is behind a paywall?

makeavish4• 1 year ago

Only SAML, Multiple ingestion keys and Premium Support is under paywall. SSO is not under paywall. Check pricing page for detailed comparison: https://signoz.io/pricing/

127dot1• 1 year ago

That's a poor title: the article is not about CI/CD, it is particularly about GitHub CI/CD and thus is useless for the most CI/CD cases.

dang• 1 year ago

Ok, we've added Github to the title above.

ankit01-oss• 1 year ago

thanks dang

remram2• 1 year ago

I have thought about that before, but I was blocked by the really poor file support for OTel. I couldn't find an easy way to dump a file from the collector running in my CI job and load it on my laptop for analysis, which is the way I would like to go.

Maybe this has changed?

sweetgiorni• 1 year ago

https://github.com/open-telemetry/opentelemetry-collector-co...

remram2• 1 year ago

And the receiver: https://github.com/open-telemetry/opentelemetry-collector-co...

I'll have to try this!

edit: actually Jaeger can just read those files directly, so no need to run a collector with the receiver. This is great!

drzaiusx112• 1 year ago

we run a prom+otel+xray stack at scale with grafana as the interface. i honestly don’t miss datadog at all at this point. trivial to add new alarms, dashboards etc

GitHub CI/CD observability with OpenTelemetry step by step guide

Related Stories