vadman97

If there was only one alert criteria, that'd be simple. Our alerts can be configured for any data filters (eg. only matching logs with column `level='error'`); we would have to create a unique MV for each alerts' filter condition.

We insert from our alerts worker because we want the aggregation to happen per alert (with the aggregated data filtered by the particular alert definition). As each alert is evaluated, we run the following [1] INSERT INTO ... SELECT ... statement based on the alert definition. We can't aggregate with an MV since we'd need to create an MV per unique alert that a customer may set up.

[1] https://github.com/highlight/highlight/blob/c526daea31fdf764...

Forget the exact details, but we ran into issues with tuning the two setting you're describing on the Cloud instance (these settings were user level settings that seemed to get reset by ClickHouse Cloud because they were updated based on the cluster size). Perhaps this could change with Cloud and isn't an issue with the OSS version.

Once we hit >100k inserts per second, async inserts didn't work well for us because we had limited control over the background async insert batching happening on the cluster. The background inserts would be too small, resulting in many merges running, causing high CPU, causing high back-pressure (latency) on async inserts which would just result in an ingestion delay.

Plus, async inserts are only available on ClickHouse Cloud.

We do via the OpenTelemetry Elixir SDK (and the Phoenix extension) https://opentelemetry.io/docs/languages/erlang/

Materialized views can help change the ORDER BY with 0 downtime:

* Create a new version of the table with the new ORDER BY.

* Create a materialized view that will insert from the old table to the new table.

* Update your application code to query the new table.

* Start inserting data into the new table.

Exploring bloom filter index merges would be an interesting addition. I do wish it were easier to profile merge performance to break down where most of the CPU time is being spent.

3 nodes of 120 GiB and 30 vCPU each.

I see what you did there.

Our browser client would help with tracing what network requests are being made by said apps since it would capture all network requests. However, Shopify may restrict what our browser client can do in your frontend store (I'm guessing that the 3rd party apps are added to a shop as iframes or are otherwise sandboxed).

Hey HN, I'm Vadim, the CTO at highlight.io. We offer session replay to engineering teams and have had a lot of questions around client-side performance.

We just published a benchmark along with a short blog post covering the results. If you'd like to try it yourself, check out https://github.com/highlight/session-replay-performance-benc...

Presumably, they mean millions of logs a day when saying "millions of events a day", of which some subset corresponds to errors.

Our errors ingest is quite affordable (https://www.highlight.io/pricing) and let's you filter out data you are not interested in ingesting. Our logs ingest is even cheaper because of the underlying ClickHouse storage we use.

Regarding model adaptation, we haven't yet explored a fine-tuned model, but it makes a lot of sense for a given class of errors. For a given code-base that is using highlight, the errors will typically be of a given language / infrastructure, so fine-tuning the model to those errors should be beneficial.

As for predicting potential errors, this will particularly make sense as an anomaly detection mechanism across metrics and logs. A class of 'important' errors based on the LLM's understand of the error, as well as historical comparison to normal error rates, is something we're exploring with an 'interesting errors' concept - stay tuned for more there!

Have you looked into finetuning the embedding model to your use case?

Not yet, though this is definitely one of the next steps for us. The `gte-large` model we use is trained on a variety of text, but a hypothesis is one trained or fine-tuned on technical / code-related content may work better.

Have you considered faster foundation models?

Any in particular that you would suggest? We're still pretty new to this so would love to learn about other recommendations. Would a foundation model perform as well at this task?

How far could you go with this idea? Could this be the basis of a new monitoring platform?

Certainly; there are traditional ML approaches that could be applied to monitoring as well and we're heavily exploring this (ie. for metric anomaly detection). Another area we're exploring embeddings grouping is for filtering ingest to help folks only ingest / retain data that they actually want, but without the overhead of strict filter rules. Tons more to explore in this space, and you will certainly be hearing more from us here.

Agreed, but the argument for causing damage still requires a human actor to execute GPT-4's advice. Should we be censoring LLMs just because we are offended by the output or find it misleading? There is plenty of misinformation on the internet already; in my opinion, the power to deal damage is still in the hands of humans.

ClickHouse Async insert docs [1].

We ran into some challenges with async inserts at highlight.io [2]. Namely, ClickHouse Cloud has an async flush size configured (that can't be changed AFAIK) that isn't large enough for our scale. Once you async insert more than can be flushed, you get back pressure on your application waiting to write while Clickhouse flushes the queue. We found that implementing our own batched flushing via kafka [3] is far more performant, allowing us to insert 500k+ RPS on the smallest cloud instance type.

[1] https://clickhouse.com/docs/en/optimize/asynchronous-inserts [2] https://github.com/highlight/highlight/tree/main [3] https://github.com/highlight/highlight/blob/4d28451b1935796d...

How do you think about the query syntax? Are you defining your own or are you following an existing specification? I particularly love the trace view you have, connecting a frontend HTTP request to server side function-level tracing.

The hardest errors to group / tag are ones that are further removed from the actual error cause (think frontend bug that causes a bad request that then causes a backend error, or an OOM because of a gradual memory leak). For these, we're looking at other ways to give more context to the LLM about an error (for example, associating frontend errors with backend ones and providing both as part of the request) but we also have limits on the LLM input size (512 tokens for the mdoel we're using, gte-large).

Both - tagging errors into broad buckets can help with filtering and searching for categories of errors, while finding similar errors surfaces ones that might be more familiar to an unknown one. We're learning as we go though and also want to surface solutions to errors long term.

As for hosting the model, Hugging Face has a number of guides (ie. [1]) for this but overall the process was quite simple. All you need to do is find a model compatible with the kind of inference you want to do (ie. sentence embeddings vs. text generation). Once it's deployed, Hugging Face gives you an API endpoint similar to OpenAI's embeddings API that returns the embedding vector for a given input. For picking the model, there are a bunch of good resources with benchmarks comparing them depending on the use case (ie. [2]).

[1] https://huggingface.co/blog/getting-started-with-embeddings [2] https://supabase.com/blog/fewer-dimensions-are-better-pgvect...

Have you seen the OTEL auto instrumentation package[1]? It supports a number of common frameworks (as middleware) and libraries (via monkeypatches) setting up spans for you to capture things like external http requests, database/cache queries, etc. [2]

[1] https://opentelemetry.io/docs/instrumentation/js/automatic/ [2] https://github.com/open-telemetry/opentelemetry-js-contrib/t...

Clarity is meant for high level analytics around product usage, while Highlight is built for engineering observability (catching and debugging errors). Highlight offers network tracing, log ingest (both client and server side), and error monitoring to serve that use-case which Clarity does not.

This would be more involved as you'd need to run and configure the infra dependencies yourself. For every container listed in the compose file [1], you'd want to set up that service. After that, you can use the shell scripts referenced in the dev deploy guide [2] to run the app's backend + frontend (having installed application dependencies mentioned, like node.js and go).

[1] https://github.com/highlight/highlight/blob/main/docker/comp... [2] https://www.highlight.io/docs/getting-started/self-host/dev-...

Looking forward to dependencies being cut down a bit as it should make it almost trivial to set it up for k8s (both hosting and ingesting logs).

We're planning to support k8s (openshift etc.) in the next 3-6 months. If you have another platform you'd like to see supported, please commented on the issue you linked.

Yes, a good number of folks have come from those tools. Not only to consolidate tools, but to get a cohesive debugging experience (see the frontend session and the server side errors + logs all in one place).

Appreciate it. Let us know what you think when you have a chance to try it!

Highlight can monitor server-side web apps; I'm assuming that's what you're running on those servers? I'd suggest instrumenting the common application code with our backend SDKs and deploying that change to all on-premise servers. Alternatively, you can run an OpenTelemetry collector (https://www.highlight.io/docs/getting-started/backend-loggin...) on each server to stream logs.

Mind asking in our Discord community (https://highlight.io/community)? Happy to help there - looks like you are missing the docker compose v2 plugin (https://docs.docker.com/compose/).

Mind trying https://discord.gg/SEJEfSqctQ ?

Do you mind joining our Discord (https://highlight.io/community) and asking there? Happy to help over a DM once you are in.