GCP charged $1.3k for stdout logs (9x my cluster cost). Refund denied twice

by nthypes • 8 months ago

2 2 8 months ago

nthypes• 8 months ago

Last month, a single container in my GKE cluster (Sao Paulo region) entered an error loop, outputting to stdout at ~2k logs/second.

I discovered the hard way that GKE's default behavior is to ingest 100% of this into Cloud Logging with no rate limiting. My bill jumped nearly 1000% before alerts caught it.

Infrastructure (Compute): ~$140 (R$821 BRL) Cloud Logging: ~$1,300 (R$7,554 BRL)

Ratio: Logging cost 9.2x the actual servers.

https://imgur.com/jGrxnkh

I fixed the loop and paused the `_Default` sink immediately.

I opened a billing ticket requesting a "one-time courtesy adjustment" for a runaway resource—standard practice for first-time anomalies on AWS/Azure.

I have been rejected twice.

The latest response: "The team has declined the adjustment request due to our internal policies."

If you run GKE, the `_Default` sink in Log Router captures all container stdout/stderr.

There is NO DEFAULT CAP on ingestion volume which is an absurd!

A simple while(true); do echo "error"; done can bankrupt a small project.

Go to Logging -> Log Router. Edit _Default sink.

Add an exclusion filter: resource.type="k8s_container" severity=INFO (or exclude specific namespaces).

Has anyone successfully escalated a billing dispute past Tier 1 support recently?

It seems their policy is now to enforce full payment even on obvious runaway/accidental usage.

salawat• 8 months ago

You could stop renting someone else's computer and run your own clusters til you have all the surprises figured out you know.

This is the danger of metered compute. Your ignorance is their profit opportunity.

GCP charged $1.3k for stdout logs (9x my cluster cost). Refund denied twice

Related Stories