Telemetry accumulates like technical debt. We built a tool to clean it up.
We set out to build an automated incident response solution. Then we talked to more than 50 companies, and heard the same objection repeating: it won’t work on our data. The telemetry is too noisy, too low quality, too misleading.
So we went digging, and found the problem underneath the problem. Logs, metrics, and traces accumulate for years, and almost nothing is ever removed.
Telemetry behaves like code. Engineers add it while shipping features or debugging incidents. Debug logs ship to production. Incident-specific logging outlives the incident by years. Auto-instrumentation and third-party libraries emit spans everyone’s learned to ignore. Metrics pick up high-cardinality tags that quietly multiply cost.
The result is technical debt—except instead of making code harder to maintain, it makes observability more expensive and production debugging harder.
The obvious question is: why doesn’t anyone clean it up?
Because doing it safely is surprisingly difficult, and developer teams usually have other priorities.
Before removing a log or metric you need to know:
- Is anyone relying on it?
- Does the same information exist in another form?
- Does an alert or dashboard depend on it?
- Can it be replaced by something cheaper?
Gathering all that context usually makes it not worth the trouble.
We built Obics to automate that process.
Why telemetry debt matters
Observability vendors typically charge based on ingest, indexed volume, or metric cardinality. As telemetry grows, so does the bill.
But storage isn’t the only cost. Every unnecessary log also consumes CPU, network bandwidth, and indirectly LLM tokens. During incidents, your AI agents have to sift through significantly more irrelevant data before finding what actually matters.
Metrics have their own version of the problem. A single unnecessary high-cardinality tag can multiply time series counts while adding little or no operational value.
The ownership gap
One thing we kept noticing is that telemetry cleanup falls between teams.
Developers own the code but rarely think about observability costs. They care about developing the product.
Platform and DevOps teams own the observability budget but can’t modify application code without approval from the developer teams.
In larger organizations the problem gets worse because deleting telemetry might accidentally break another team’s dashboards or alerts.
As a result, everyone agrees there’s waste, but nobody owns removing it.
How Obics works
Obics connects to your observability platform and your source control repository.
Every day it:
- analyses telemetry volume and cost
- groups similar logs together
- maps telemetry back to the source code
- analyses dashboards, alerts and other consumers
- identifies optimization opportunities
- estimates potential savings
- opens a pull request if you choose to apply the change
Some examples of issues it detects:
- duplicate logs
- forgotten debug logging
- unnecessary metric labels causing cardinality explosions
- equivalent metrics
- repetitive logging
- PII exposure
- excessive object serialization
Whenever possible, the preferred solution is fixing the source code.
Sometimes that’s impossible, like with third-party libraries, log-generated metrics or backend retention policies. In those cases Obics can instead configure exclusion filters, modify log-generated metrics or remove unnecessary metric labels using the capabilities provided by the observability platform.
Difference from existing approaches
Reducing observability costs isn’t a new problem, and there are already several practices to minimize them.
These approaches deal with the symptom of cost, however, not with the underlying root cause of the problem.
Sampling
Sampling reduces costs by discarding telemetry. It’s effective, but it deliberately trades visibility for lower bills.
Tiered storage
Cold and warm storage reduce retention costs while preserving historical data.
But the colder tiers come with slower queries, rehydration complexity, and limitations in capabilities.
Telemetry pipelines
Products like Cribl, Edge Delta and Mezmo act as proxy collectors that filter and route telemetry before it reaches the backend.
These solutions can reduce vendor costs, but the applications still generate the same telemetry, consuming CPU, and adding to egress costs. The optimization lives in the pipeline rather than in the code.
Switching vendors
Moving to a less expensive observability platform can reduce costs considerably.
However, it’s an expensive transition. Alerts, dashboards and log-generated metrics all need to be migrated, often to non-equivalent features. Add onboarding cost across every team, and if you’re not already on OpenTelemetry, multiply that.
Raising the minimum log level
The bluntest solution is raising everything to WARN.
That certainly reduces costs, but log level is a poor proxy for usefulness. Some of the most valuable production logs are INFO-level.
Where we’re heading
Our view is that telemetry should be maintained like any other part of a codebase.
Engineers already use static analysis, linters and dependency scanners to remove unnecessary code. We think telemetry deserves similar tooling: continuous analysis that identifies telemetry nobody needs anymore and helps remove it safely.
We’re currently working with early users and would love feedback from engineers who’ve dealt with observability costs, telemetry sprawl or high-cardinality metrics.
You can try Obics using our permanent free tier (up to 50 GB/month of telemetry), or reach out if you’d like to evaluate it on a larger environment.