The Label Changed. The Prometheus Series Didn't Disappear.

A database node changed operational role during a topology change. The collector began publishing the new label set correctly, but the dashboard still behaved as though the old role existed. Worse, an aggregate health expression remained healthy even when current state should have triggered an alert.

The confusing part was that the collector’s current value was correct. The bad result came from an old label combination.

What the exporter thought it had§

Before the change:

db_replica_healthy{
node="db-node-a",
role="live",
cluster="cluster-a",
replication_pair="pair-a"
} 1

After the node changed role and topology:

db_replica_healthy{
node="db-node-a",
role="dr",
cluster="cluster-b",
replication_pair="pair-a"
} 0

It is tempting to think the second update changes the first metric. It does not. From the metric registry’s point of view these are different children, because the label tuple is different — (metric_name, node, role, cluster, replication_pair) is the identity. Change role or cluster and you have created another series, not updated the old one.

The in-process surprise§

The Python Prometheus client keeps a child metric for every label combination created in the process. Changing the labels used by later collection cycles does not automatically delete the old child from the registry. The exporter could therefore still expose both series at once:

db_replica_healthy{node="db-node-a",role="live",cluster="cluster-a",replication_pair="pair-a"} 1
db_replica_healthy{node="db-node-a",role="dr",cluster="cluster-b",replication_pair="pair-a"}   0

The first series was stale from the application’s point of view, but the exporter process was still serving it with its last value. This was not just a Prometheus-retention issue — the exporter itself was continuing to present the obsolete identity.

Why the dashboard looked healthy§

The dashboard used an aggregate similar in shape to:

max by (replication_pair) (
  db_replica_healthy
)

This is the dangerous form of observability failure: not a missing graph, but a graph that confidently says the system is fine.

The fix: prune label tuples that are no longer valid§

The collector already knew the current label tuple for the node. Before publishing current health, I added a pruning step: for the logical node, inspect the registered child label tuples, remove any whose mutable labels no longer match current metadata, and only then set the current metric.

The production implementation used the Python client’s internal metric-child state to identify and remove stale labelsets. That made the exporter self-healing on later collection cycles — when a role or cluster assignment changed, the old child was removed before the new state was published. Existing stale children from the previously running version disappeared after restart/redeployment.

The better modelling question§

The bug exposed a more important issue: should role and cluster have been mutable labels on that gauge at all?

Prometheus works best when labels describe relatively stable dimensions of a series. If a property changes during the normal lifecycle of an object, putting it into series identity creates lifecycle work — how is the old series removed, how long can both versions coexist, which one should an aggregate use, and what does historical continuity mean across the change? Possible approaches are to keep identity labels stable and expose mutable topology metadata separately, or to make child deletion an explicit part of the collector protocol.

A useful mental model§

I now treat a Prometheus metric approximately like a table whose primary key is metric name + full label set. Changing a label is closer to inserting a new row than updating an old one, unless something explicitly deletes the old identity. That model makes the failure obvious.

Takeaway§

The collector had the correct current value. The dashboard query was syntactically valid. Prometheus was doing what it was asked to do. The bug lived in the identity model between them.

That is why observability code deserves the same data-modelling discipline as application data. A label is not just text attached to a number. It is part of what the number is.