Case study

Expired certificates broke production twice, and nothing turned red

Once as a load-balancer black hole that swallowed 23 of 24 connections for fifteen hours, once as a silent 30-hour replica lag behind a database that kept answering. Both times the metrics existed and the alerts did not.

3 min read · Markdown

TL;DR

• A PostgreSQL read-only pool. One replica's client certificate between the pooler and PostgreSQL expired. The node failed every login instantly, so a least-connections balancer kept sending it new connections: 23 of 24. The pool was broken for 15 hours before anyone noticed.
• A two-node ClickHouse cluster. A 90-day certificate expired on both nodes at once. Replication between them stopped; one replica fell 30 hours behind on 37–39 tables while clients saw no errors, so half of all reads returned day-old data.
• The lessons: alert on expiry, make health checks perform the real handshake, never pair least-connections with a backend that fails fast, and watch lag on every replica.

Incident one: the black hole

The read-only port of a PostgreSQL cluster is a virtual IP in front of three replicas. On each node pgbouncer runs in session mode and connects to the local PostgreSQL with a TLS client certificate. HAProxy balances with leastconn.

What happened:

• On one node the pooler's client certificate expired in the middle of the night. PostgreSQL rejected the pooler: certificate verify failed. The pooler told clients server login has been failing, cached error: server conn crashed?.
• The node was healthy by every check that existed: PostgreSQL was up, TCP accepted connections.
• A node that fails every connection instantly always has the fewest active connections, so least-connections sent it almost everything new. We counted 23 of 24 new connections going to the broken replica.
• Errors looked intermittent from the client side, because the occasional connection that reached a healthy replica succeeded. The outage lasted fifteen hours before anyone looked.
• The certificate's expiry date was already collected as a metric. Nobody had put an alert on it.

How to find it fast. Take the certificate straight from the virtual IP (a TLS handshake after PostgreSQL's SSLRequest) and read its subject alternative names: they name the node behind it. The workaround is to take the node out of the pool, or to point the service at the primary until the certificate is reissued.

Incident two: the silent lag

A two-node ClickHouse cluster served reads for a seller-facing portal, split roughly evenly between the nodes. Both nodes used one certificate with a 90-day lifetime for the native TLS port and for interserver replication. It was issued on the same day for both, and it expired on the same day for both.

At expiry:

• Replication stopped. Fetching parts between replicas failed: CERTIFICATE_VERIFY_FAILED on one side, SSLV3_ALERT_CERTIFICATE_EXPIRED on the other.
• Retries exhausted the ephemeral ports. Cannot assign requested address, and about 1.65 million ServerErrorHandler errors an hour in text_log, up from dozens.
• The lag grew to about 30 hours on 37–39 tables, with a replication queue of about 19,000 entries. The other node lagged on the three tables that were written through the first.
• Clients saw nothing. Queries succeeded on both nodes, and half of the answers came from data a day old.
• It was found by accident, a day later, while investigating the noise in text_log.

The certificate was reissued the next evening; the replicas caught up within a week. The next expiry is in 90 days.

How to find it fast. When replicas lag or network errors spike, check the certificate first: openssl s_client -connect <host>:<tls-port>. Look at lag on every replica, not just the one your tools connect to by default.

Takeaways

• Every certificate needs an expiry alert. Usually the metric already exists; the alert is a line of configuration. 90-day certificates need automated renewal, or at least a check that runs weeks before expiry.
• Health checks must exercise the real path. "Port open" and "process up" said nothing about whether a client could log in.
• Least-connections plus a fast-failing backend is a black hole. Use health checks that detect failed logins, or a balancing method that does not reward failure.
• Errors that clients don't see are the dangerous ones. Replication lag needs its own alert on every replica, independent of query errors.
• Shared certificates fail together. Two nodes with the same issue date gave us zero redundancy against expiry.

All writing · A similar problem? contact@ivanblinov.com