MongoDB + Let's Encrypt Replica Set Issue: The tlsWithholdClientCertificate Gotcha

Summary

Let's Encrypt stopped issuing certificates with the clientAuth EKU; MongoDB replica set members use the same cert as both server and client, so member-to-member connections started failing. Fix: set tlsWithholdClientCertificate: true under setParameter in your mongodb config — not under net.tls.

What happened

Our MongoDB replica set had been running with Let's Encrypt TLS in production for a long time. After a certificate renewal it stopped working. We had to solve two things: (1) what went wrong — the root cause, and (2) how to fix it correctly — where to put the setting.

Root cause: Let's Encrypt changed certificate issuance so certs are for server authentication only (discussion and examples; see also Let's Encrypt's announcements on certificate usage and EKU). MongoDB nodes act as both TLS servers and clients; when they present the cert as a client, the primary can reject it ("unsuitable certificate purpose"). The fix is to tell MongoDB not to send the cert on outbound member-to-member connections: tlsWithholdClientCertificate: true.

What pointed us to it: The log line "unsuitable certificate purpose" (in the SSL peer certificate validation failed / SSLHandshakeFailed errors above) was the clue. It means the peer's certificate was valid but not allowed for the use the server expected — i.e. the cert didn't have the client-auth EKU that Let's Encrypt no longer issues. Seeing that in the logs led us to the EKU/client-auth explanation and then to tlsWithholdClientCertificate.

The gotcha: That option is a MongoDB server parameter, not a TLS config option. If you put it under net.tls, MongoDB will ignore it or misbehave. Put it under setParameter:

setParameter:
  tlsWithholdClientCertificate: true

Example working config

net:
  tls:
    mode: requireTLS
    certificateKeyFile: /etc/mongodb/ssl/client.pem
    allowConnectionsWithoutCertificates: true

security:
  keyFile: /etc/mongod-replica-set-access-control-key

setParameter:
  tlsUseSystemCA: true
  tlsWithholdClientCertificate: true

So: use tlsWithholdClientCertificate, and put it under setParameter, not net.tls. Sometimes the hardest production bugs come down to a single indentation level.

Symptom: what you'll see

Replica set members stop talking to each other. New members can't join; existing members may show as unreachable. Connections are accepted, the TLS handshake completes on the wire, then the server closes the connection. In the MongoDB logs on the node that's receiving the connection (typically the primary), you'll see repeated errors like:

{"s":"E","c":"NETWORK","id":23256,"msg":"SSL peer certificate validation failed","attr":{"error":"SSL peer certificate validation failed: unsuitable certificate purpose"}}
{"s":"I","c":"EXECUTOR","id":22988,"msg":"Error receiving request from client. Ending connection from remote","attr":{"error":{"code":141,"codeName":"SSLHandshakeFailed","errmsg":"SSL peer certificate validation failed: unsuitable certificate purpose"}}}