Debugging TLS failures in distroless containers

Summary

Last week at work we had (yet another) TLS-related incident. Services owned by my team have to interact with a multitude of 3rd party providers via HTTP, each of them requiring mTLS as additional security constraint: as such, most of the times when we see a degradation of service of any sort related to TLS, we're quite confident pointing the finger at certificates we present as clients during TLS handshake. This time it wasn't the case: this time, the issue was caused by our client don't trusting the CA root certificate presented by the server (the 3rd party service). Before diving into how we troubleshoot this problem, let's refresh our mind on basic TLS concepts. What is TLS? Transport Layer Security (TLS) - formerly called SSL - is an encryption protocol used to secure communication between a client and a server, preventing risks like eavesdropping and man-in-the-middle attacks. TLS utilizes public key cryptography, which involves a pair of keys: a public key and a private key. Data encrypted with the public key can only be decrypted using the corresponding private key. As a result, when a server successfully decrypts a message encrypted with its public key, it verifies that it holds the associated private key. The public key is accessible to anyone through the server’s or domain’s TLS certificate. A TLS certificate is a data file that contains important information for verifying a server's or device's identity, including the public key, a statement of who issued the certificate (TLS certificates are issued by a certificate authority), and the certificate's expiration date. The TLS handshake is the process for verifying the TLS certificate and the server's possession of the private key. The TLS handshake also establishes how encryption will take place once the handshake is finished. For what concerns this article, we're mainly interested into point 3, where client verifies server's certificate: one of the checks done is to verify if it can trust the CA (Certifica...