How do you manage the distribution of internal TLS network certificates? I’m using cert-manager to generate them, but the root self-signed certificate expires monthly which makes distribution to devices outside of K8s a challenge. It’s a PITA to keep doing this for the tablet, laptop and phones. I can bump the root cert to a year, but I’m concerned that the date will sneak up on me. Are there any automated solutions?
For most of my internal services that are sitting behind Traefik I use step-ca which basically gives you a Let’s Encrypt style certificate while working over the local network. The root CA has a long expiry (so might not be what you want if your goal Is a short lived root CA) but the actual certificates for each service are short lived (a touch over 24 hours from memory?)
I tried step-ca to start with, but my primary use case was for certs in the cluster, which cert-manager is more suited for natively. Maybe step-ca has improved, I was using it in the early days. My goal isn’t a short lived cert as much as it is to have an easy configuration and to learn.
Bump root cert to 10 years and use intermediate with shorter lifetime. root cert should be stored and processed off net.
I think this is what I’m going to do.
Some of my internal stuff goes out to Let’s Encrypt, so I don’t worry about it at all. My internal AD stuff is set for like three years. If anyone has compromised the CA, they’re already past where issuing malicious certs would be useful.
I would up your root cert expiration. You can keep the root CA offline if you’re concerned about compromise.
There are also ways to run LE-style automatic renewals internally, but I’ve never bothered because what I’ve described above means I don’t need it.
Use a secret manager?
Cert is a secret, add a small agent to your containers that pings your secret manager and gets back the current cert. Then saves / imports it (or whatever is appropriate).
DNS challenge with a reverse proxy is that answer. I’ve been doing this for a while now and it works great. Most other answers here are work arounds or not very robust.
This is the way: https://youtu.be/liV3c9m_OX8
I do this with authentik for sso
I have local only things like vaultwarden and external things like seafile.
I started watching the video. I was not aware that LetsEncrypt supported wildcard certificates. Does this mean that your internal network uses the same domain name as your externally-hosted services?
Yes.
And
This is the best and most robust way to do this
Not sure if it’d fit your use case 100%, but this has been a nice middle ground solution for LE certs in my lab: https://www.certwarden.com/
Hadn’t heard of this before. I’ll give it a whirl.
Is *warden the new hotness? Bitwarden, vaultwarden, linkwarden and now certwarden?
I looked into this recently.
There’s a DNS challenge designed for this exact scenario called (from memory) DNS01, but it’s more of a faff than I’m willing to get involved with.
Basically you push proof that you own the domain into a DNS record instead of to a file on a web server. It requires a DNS provider that has an API and a client that speaks that API.
It also leaks private DNS stuff into the public domain.
I’d love it if someone devised an easier way. Maybe there is an easier way?
This is me.
In public dns, configure *.home.example.com as an A record pointing to the local IP for my traefik container.
Traefik then manages all certificates. It sets a TXT record with my dns providers API like privatesercice.home.lebowheatcroft.com, requests the cert from letsencrypt, then deletes that TXT record.
Yes the local IP of my server is leaked, but names of services are not.
MONTHLY?? That’s a bit much, don’t you think?
If you’re regenerating certa that fast, I can’t think of anything that’s going to secure AND easy enough to satisfy automating this.
Whatever tool you want to use to secure the contents of the cert from its initial creation, to distribution, is fine enough. If you want super easy, use an SSH/SCP script. If you want something more elegant, think Hashicorp Vault or etcd.
Ansible is probably more effort than it’s worth (plus securing the secrets of the cert), and any other config mgmt tool won’t deal with the distribution portion simply, so I’d skip all of that.
Yeah, either secure copy (SCP) or rsync are the ways to do this securely.