Kubernetes: Custom certificate management (e.g. LetsEncrypt)?

So I've got the k8s plugin installed and I'm so excited to see cert-manager already included. I'm hoping to automate my letsencrypt setup and with a few yamls like ClusterIssuer and so on, I've got a secret populated with my own certificate via ACME. (So far nothing specific to OMV or k3s.)

However, I'm at a loss on how to configure Traefik to use my own secret (say my-cert-secret) instead of the builtin (selfsigned) default-tls-cert. Here's my thought process and lmk if I'm missing anything obvious.

Ideas / workarounds

First thing that came to mind is to use the certificate setting offered by OMV /#/services/k8s/settings . However, it reads from OMV config values, not a k8s secret. So I guess one way to do it is to add a Pod, mount my-cert-secret, mount a host file system directory, add a Bash script to write that secret into a file on OMV host, and then somehow run the update_cert script as root on host (with k3s added in deploy commands). And I'll also need some logic to detect that the secret has changed or a cron job. That seems to defeat the purpose of k8s and doesn't seem very config-driven. I may as well just do certbot on OMV host.

The second idea is to update the Traefik TLSStore to point to my-cert-secret in addition / instead of default-tls-cert. According to Traefik, I need to update the one named default but I quickly found out that it is part of a "Auto-Deploying Manifest" managed by k3s, generated by OMV. This means if I were to make my own modifications, it will be lost on restart / file change. While I can definitely override the SALT stack or add a .skip file, I'd be stuck maintaining my own version of /var/lib/rancher/k3s/server/manifests/openmediavault-traefik.yaml, which contains more than just the TLSStore itself. (A similar idea is to overwrite the builtin Certificate resource and point it to my own issuerRef. But the containing file has even more things such as the cert-manager Helm Chart installation.)

Suggestions

I understand that the plugin announcement says "Users who want to customize their Kubernetes environment are not the target of this plugin", but does it really require so much customization to run ACME within k8s itself, or am I on the wrong path? If this use case is worth supporting, I'd suggest some changes to OMV to unblock it, in the order of increasing complexity:

1. Expose an env var override for issuerRef.name, defined here. Say call it OMV_K8S_DEFAULT_TLS_CERT_ISSUER, or

2. Put the TLSStore or Certificate in a separate YAML file and give it an explicit name, so users can skip / SALT-override it as needed, or

3. Expose an env var to add more certificate secretNames to TLSStore, similar to how OMV_K8S_TRAEFIK_PORTS adds more ports, or

4. Fully-fledged LetsEncrypt (ACME) support, maybe as a recipe (fwiw, I see commented-out code for letsencrypt, so maybe it's on the roadmap)

I think any of the above will solve the use case but I'm not sure which ones aligns the most with the OMV project goal. #4 may be appealing but it involves configuring DNS-provider-specific settings, cert-manager plugins / webhooks, and username / passwords. I'll leave it for the project maintainers to decide but if anyone's open to starting small, I'm happy to contribute PRs for #1 or #2.

chente and other mods: Mind moving this thread to the Kubernetes board? Sorry I didn't realize I posted to the Storage board somehow. If not, feel free to delete it and I'd be happy to repost

Related, I've found something awkward about the domain names while I'm hacking around.

If I own the domain example.org and try to acquire the certificates for example.org + *.example.org, I'd have to set OMV network config to have hostname=example and domainname=org, for the k8s plugin to use example.org as the FQDN (in Certificate requests and so on).

But domainname is also used as the search suffix for netplan and other things in OMV salt. This means that OMV now sees my other machines on the same LAN as in .org. As in, if anything on the OMV machine tries to talk to my-laptop, it may issue a DNS query my-laptop.org, and that may reach the public internet to resolve to a public IP address (if anyone owns that domain name).

Quote from votdev

And that's why you should use internal or home as domain name as suggested by ICANN.

But this means the k8s plugin (as it is implemented right now) can never request any certificate other than selfsigned and the whole thread about ACME / LetsEncrypt is moot.

Is there a way to configure an internal / home domain for OMV in general as you suggested, but request a cert for public domain names for the public-facing parts like k8s traefik? Maybe another override env var? Or is that not a supported use case either?

(And I have a feeling that this may also necessitate two certs, one self-signed for LAN usage and another LetsEncrypt server for remote access.)

Sure, thanks for the disussion, votdev ! I've got this working end to end. Here are the testing instructions.

First edit and apply this recipe (works before any env vars, no side effects until enabled):

# It is highly recommended to try staging first to avoid rate limiting.
# Once it works, copy the solvers config to the non-staging ClusterIssuer below.
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-staging
    solvers:
      # dns01 must be used instead of http because OMV requires a wildcard cert.
      - dns01:
        # TODO: Configure your own solver, such as acmeDNS, webhook, etc.
        # https://cert-manager.io/docs/configuration/
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt
    solvers:
      # TODO: Copy your staging solver config here once it is working.

Then use the PR at: https://github.com/openmediavault/openmediavault/pull/1958

Run as root:

# omv-env set OMV_K8S_DEFAULT_TLS_CERT_ISSUER "letsencrypt-staging"
# omv-env set OMV_K8S_FQDN "my-registered-domain.example.org"
# monit restart omv-engined
# omv-salt stage run prepare
# omv-salt stage run deploy

If you go to the Dashboard etc. and you'll see the new staging Let's Encrypt cert being presented. Your browser won't trust it because it's issued by the Staging CA for now and the domain name is probably wrong.

To prove it working e2e, deploy the whoami recipe i.e. http://your-omv-server-addr/#/services/k8s/apply?recipe=whoami

You have to add a whitespace and delete or something to touch that field before you can click on the Apply button (an unrelated bug). But once it's applied, you can visit https://whoami.my-registered-domain.example.org:<your-configured-https-port> and it should again present the staging cert. You may have to add a hosts entry on your browser machine if you're trying to test this on the LAN without port forwarding. (You can see the whoami output if you ignore the cert warning, this proves that the IngressRoute routing is working, because the FQDN override applies to not only the k3s config but also the recipe.)

# /etc/hosts
# Replace with your OMV internal IP address
192.168.1.222 whoami.my-registered-domain.example.org

Finally, switch to omv-env set OMV_K8S_DEFAULT_TLS_CERT_ISSUER "letsencrypt" and re-run the salt commands above to get a real certificate from a trusted CA once you're ready. It will take a minute or two to get a real certificate and you can monitor the progress in k8s.

Quote from votdev

make use of them in the IngressRoute

Sure, but with more routes, this part isn't as easy as it seems. While one may not need to touch the default cert, they may still want to configure the default TLSStore. I've replied on https://github.com/openmediava…7#issuecomment-2864816500

Quote from votdev

special and exotic requirements.

That brings us to the earlier question of whether the overall use case of using LetsEncrypt is supported. It's not uncommon for the average user to want a LetsEncrypt certificate and it's not a far stretch to have a few apps and want this certificate to be used for them all. If you're arguing that the solution being proposed (i.e. configuring cert-manager and somehow add it to the TLSStore) is special and exotic, then I'm hoping we can agree on a more general solution, such as:

Quote from votdev

The best would be to make the https://github.com/OpenMediaVa…penmediavault-letsencrypt plugin ready for OMV7.

(Or is that also too special and exotic? Where do we draw the line?)

Suppose we can agree that the plugin is the end goal, what's the workaround until someone contributes a full port / revamp?

1. Just run acme.sh on the host and run another script such as update_cert.sh to bridge it into k8s (requires no changes to the k8s plugin)

2. An env var to the k8s plugin to add additional certs on TLSStore

3. Just install K8s on their own and configure everything manually

As I said, I'd be happy to contribute guides or edits to the wiki, but I don't want to be pushy for pull requests -- I'm really hoping to reach a consensus here. I think #3 is going a bit too far, and I think #1 is a nice thing to document in any case (correct me if I'm wrong). Does #2 fit into the picture? I agree it's not for the simplest use cases, but env vars are a bit of an advanced use case even on the wiki. Right now, the user has to do something similar to add an additional port, so adding another cert feels like the same level of customization to me. I'd be curious to hear your thoughts here. (And if the answer is a clear no on the end goal or all of #1-3, I'm not going to argue any further either.)

Quote from votdev

The better solution in my opinion would be to write an OMV K8s recipe that explains step by step how to create a Let's encrypt certificate in k8s via CR and add an explanation that the applications and Ingress must be in the same namespace. It can also be mentioned that the existing OMV K8s recipes can be reused by slightly customizing them, i.e. changing the namespace and explicitly referencing the certificate in the tls field of the IngressRoute CR. All without changing the plugin code.

I agree. Thanks for the discussion. Let me play with this setup for a few days and see how it works out. If everything goes well, I can solidify what I have into a PR for the recipes repo.

Quote from votdev

This is not possible according to what i found out. You can only store one cert in a TLSStore.

It's actually possible to have multiple, see https://doc.traefik.io/traefik…rnetes-crd/#kind-tlsstore

Yes, there can only be one defaultCertificate, which is used "for connections without a SNI, or without a matching domain". But other certificates can be added to certificates, which is an array that can contain multiple secretNames. And when the domains match (the usual case), they should just work. However, now that you've made yourself clear that it's undesirable to change the plugin at all, I don't think we need to research any further on this route.