How to remove certificate from Traefik acme storage when saved to consul KV - traefik

Question

I have Traefik running with a Consul KV store. How do I remove a record from the acme certificate storage in Consul, or force a renewal for just one domain/frontend?
Problem:
Somehow one of the frontend domains has saved with the wrong certificate. It's referencing a certificate from a different domain (which is also a frontend in Traefik).
I was able to inspect the acme json by getting the consul value for the traefik/acme/account/object key, decode and unzip it and this is the record from the Certs array:
{
"Domains":{
"Main":"my.domain1.com",
"SANs":null
},
"Certificate":{
"Domain":"my.domain2.com",
"CertURL":"https://acme-v02.api.letsencrypt.org/acme/cert/idfordomain2",
"CertStableURL":"https://acme-v02.api.letsencrypt.org/acme/cert/idfordomain2",
"PrivateKey":"...",
"Certificate":"..."
}
}
As you can see, somehow the cert for my.domain2.com has been saved against the record for my.domain1.com so this results in an invalid certificate warning in the browser. I want to clear out the whole record so Traefik will get a fresh cert. I'm using Consul and its saved in binary so I can't just edit the json.

Answer 1

Here is how I solved this issue:
Your traefik network should be marked as attachable: true
Run on host:
docker run -it --rm --name consul-client --network traefik_traefik consul sh
Then run in created container:
export CONSUL_HTTP_ADDR=consul:8500
# get value from consul and store it to acme.json
consul kv get traefik/acme/account/object | gzip -dc > acme.json
# remove invalid domain and store it to acme-fixed.json
cat acme.json | jq -r 'del (.DomainsCertificate.Certs[] | select(.Domains.Main=="'yourdomain.com'"))' > acme-fixed.json
# gzip it
cat acme-fixed.json | gzip -c > acme-fixed.json.gz
# upload fixed and gzipped json back to consul
consul kv put traefik/acme/account/object #acme-fixed.json.gz

Answer 2

Simplest way is to use consul CLI utility. The utility is also used to run server and ideally you should use same version as the one used for your servers. Make sure you export environment variables: CONSUL_HTTP_ADDR - points to consul server, default is http://127.0.0.1:8500 and CONSUL_HTTP_TOKEN - is ACL token, if you have ACLs on your server enabled, as it should be on production environments.
Then you just run following command
consul kv put traefik/acme/account/object #traefik.json
Where trafik.json is json file that has updated values, that you wish to change in Consul KV store.
Or you can use HTTP API: Consul Create/Update Key
curl -X PUT --data #traefik.json http://<your-server-url>:<port>/v1/kv/traefik/acme/account/object
If your server is ACL enabled, you need to add following header to curl request, with <your-acl-token> that was issued to you: -H "X-Consul-Token: <your-acl-token>"