autohttps: Implement auto_https prefer_wildcard option

[francislavoie]

Closes #5447

This implements a new auto_https prefer_wildcard option, which drops automation policies for non-wildcard domains when there's already a wildcard in another policy.

This allows users to flatten their config, and instead of using a pattern like https://caddyserver.com/docs/caddyfile/patterns#wildcard-certificates they can instead do something like:

{
	auto_https prefer_wildcard
}

*.example.com {
	tls {
		dns <provider>
	}
	respond "fallback"
}

foo.example.com {
	respond "foo"
}

This would only produce a single wildcard certificate, and no individual certificate for foo.example.com since it's already covered by the wildcard.

This also allows specifying multiple arguments to auto_https, so you can do the following to set multiple Automatic HTTPS options. Previously only one could be set at a time, which was generally fine because there wasn't actually any usecase where it would be useful:

{
	auto_https prefer_wildcard disable_redirects
}

I've only manually (visually) tested with a few simple usecases. Unfortunately we have a big lack of tests for the Automatic HTTPS logic because it manipulates config at runtime. I probably need help with testing this to make sure it doesn't have weird side effects. Thankfully, this should be safe/backwards compatible as long as users don't enable this option. We could call it experimental for now.

[mholt]

Thanks Francis, this looks appealing. Will wait for one or two people to field-test it.

[abjugard]

This looks amazing and a super useful feature to reduce subdomain discovery through certificate transparency!

I would suggest a small addition in order to minimise this even further.

E.g. to make Caddy always use wildcards.

This could maybe be implemented as another option for this directive, called force_wildcard, which figures out the minimal number of wildcard certs that need to be generated to cover all sites, and then uses your logic from this code to apply those certs to the correct sites.

Say I have two sites setup, for serviceA.domain.tld and serviceB.domain.tld, and use this setting, Caddy would then generate *.domain.tld and use it.

If I have hostX.serviceA.domain.tld, hostY.serviceA.domain.tld, hostX.serviceB.domain.tld, hostY.serviceB.domain.tld, then all of those would be covered by the *.*.domain.tld certificate (does ACME even allow such certificates? 🤔).

Alternatively, add another directive that allows users to specify a list of wildcard certs that will always be managed regardless of if they're used by sites or not, and then your feature here will be able to use those to avoid generating specific certs. A little less "magic" than the force_wildcard idea, but probably much easier to implement.

What do you think about this @francislavoie?

I guess it might be necessary to give Caddy a list of domains we control in order for this to work when using multiple domains so it doesn't see domain1.tld and domain2.tld and tries to generate *.tld... 😂

[JeDaYoshi]

If I have hostX.serviceA.domain.tld, hostY.serviceA.domain.tld, hostX.serviceB.domain.tld, hostY.serviceB.domain.tld, then all of those would be covered by the *.*.domain.tld certificate (does ACME even allow such certificates? 🤔).

@abjugard *.domain.tld won't cover sub-sub-domains. Caddy would need to generate certs for *.serviceA.domain.tld and *.serviceB.domain.tld. At least Let's Encrypt doesn't allow for *.*.domain.tld, AFAIK.

[abjugard]

If I have hostX.serviceA.domain.tld, hostY.serviceA.domain.tld, hostX.serviceB.domain.tld, hostY.serviceB.domain.tld, then all of those would be covered by the *.*.domain.tld certificate (does ACME even allow such certificates? 🤔).

@abjugard *.domain.tld won't cover sub-sub-domains. Caddy would need to generate certs for *.serviceA.domain.tld and *.serviceB.domain.tld. At least Let's Encrypt doesn't allow for *.*.domain.tld, AFAIK.

Right, unfortunate limitation but understandable.

Then I think we can simplify the behaviour and not need to specify which domains are owned, but just let Caddy automatically figure out to make wildcards for the leftmost label.

Perhaps we should just wait for this PR to get merged then I can take a crack at a force-wildcard PR?

[mholt]

I'm actually implementing part of this in CertMagic -- likely will become a "subject transformer" that allows changes to the subject name when managing certs, so, e.g. one can lob off the leftmost label and replace it with * to make it a wildcard. Of course, the Caddy interface will still need to be developed, which this PR could be. But I kind of want to come back to this after CertMagic has its change made, since it coincides with caddyserver/certmagic#280.

[mholt]

@francislavoie How would you feel if we/(I?) updated this PR to use the new SubjectTransformer introduced in the linked issue/commit? It should be relatively simple I think, if the global option is set, then set the SubjectTransformer for CertMagic configs to be a 3-line function.

[francislavoie]

I'm not sure how that would look 🤔 maybe make a branch off this one if you think it's simple?

[mholt]

I will try soon! :)

[mholt]

Actually the SubjectTransformer might be slightly tangential to this, rather than directly related -- the transformer is for, like, "I have a.b.c and b.b.c, but I want you to manage a single wildcard instead of individual certs for each specific domain." Whereas this change is, "I have *.b.c and a.b.c, and I want a.b.c to use the wildcard cert."

There's another aspect I want to consider as well, that is some users want just specific domains to be served under a wildcard, while the others shouldn't be. For example, in the config above, if there was also a site, bar.example.com, but they wanted that served with its own cert (maybe to try to obscure the fact that other subdomains are being served?) how would they do that?

I think I want to give this more thought, even before I implement anything here.

This is a needed change though and I think it's a good start. I just don't want to commit to its API/syntax/implementation quite yet until we have a better picture of the bigger picture.

[francislavoie]

There's another aspect I want to consider as well, that is some users want just specific domains to be served under a wildcard, while the others shouldn't be. For example, in the config above, if there was also a site, bar.example.com, but they wanted that served with its own cert (maybe to try to obscure the fact that other subdomains are being served?) how would they do that?

I think that's already handled by my approach, because prefer_wildcard only applies to a subdomain if there's already a wildcard cert in the config being managed (in which case it uses that policy). For other domains if you just don't have a wildcard that covers it, it'll still make a cert for that single domain. If you want to opt-out for just one domain that's covered by a wildcard, then don't use this feature and do it the handle way 🤷‍♂️

[mholt]

Ah, okay. Hmm.

I might still wait on this until after 2.8 so I can give this a little more thought. We now have a way, in CertMagic, of mapping/transforming one subject name (domain name) to another, for the purposes of cert management. Even if this PR ends up being good as-is, I just want a little more time on it.

[omltcat]

Glad there is a PR for this and I really look forward to it. Thank you for your great work. One big advantage of having separate site blocks is when using caddy-docker-proxy labels.

Since these labels can be distributed across multiple docker-compose files, there can be some possibilities of misconfiguration somewhere (especially done by multiple people). With the current handle approach, if one of the subdomain is misconfigured, causes the ENTIRE wildcard site block to be removed, bringing down everything. With this PR, the failure would at least be localized (hopefully😊)

[polarathene]

If you want to opt-out for just one domain that's covered by a wildcard, then don't use this feature and do it the handle way 🤷‍♂️

Could there not just be a tls or similar directive for a more explicit opt-out? (I don't need such functionality myself though)

It already seems to be an issue according to this report where an internal wildcard cert is being used instead of the LetsEncrypt one for an explicit site address.

Alternatively, you could go the other way around like with local_certs / tls internal, and instead have something like tls internal_wildcard or prefer_wildcard in the actual tls directive options? (assuming that could also be used to prefer FQDN as an override too).

---

Since the certmagic subject transformer feature is available and Caddy 2.8 is released, is there anything that can be done to assist moving this feature forward?

I've only manually (visually) tested with a few simple usecases. Unfortunately we have a big lack of tests for the Automatic HTTPS logic because it manipulates config at runtime.
I probably need help with testing this to make sure it doesn't have weird side effects.

If you can provide a rough outline of what to test, I could put together configs to verify?

---

One potential bug (without this PR) that already appears to exist is assigning a domain a wildcard cert from external files, and another site block for that domain with a different subdomain implicitly using the wildcard. (EDIT: Nope, that was due to incorrect auto_https mode set)

[mholt]

I am now wondering if we should make preferring wildcards the default behavior as @abjugard suggested above.

In Slack, it was expressed that it would be a breaking change, but I am not sure if there are (m)any(?) use cases that would actually break. A wildcard cert is just as good as a subdomain cert.

[polarathene]

EDIT: Ignore me. I mistook the "prefer" wildcard cert to prefer provisioning an explicit subdomain with a wildcard cert (even if no wildcard was explicitly configured/requested).

---

A wildcard cert is just as good as a subdomain cert.

Isn't it generally considered better practice to prefer explicit SAN for certs than a wildcard? Some businesses may have compliance requirements related to that expectation?

That concern is only relevant if the private key was compromised for the attacker to leverage it before expiry/detection, but that may be sufficient time for the attacker? 🤷‍♂️

I don't operate at a scale where I'm paranoid about such personally, but I could understand it being a compliance concern elsewhere, which would make it a breaking change for those users? Thus you may want to introduce as opt-in, and switch to opt-out when a breaking change is acceptable?

Unless you don't consider such implicit assumptions/trust in software to be a breaking change? You could also take the route of adding a notice (to release notes and pinned issue) for awareness before switching to opt-out, giving any potential users time to become aware of the change landing in future?

To be fair though, I think when compliance matters that upgrading to newer releases would have a more strict policy than trusting semantic versioning 😅 So perhaps I'm rambling about a non-issue.

[kanashimia]

In Slack, it was expressed that it would be a breaking change, but I am not sure if there are (m)any(?) use cases that would actually break. A wildcard cert is just as good as a subdomain cert.

If Caddy fails to obtain cert for *.example.com then foo.example.com won't have any cert too right? That seems problematic, especially because LetsEncrypt won't give you wildcard cert without DNS-01 challenge.
Anyways, default behavior can be changed later.

[abjugard]

If Caddy fails to obtain cert for *.example.com then foo.example.com won't have any cert too right? That seems problematic, especially because LetsEncrypt won't give you wildcard cert without DNS-01 challenge.

The scenario you describe here isn't really a concern as the requirements for procuring a wildcard cert are strictly different from certs for specific subdomains. Caddy (I'm guessing) won't try to get a wildcard cert unless its able to, and if its able to then it will either succeed or fail in which case you've configured it wrong and it should correctly refuse to continue starting up the site in question.

[mholt]

@kanashimia

If Caddy fails to obtain cert for *.example.com then foo.example.com won't have any cert too right?

Wildcard certs require DNS challenges which have to be explicitly configured. If someone is configuring a wildcard into their server, then they probably don't intend to get subdomain certs at all. IMO.

Anyways, default behavior can be changed later.

Well, if we default to always preferring the wildcard cert, that changes this PR. I just want to do what is most expected and intuitive, and I feel like setting up wildcard hostnames is juuust explicit and involved enough that a user probably wants to use that cert for the rest of the subdomains.

Maybe an inverse of this PR would be useful then, that is, to ignore wildcard certs.

[abjugard]

I feel like setting up wildcard hostnames is juuust explicit and involved enough that a user probably wants to use that cert for the rest of the subdomains.

100% agree.

[kanashimia]

Well, if we default to always preferring the wildcard cert, that changes this PR.

Ah yes, you are right, as specified here auto_https can't be used for disabling wildcard preference.
I was thinking about something along the lines of:

auto_https {
	disable_redirects
	prefer_wildcard off
}

Unfortunately auto_https currently lists the parameters as a disable list, with disable_redirects off it doesn't look very good. That is consistent with how it is in the JSON config though.

Wildcard certs require DNS challenges which have to be explicitly configured. If someone is configuring a wildcard into their server, then they probably don't intend to get subdomain certs at all. IMO.

Well users also could have been running with a broken wildcard cert for years and not notice anything wrong, because subdomains continue to work. I won't say that it is a strong point, I agree that it makes more sense to prefer them by default, just need to make sure that nothing major will break, and for that more testing is needed, hence why I suggest changing the default after this PR is merged, so that real world users could test it. Maybe that is too defensive 🤷🏻‍♀️

[francislavoie]

I really don't agree with any of this @mholt. As I said in Slack, I've seen many users (people asking for help in the forums) that try to load a wildcard cert into Caddy e.g. a Cloudflare Origin cert, which should not apply to all their subdomains because they might have some subdomains which are not proxied by Cloudflare (e.g. resolve to their private IP address for their LAN, non-public subdomain).

If we change the default behaviour right now, it could break existing configs that do this sort of thing. I don't feel comfortable with that at all, especially when the implementation in this PR is not vetted enough to act as a default.

I much rather take the current implementation (which is opt-in, not opt-out), have users try it and find the edgecases over time, and then consider making it the default later on.

We've been sitting on this too long, I don't want to bikeshed making it on-by-default which is a different scale of consideration than having it opt-in, I want to get this out there so people can use it ASAP, as an experimental feature.

[polarathene]

If someone is configuring a wildcard into their server, then they probably don't intend to get subdomain certs at all. IMO.

Well there is this report where someone does want some subdomains to not use wildcard.

While the feature is named as prefer wildcard, I don't suppose there would be a way to ensure that only wildcards are provisioned when valid? As in if it fails, to opt-out of a fallback to explicit FQDN?

This is due to CT public logs concern, which presently is avoided with a wildcard site address, but with this feature the benefit is explicit site addresses that could use a wildcard won't be as trustworthy for that convenience if wildcard provisioning falls back to explicit FQDNs.

---

I've seen many users (people asking for help in the forums) that try to load a wildcard cert into Caddy e.g. a Cloudflare Origin cert, which should not apply to all their subdomains because they might have some subdomains which are not proxied by Cloudflare (e.g. resolve to their private IP address for their LAN, non-public subdomain).

Could we have an example of that? I have seen users mixing public + internal certs for the same domain, and earlier in this PR discussion suggested tls directive option for preference instead of auto_https global might be worth considering.

I've got a public wildcard and server, as well as local services that I can connect to over my LAN with split DNS using that same domain and wildcard cert but resolving to a different IP. That is quite a common practice for self-host enthusiasts.

I'm not familiar with Cloudflare Origin, but assume that's different to a public CA cert like from LetsEncrypt with the bulk of devices will have trust store coverage. I could understand with tls internal wildcard where that would not be the case, if I did want to expose some services with a LetsEncrypt cert for devices to use instead of requiring adding my private CA to their trust stores that'd perhaps be similar to whatever the Cloudflare Origin concern is?

---

I much rather take the current implementation (which is opt-in, not opt-out), have users try it and find the edgecases over time, and then consider making it the default later on.

I would also suggest this approach 👍

You can always bump to a Caddy 3.0 release at a later point when it makes sense. A new major release doesn't need to be due to a massive rewrite / refactor. Or if acceptable switching the default after sufficient notice in a future minor release as suggested.

I want to get this out there so people can use it ASAP, as an experimental feature.

There's already some features marked as experimental right? Or has been in the past, that's a great approach 🚀

[francislavoie]

I'm not familiar with Cloudflare Origin, but assume that's different to a public CA cert like from LetsEncrypt with the bulk of devices will have trust store coverage.

It's a cert issued by Cloudflare's non-publicly-trusted CA which only their servers trust, allowing HTTPS between Cloudflare and the upstream server without the upstream needing to have a publicly trusted cert. It was the go-to option before ACME & DNS challenge became more common-place.

Could we have an example of that?

I didn't make note of them, but one recently this week on discord that I can't really link to.

You can always bump to a Caddy 3.0 release at a later point when it makes sense.

No plan to do a v3 anytime soon. Problem is it would be a breaking change to the entire plugin ecosystem because we'd need to change the Go package to have /v3 in it. Go's module ecosystem kinda tied our hands. Caddy is not semver.

There's already some features marked as experimental right?

Many, yes. Via godoc comments or elsewhere in the docs with ⚠️ Experimental.