Those tickets sound fairly non-dramatic - a trusted root certificate was dual-use, in breach of root certificate policy, and have committed to transition to TLS-only certificates and remove trust in the old ones.
But the Go issue is market urgent, and assigns it a CVE, so clearly there's more to it than that.
It is a shame that you cannot have a cert signed by multiple CAs.
(The format seems like it would allow it, but nothing supports it, and in practice CAs demand the ability to actually modify some fields, and then sign that.)
It is done with intermediate certs, though! It's a common method of establishing trust in new root certs: sign the intermediate with an already-trusted root cert, then wait for the community to adopt the new root cert over the years.
For example, the initial Let's Encrypt Authority X3 was signed by both their own (new, not-yet-universally-trusted) ISRG Root X1, and the universally-trusted third-party IdenTrust DST Root CA X3.
One crucial detail is that it isn't the same certificate. Rather, it is two separate certificates which just so happen to be using the same key pair, so the leaf cert is in essence signed by both intermediate certs at once.
I wonder if it would be possible to do the same thing on your own, actually: can a server present two completely separate cert chains to a connecting client - as long as the leaves use the same key pair? Technically AFAIK the only requirements should be 1) the key pair in use for the connection matches the cert, 2) the domain matches the cert's Subject, and 3) the cert is trusted, so at first glance it might be possible...
If the client uses "certificate_authorities" extension to list support CA's, a TLD server that holds multiple x509 certificates can present to the client the certificate signed by the clients trusted CA.
Of course, the browsers don't use the "certificate_authorities" because the list would be hilariously large for a TLS CLient Hello packet.
Yeah, that's one of the complications. It would have to be ignored in processing, or multiple variants be presented at once (preferred).
Certificates are not the same as the public key; multiple certs sharing the same key is essentially the way around this. It's not as elegant as identical signing of the same cert, but it could work.
Identical signing of the same cert also runs into issues like "who gets to decide the serial number" and "how to deal with revocation lists: all of, or any of?".
And turning fields like issuer and serial into arrays risks someone accidentally using index 0 for one field and index 1 for another.
Combine that with the insane back-and-forth nightmare that the issuance process would turn into, and I have a very hard time seeing the benefit in it. After all, nobody is stopping you from requesting multiple certs independently, watching the revocation logs, and switching to the backup when required.
The point is limit the damage that distrusting badly-behaved CAs do. Right now there's a lot of blast damage as anyone using them suddenly has their stuff break. If multiple signatures were common, then a given CA could be distrusted without hurting as many others, so it would actually be a viable threat to keep them in line.
Another option: Use ACME together with ARI (RFC 9773). The client checks ARI data every hour. If the cert needs to be replaced – for example like in this scenario – the CA will inform the client, which in turn would then request a new cert.
I know it’s not convenient to automate for every kind of device though. I guess in some of those cases it might be better to have a load balancer in front of the device. Or use an alternative PKI.
I think certificates should be like badges that you can collect – the more badges the higher the trust. So if you are a French company, maybe the tax office in France can sign your certificate to show that you indeed is a real company. If you are accepting card payments – Visa and Mastercard could sign your certificate. If you have a domain the TLD owner could sign your certificate to show that it's indeed your domain… And so on…
And then the browser could present these badges in a nice way to the user.
I just worry that this would just bring us back to the: "Pay $$$$$$ for a certificate, those pesky lets encrypt free certs are not expensive enough to give trust", BS that always happens.
Every time there is a CA issue, someone comes up with "this would be so much better if we'd make it more complex than it already is, as it would make it a bit more convenient to deal with the most recent issue".
I don't think that's the right way to think about it.
Web of trust required a lot of work by users and was pretty bad in practice. This isn't that.
This would effectively be the same "root CAs come pre-installed" but they would be notaries rather than root CAs, with a predefined trust policy (maybe minimum of 2 notaries per leaf certificate)
Users would still just see a green lock icon same as before, no action needed.
Clearly the platform isn't designed to do this. Nobody thought this was going to be a necessary facility. They have just crammed an array of numbers as literal values deep into the standard library. So clearly this is an exceptional event, and not the exercise of a feature that was foreseen.
It's mostly historical. They used to suck in the past, but the implementations remain. Nowadays you are better off configuring to use OS certificates if possible.
I really appreciate Firefox to have their own certificate DB because it allows to create separate profiles with different trusted certificate sets. The country where I live issued it's own root certificates which I have reasons not to trust in all cases but when accessing government services etc. So I've created a separate Firefox profile where I've installed those malicious certificates, keeping the main Firefox profile clean.
It's nice that firefox implemented that but it's the wrong place to do it IMO. It's the sort of thing that should be built into the OS along with fine grained permissions and sandboxing. The browser is merely providing a bandaid over a deficiency.
Yes, as I understand it, this sort of problem is what “Name Constraints”[0] were designed to solve. (Well, really, they were intended to be used with corporate internal CAs[1], but there's no reason they wouldn't work more broadly, too.)
At least it kind of makes for them specifically, since Netscape did SSL initially before OSes had that functionality. Their mailing list, as far as I know, is still the canonical place for public discussion about the infrastructure.
Firefox is a browser so it make sense to implement their TLS like that.
But we're talking about a lib of a programming language here, things happened in there effects your own reputation when you ship something built on top of it.
So the real question is, has others such as OpenSSL done the same? If yes, then yeah it's normal, but if no then I don't trust people from Google.
It's probably about power. Certs control is getting higher value, so control over this policy means more negotiation power with whatever states (governments) need from platforms.
It's easy to imagine something like: "give us some grant taxpayers money and we'll add your cert to our trusted list/remove theirs cert/disable cert when run on this hardware/etc."
I mean, they're the TLS certs are baked into the stdlib 'tls' package, not the compiler itself.
It's just an unfortunate unforced error that the go TLS package is bundled along with the compiler, in the same way Safari being bundled with iOS updates is an unforced error meaning a safari security bug requires a full OS update.
Oh, wow, that makes it even sillier. If I want to delete certs from my system certificate store, I can do that, the stdlib should not be making that choice if it's not even packaging the things.
These are intermediate certificates, and the system certificate store on Linux provides no way to tell verifiers such as Go's crypto/x509 that an intermediate is revoked. So Go really has no other option here.
Yeah, this is making me real glad that I know C well enough to stop being afraid of it.
Pinning certificates in a browser was a brilliant idea for Chrome because web browsers are not made for technical users.
But pinning certificates in a compiler? It's reminding me of that one time where the NSA handed out DES constants to use and said "just trust me, bro". I don't even care that the DES constants ultimately ended up being protections against differential-cryptanalysis SCA (which, btw, the NSA then undermined by arguing for shorter keylengths in "export-grade crypto").
https://gist.github.com/AGWA/5a78186e0f4a7d537b8891a7b46f945... is the list of certificates in Certificate Transparency issued by the blocked CAs as of this time. They appear to be non-malicious test certificates, and the DNS name in the first certificate indicates who likely discovered the issue.
What's alarming and unprecedented is that yesterday, two Certificate Transparency log implementations were updated to block the logging of certificates issued by the CAs. This likely indicates total private key compromise, and the blocking by CT logs is to prevent the logs from being spammed by forged certificates. However, it also means there will be no place to record forged certificates found in the wild. (These certificates won't validate in CT-enforcing clients since they won't be in CT logs, but could be used to attack non-CT-enforcing clients.)
Am I off base, or should I be surprised that you were able to get the mentioned cert issued in the first place this recently (yesterday)? It seems unexpected that certificates are being actively issued and logged coming from a to-be-forcibly-distrusted part of the hierarchy.
The bugzilla issues mentioned elsewhere suggest at least a mostly orderly removal of trust with issuance stopped several months ago. Your new cert was issued by “CN=EM DV TLS CA - G3B,O=eMudhra BV,C=NL”, which presumably chains to the G3 root, and the bugzilla issue seems to say that G3 was going to be dropped from the root lists some time ago but that maybe it would be cross-signed by something still trusted. So I assume that something else is going on.
Was that cert issued via normal means (e.g. ACME)?
CT is for logging certificates from trusted CAs. Logging certificates from untrusted CAs is out of scope, and can be harmful as untrusted certificates could be used to spam the log until it's unavailable.
Of course, the notion of "trusted" is complicated. These CAs are no longer trusted by up-to-date clients, but out-of-date clients may continue to trust them. The needs of up-to-date clients take priority over the needs of out-of-date clients.
Alice User goes to Bob's Website on the wifi at Eve's Coffeeshop. Can Eve read or inject sensitive information into Alice's browsing session?
Normally Eve's attempts fail because Alice requires a message to be signed with the private key matching the public key in a certificate for bob.example.com signed by a legitimate CA.
Unfortunately Bob bought his CA from Mal's Domains, and Mal is working with Eve for some reason (for example, Mal has been bribed, hacked, intimidated, legally pressured, etc.)
So in addition to Bob's regular on-the-books cert, Mal issues a second off-the-books cert for bob.example.com and gives it to Eve.
With CT, Alice's browser doesn't just require the cert's pubkey, it requires the server to provide a Merkle proof it was included in a CT log.
The CT log puts Eve and Mal on the horns of a dilemma: If they keep their new cert out of the CT logs, Alice's browser won't accept it. If they put it in the CT logs, Bob can read the log himself (it's public) and Bob can say "Hold it! I didn't order this!"
Bob will switch away from Mal as a TLS provider and complain "Hey everybody, Mal issued this cert I didn't order, customers should stop using him and the people who create CA lists for browsers and OS's should closely investigate whether Mal's CA activities are legitimate." Mal can't deny issuing the questionable certificate, the evidence is in public append-only CT logs for everyone to see.
Did the log operators offer an explanation? You say "likely indicates" and I agree, but I guess I'd expect some noise on m.d.s.policy or somewhere similar, maybe a CT-related site I don't look at.
> the blocking by CT logs is to prevent the logs from being spammed by forged certificates. However, it also means there will be no place to record forged certificates found in the wild.
This is indeed huge. Allegedly forged certs aren't spam and do need to be logged. Refusal to log them is a dereliction of duty and will righly undermine trust in the system.
The whole premise of CT logs is that issuance of a cert is an action taken by the CA or at least by the HSM holding the private key. If the entire private key leaks, which appears to be what happened here, then the existence of a certificate containing a valid signature means nothing at all. The CT log might as well record that every possible string of bytes has been signed.
"Certificate Transparency logs are "append-only" and publicly-auditable ledgers of certificates being created, updated, and expired."
https://certificate.transparency.dev/logs/
Secretly excluding some issued with a key leak that "appears to be what happened" is not transparency.
CT aren't supposed to be gatekeepers.
If the problem is spam then the solution should be antispam - regardless of allegations of leak.
CT logs are supposed to be useful. They serve at least two major purposes: visibility and auditing. Assuming that a CA is remotely well behaved, a site owner can trust that, if a certificate has been issued, then it's in the CT log. An auditor can read a complete record of certificates issued by a CA and attempt to determine whether they all follow the rules. And a CA that issues certificates without logging them is quite likely to get caught. Some browsers won't even accept certificates that aren't logged. And anyone who sees a signed certificate that isn't in the log holds evidence that the CA is misbehaving.
Once the private key has been leaked, this is basically all pointless.
Can a site owner trust that a certificate in their name has been logged? Of course not! An attacker could be signing it on the fly.
Can an auditor read the complete record? The complete record is kind of meaningless if the private key is leaked. But the auditor already knows that the private key is leaked!
Will the CA get caught if they issue a cert that isn't in the log? Yes! In fact, all newly signed certs won't be in the log.
Will browsers that check for log inclusion be protected? Yes, because newly signed certs won't be in the log.
Will anyone who sees a signed cert that isn't in the log hold evidence of misbehavior? Yes!
One might argue that the actual change to the CT system that should be made is a way for a CT operator to insert a statement into the append-only log that they believe a signing key to be compromised and that they will therefore no longer accept certificates signed by that key.
So even if your OS trusts these certs, programs written in Go won't. This is extremely unusual and makes me think someone got a National Security Letter.
Why the hell does Go ship its own root CA list? This is a task for the operating system, and at the very least Windows, macOS and most flavors of Linux ship an OS-wide truststore. It's annoying enough with Java to keep stuff updated...
> This is an additional, ad-hoc blocklist for compromised intermediaries, it's not even roots.
Nonetheless, there shouldn't be a mechanism to override or modify the OS trust list embedded in the runtime. If the application code itself wants to do that (e.g. cert pinning), fine. But as an ops person, I repeat myself, it's an utter headache to deal with SSL trust roots already, and really I don't want to have to learn about N different ways that applications manage this crap.
And how many more times do we need to read news like this before we replace Web PKI with a system that isn't as completely insane?
We already have a functioning system for delegating domain ownership in a hierarchical fashion where compromises are limited to only the subtree under the compromised entity. Let's start using it for domain validation sooner rather than later.
Eek! You've summoned Tom Ptacek to tell you why it's obviously better (and simpler) to maintain a second, parallel system that trusts dozens of parties to sign for any name they feel like.
Of course we should have done that ages ago, but, you see, 20 years ago or so, Verisign (and maybe others) had a profit stream to protect and people to send to the IETF. And it's just possible there may have been other players with vested interests in things not getting too secure.
[OP] mendym | 19 hours ago
data from https://www.ccadb.org/resources
ericbarrett | 18 hours ago
https://bugzilla.mozilla.org/show_bug.cgi?id=2083709
https://bugzilla.mozilla.org/show_bug.cgi?id=2018979
tveita | 11 hours ago
But the Go issue is market urgent, and assigns it a CVE, so clearly there's more to it than that.
ericbarrett | 9 hours ago
wnoise | 18 hours ago
(The format seems like it would allow it, but nothing supports it, and in practice CAs demand the ability to actually modify some fields, and then sign that.)
crote | 17 hours ago
For example, the initial Let's Encrypt Authority X3 was signed by both their own (new, not-yet-universally-trusted) ISRG Root X1, and the universally-trusted third-party IdenTrust DST Root CA X3.
One crucial detail is that it isn't the same certificate. Rather, it is two separate certificates which just so happen to be using the same key pair, so the leaf cert is in essence signed by both intermediate certs at once.
I wonder if it would be possible to do the same thing on your own, actually: can a server present two completely separate cert chains to a connecting client - as long as the leaves use the same key pair? Technically AFAIK the only requirements should be 1) the key pair in use for the connection matches the cert, 2) the domain matches the cert's Subject, and 3) the cert is trusted, so at first glance it might be possible...
wnoise | 17 hours ago
GoblinSlayer | 14 hours ago
happosai | 9 hours ago
Of course, the browsers don't use the "certificate_authorities" because the list would be hilariously large for a TLS CLient Hello packet.
lmz | 17 hours ago
wnoise | 17 hours ago
Certificates are not the same as the public key; multiple certs sharing the same key is essentially the way around this. It's not as elegant as identical signing of the same cert, but it could work.
crote | 16 hours ago
And turning fields like issuer and serial into arrays risks someone accidentally using index 0 for one field and index 1 for another.
Combine that with the insane back-and-forth nightmare that the issuance process would turn into, and I have a very hard time seeing the benefit in it. After all, nobody is stopping you from requesting multiple certs independently, watching the revocation logs, and switching to the backup when required.
wnoise | an hour ago
cpach | 16 hours ago
Another option: Use ACME together with ARI (RFC 9773). The client checks ARI data every hour. If the cert needs to be replaced – for example like in this scenario – the CA will inform the client, which in turn would then request a new cert.
I know it’s not convenient to automate for every kind of device though. I guess in some of those cases it might be better to have a load balancer in front of the device. Or use an alternative PKI.
Puts | 14 hours ago
And then the browser could present these badges in a nice way to the user.
LoganDark | 14 hours ago
And then you have that one website trying to collect them all.
calgoo | 13 hours ago
hannob | 13 hours ago
I don't think that's the right way to think about it.
ycombinatrix | 13 hours ago
shame it wasn't adopted. rent seeking CAs would never go for it.
cowlevel | 9 hours ago
ycombinatrix | an hour ago
This would effectively be the same "root CAs come pre-installed" but they would be notaries rather than root CAs, with a predefined trust policy (maybe minimum of 2 notaries per leaf certificate)
Users would still just see a green lock icon same as before, no action needed.
jeffbee | 17 hours ago
fractorial | 17 hours ago
jeffbee | 17 hours ago
fractorial | 17 hours ago
zx8080 | 14 hours ago
captn3m0 | 9 hours ago
stephbook | 17 hours ago
I wonder what's so annoying about OS certs.
whatevaa | 17 hours ago
quiet35 | 14 hours ago
fc417fc802 | 12 hours ago
MrDOS | 11 hours ago
[0]: https://www.rfc-editor.org/info/rfc5280/#section-4.2.1.10 [1]: https://utcc.utoronto.ca/~cks/space/blog/tech/TLSInternalCAN...
mook | 6 hours ago
nirui | 9 hours ago
But we're talking about a lib of a programming language here, things happened in there effects your own reputation when you ship something built on top of it.
So the real question is, has others such as OpenSSL done the same? If yes, then yeah it's normal, but if no then I don't trust people from Google.
zx8080 | 14 hours ago
It's easy to imagine something like: "give us some grant taxpayers money and we'll add your cert to our trusted list/remove theirs cert/disable cert when run on this hardware/etc."
Jarred | 13 hours ago
computerfriend | 12 hours ago
Aissen | 12 hours ago
direwolf20 | 10 hours ago
aatd86 | 9 hours ago
cowlevel | 9 hours ago
Aissen | 9 hours ago
maerF0x0 | 5 hours ago
xtajv | 10 hours ago
accountrequired | 17 hours ago
TheDong | 16 hours ago
It's just an unfortunate unforced error that the go TLS package is bundled along with the compiler, in the same way Safari being bundled with iOS updates is an unforced error meaning a safari security bug requires a full OS update.
rollulus | 14 hours ago
TheDong | 12 hours ago
agwa | 11 hours ago
xtajv | 10 hours ago
Build another one.
xtajv | 9 hours ago
Pinning certificates in a browser was a brilliant idea for Chrome because web browsers are not made for technical users.
But pinning certificates in a compiler? It's reminding me of that one time where the NSA handed out DES constants to use and said "just trust me, bro". I don't even care that the DES constants ultimately ended up being protections against differential-cryptanalysis SCA (which, btw, the NSA then undermined by arguing for shorter keylengths in "export-grade crypto").
Get off my lawn.
cowlevel | 9 hours ago
agwa | 15 hours ago
What's alarming and unprecedented is that yesterday, two Certificate Transparency log implementations were updated to block the logging of certificates issued by the CAs. This likely indicates total private key compromise, and the blocking by CT logs is to prevent the logs from being spammed by forged certificates. However, it also means there will be no place to record forged certificates found in the wild. (These certificates won't validate in CT-enforcing clients since they won't be in CT logs, but could be used to attack non-CT-enforcing clients.)
FiloSottile | 13 hours ago
rollulus | 13 hours ago
amluto | 13 hours ago
The bugzilla issues mentioned elsewhere suggest at least a mostly orderly removal of trust with issuance stopped several months ago. Your new cert was issued by “CN=EM DV TLS CA - G3B,O=eMudhra BV,C=NL”, which presumably chains to the G3 root, and the bugzilla issue seems to say that G3 was going to be dropped from the root lists some time ago but that maybe it would be cross-signed by something still trusted. So I assume that something else is going on.
Was that cert issued via normal means (e.g. ACME)?
noja | 11 hours ago
agwa | 11 hours ago
Of course, the notion of "trusted" is complicated. These CAs are no longer trusted by up-to-date clients, but out-of-date clients may continue to trust them. The needs of up-to-date clients take priority over the needs of out-of-date clients.
csense | 3 hours ago
Normally Eve's attempts fail because Alice requires a message to be signed with the private key matching the public key in a certificate for bob.example.com signed by a legitimate CA.
Unfortunately Bob bought his CA from Mal's Domains, and Mal is working with Eve for some reason (for example, Mal has been bribed, hacked, intimidated, legally pressured, etc.)
So in addition to Bob's regular on-the-books cert, Mal issues a second off-the-books cert for bob.example.com and gives it to Eve.
With CT, Alice's browser doesn't just require the cert's pubkey, it requires the server to provide a Merkle proof it was included in a CT log.
The CT log puts Eve and Mal on the horns of a dilemma: If they keep their new cert out of the CT logs, Alice's browser won't accept it. If they put it in the CT logs, Bob can read the log himself (it's public) and Bob can say "Hold it! I didn't order this!"
Bob will switch away from Mal as a TLS provider and complain "Hey everybody, Mal issued this cert I didn't order, customers should stop using him and the people who create CA lists for browsers and OS's should closely investigate whether Mal's CA activities are legitimate." Mal can't deny issuing the questionable certificate, the evidence is in public append-only CT logs for everyone to see.
tialaramex | 9 hours ago
agwa | 9 hours ago
chrisjj | 9 hours ago
This is indeed huge. Allegedly forged certs aren't spam and do need to be logged. Refusal to log them is a dereliction of duty and will righly undermine trust in the system.
agwa | 8 hours ago
amluto | 5 hours ago
chrisjj | 4 hours ago
Secretly excluding some issued with a key leak that "appears to be what happened" is not transparency.
CT aren't supposed to be gatekeepers.
If the problem is spam then the solution should be antispam - regardless of allegations of leak.
amluto | 4 hours ago
Once the private key has been leaked, this is basically all pointless.
Can a site owner trust that a certificate in their name has been logged? Of course not! An attacker could be signing it on the fly.
Can an auditor read the complete record? The complete record is kind of meaningless if the private key is leaked. But the auditor already knows that the private key is leaked!
Will the CA get caught if they issue a cert that isn't in the log? Yes! In fact, all newly signed certs won't be in the log.
Will browsers that check for log inclusion be protected? Yes, because newly signed certs won't be in the log.
Will anyone who sees a signed cert that isn't in the log hold evidence of misbehavior? Yes!
One might argue that the actual change to the CT system that should be made is a way for a CT operator to insert a statement into the append-only log that they believe a signing key to be compromised and that they will therefore no longer accept certificates signed by that key.
direwolf20 | 10 hours ago
q3k | 9 hours ago
Yeah okay this is funny. Nice little CTF challenge :).
q3k | 5 hours ago
D=156 in my case. Yeah, that's the entire private key.
mschuster91 | 9 hours ago
q3k | 9 hours ago
mschuster91 | 6 hours ago
Nonetheless, there shouldn't be a mechanism to override or modify the OS trust list embedded in the runtime. If the application code itself wants to do that (e.g. cert pinning), fine. But as an ops person, I repeat myself, it's an utter headache to deal with SSL trust roots already, and really I don't want to have to learn about N different ways that applications manage this crap.
agwa | 9 hours ago
Node: https://github.com/nodejs/node/pull/66871
rustls/webpki: https://github.com/rustls/webpki/pull/546
Electron: https://github.com/electron/electron/pull/55126
This is very serious and you should definitely upgrade. Hopefully OpenSSL will have a patch soon as well.
vladde | 9 hours ago
> These are the eMudhra/emSign intermediates whose private keys were reported compromised in October 2026.
arvidtheking | 8 hours ago
account42 | 7 hours ago
We already have a functioning system for delegating domain ownership in a hierarchical fashion where compromises are limited to only the subtree under the compromised entity. Let's start using it for domain validation sooner rather than later.
Hizonner | 18 minutes ago
Of course we should have done that ages ago, but, you see, 20 years ago or so, Verisign (and maybe others) had a profit stream to protect and people to send to the IETF. And it's just possible there may have been other players with vested interests in things not getting too secure.