The Rust Security Response Team was notified that Miri stores all environment variables to target/, allowing secrets to persist in caches.
Then proceed to blame Github Actions.
NB: All CI platform that provide caching and would naturally cache the build folder that is target/ would have this security hole. The culprit is cargo miri, not Github Actions.
Yes, the root cause here is that it is so easy to write insecure GHA pipelines without noticing.
If you
Put some secrets into the environment
Run some arbitrary programs in that environment
Cache some parts of the output of those programs
And then make that cache publicly available
then you're doing it wrong. To make this secure, everything running in the environment would have to be carefully audited to not preserve the environment, without even knowing which parts of the environment are sensitive. That's a completely unrealistic expectation. In taint tracking terms, when you ran the program with the secret in the environment, that program's execution is now tainted with the secret, and so is all of its output.
Lucky enough, this is rare -- our ecosystem scan found only a handful of cases where people make secrets globally available to all steps of a pipeline. But you get no indication from GHA that this is a terrible idea, and that is a problem.
This applies even for stdout/stderr: if you put a token into a URL, this can easily end up being printed somewhere. CI providers know this, that's why they filter configured secrets and remove them from the output. They are, basically, removing the taint. (In principle that could be used to actually leak the secret, but with the typical size of secret tokens that is not practical.) But for all other output, this is your responsibility, the CI cannot do it for you.
Now, GHA does provide the knobs to do this properly: you can control which parts of which jobs the secrets are actually exposed to. But it's easy to forget to use these knobs, and that's a fundamental problem of GHA.
(I am a Miri maintainer and author of the patch mentioned in the blog post. I view this patch as defense in depth, similar to stack canaries and ASLR: it's always a bug if you're relying on this 2nd layer of defense, but it's still useful to have.)
The point is that it's not specific to GHA. Jenkins, Gitlab CI, CircleCI, etc... All have the same "vulnerability" of "if you leak secrets, secrets are leaked".
That's bad, but it doesn't make it any more feasible to guarantee that the secrets don't end up somewhere in the target folder.
For example, this build script also stores the entire environment somewhere. Build script outputs end up in target and that may get cached. It would be a never-ending cat-and-mouse game to try and find all those cases. Which just goes to show that it's the wrong game you play. If you rely on the entire build process keeping the env vars secret, you already lost.
And "storing the entire environment" is a good idea since when?
If the Rust ecosystem is so badly disciplined that tracking bad behavior is a "cat-and-mouse" game that is already lost, maybe I should avoid Rust then.
If you are trusting the C/Python/NPM/... ecosystem to never store the environment anywhere, I don't think I can help you but I certainly don't think you'll be avoiding such issues. Security works with clear security boundaries, and if you pull the entire build system into that boundary you're holding it wrong.
When given the choice between an approach that structurally prevents leaks, and one that depends on dozens of tools to respect a fuzzy undocumented boundary, if you are picking the latter, you are not picking security. This is not Rust specific, it's just basic security engineering. It's basically the same as the principle of least privilege -- don't give your secrets to processes that don't need them. Your proposal for how to deal with the problem is like telling people "just don't write UB in your C code. Just be careful." We all know how well that works.
But you can of course choose to ignore decades of accumulated experience in how to design secure systems and then blame others if things go wrong. shrug
Your proposal for how to deal with the problem is like telling people "just don't write UB in your C code. Just be careful." We all know how well that works.
Free shot at C :D
And a complete inaccurate take at what I am saying. You are blaming a CI platform for doing its job when a tool you developed is behaving badly (aka: storing sensitive information in a folder that is known to be cached without a second thought).
If you are trusting the C/Python/NPM/... ecosystem to never store the environment anywhere
I don't. That's not what I've been saying. But when their ecosystem behaves badly, they don't blame the CI platform that has been running the bad agent.
Security works with clear security boundaries, and if you pull the entire build system into that boundary you're holding it wrong.
Which is what Miri has been doing, not CI platforms.
But you can of course choose to ignore decades of accumulated experience in how to design secure systems and then blame others if things go wrong. shrug
Again, you are blaming the CI platform for doing its job (running code, and caching stuff). But yes, I'm the one ignoring decades of accumulated experience and blaming others...
EDIT: I personally avoid storing sensitive information in environment variables because, as said by another comment, those are public. It's already a leak. But here, Miri made it worse by storing that information in a place that will obviously be cached by all CI platforms, because it's in their design to do that.
I am blaming a CI platform for not doing its job properly wrt managing the secrets of their users.
Which is what Miri has been doing, not CI platforms.
Miri hasn't pulled anything into anything. CI platforms decided that everything they run is now a security barrier. That's a terrible decision, and I won't let the blame be shifted anywhere else.
It's not the caching that's wrong. It's giving inputs to processes whose cache will be made public. This can be avoided in multiple sensible ways (separate caches, not exposing the secrets before the cache is created, ...). But exposing secrets to the entire CI job is a gross violation if the principle of least privilege and puts unreasonable and insufficiently specified expectations on the build pipeline.
Anyway, we haven't added new information for a while here, we just repeated ourselves. Our positions are clear to each other, I think. I cannot imagine any argument you might bring up that would convince me that your stance is reasonable, and it seems like the same is true the other way around. It's unlikely we're going to make progress by repeating ourselves a few more times.
CI platforms are not secret management softwares, it's not their job to manage secrets. Their job is to run code. They are essentially "Remote Code Execution" platforms. It's in their design. You should not run untrusted code, with sensitive information.
The point I'm making is not "CI is safe, Rust ecosystem is not". CI is obviously not safe, their design isn't intended to be safe, RCE will never be safe. The question I'm asking is why should a build system need a compete copy of all environment variables, especially knowing that it will be run on a sensitive environment such as CI. Blaming the CI platform when doing obviously wrong things is like yelling at the weather because you peed in the wind.
People put secrets into them. Most of them even have an explicit concept of secret variables. That makes them secret management softwares. They are aware of that and do things like filtering secrets from the logs, which is helpful but nowhere near enough.
CI is obviously not safe, their design isn't intended to be safe, RCE will never be safe.
Please don't move goalposts. We're not talking about the RCE part, we're talking about the secret management part. There is more than one form of security.
It's not limited to Rust and it's not limited to intentional behaviour of programs, all processes get the environment, and it can easily leak by accident (program crashes, leaves a core dump).
I was taught in the 1990s by unix greybeards that environment variables must not be used for sensitive data because they are effectively public: many flavours of unix expose them to all processes via ps, and there are many ways for them to leak much further. Hence programs of that era use files or named pipes (which go through the kernel’s access controls) to provide access to secrets: X, ssh, gpg, etc. So it has been weird to see Heroku and so many subsequent designs depending on such an unsafe mechanism for security-critical information.
However, I think /proc data being world-readable is a Linux-y thing which I've often seen as somewhat undesirable (I don't like other users in a shared system seeing which files I'm opening).
IIRC, for a while I was working on OVH VPS that had some kernel patch applied that made a lot of /proc non-world readable. It was confusing, but perhaps it's the right idea?
(No matter what, the current situation is that /proc is world-readable and we should behave accordingly.)
With envvars the problem is also that they are inherited and almost no one cleans them - either correctly by removing all but safe or with hope by removing "unsafe".
The prevalent idiom is to expose envvars to some kind of build script (probably shell) and then everything (sometimes every build step) inherits them in the build process which is silly.
I usually pass through environment variables the path to a file containing the secret, and that path is on a tmpfs (for Docker, it's usually in /secrets or /run/secrets) and injected via a Secret Manager such as Vault.
What I don't get is: as I understood from their description, the way to exfiltrate the secret from the cache is by force-pushing code that reads them while it runs in CI (and then prints to logs, sends over the network, or something).
But doesn't every CI run have those env variables set anyway, cached or not? (That's how miri picked them up to write them to the cache, after all?) So, can't the force-pushed code just read the env vars directly? As I understand, the only additional thing they get by reading the miri cache is env values from previous runs? Fair enough, this is an information leak (from an older to a newer CI run) that should be fixed; but usually the current env vars are most likely to be valuable. Thus almost every CI setup that cares about this leak (i.e. has sensitive variables) will continue to expose them?
I think the idea is that you, say, publish a release, and that workflow includes running miri, then someone can create a pr and get the same cache as your release workflow.
Aha, makes sense (I wasn't aware different workflows can share the same cache, interesting?).
PS. if that's true, then can't it also happen that the cache is filled in the PR workflow and executed in the release workflow (thus again giving the attacker access to the release secrets by way of positing hijacked binaries in the cache)?
Don’t store your secrets in environment variables. Don’t say you weren’t warned.
More helpfully: our cross-platform multi-language, open SecureStore protocol, cli, and libraries makes it easy to avoid doing that, storing secrets encrypted in git: https://github.com/neosmart/securestore-rs
Available for all major operating systems, with libs for rust, .NET, go, php, python, js/ts, and more. Contributions and ports to more languages/libs are welcome!
linkdd | 2 days ago
Then proceed to blame Github Actions.
NB: All CI platform that provide caching and would naturally cache the build folder that is
target/would have this security hole. The culprit iscargo miri, not Github Actions.ralfj | a day ago
Yes, the root cause here is that it is so easy to write insecure GHA pipelines without noticing.
If you
then you're doing it wrong. To make this secure, everything running in the environment would have to be carefully audited to not preserve the environment, without even knowing which parts of the environment are sensitive. That's a completely unrealistic expectation. In taint tracking terms, when you ran the program with the secret in the environment, that program's execution is now tainted with the secret, and so is all of its output.
Lucky enough, this is rare -- our ecosystem scan found only a handful of cases where people make secrets globally available to all steps of a pipeline. But you get no indication from GHA that this is a terrible idea, and that is a problem.
This applies even for stdout/stderr: if you put a token into a URL, this can easily end up being printed somewhere. CI providers know this, that's why they filter configured secrets and remove them from the output. They are, basically, removing the taint. (In principle that could be used to actually leak the secret, but with the typical size of secret tokens that is not practical.) But for all other output, this is your responsibility, the CI cannot do it for you.
Now, GHA does provide the knobs to do this properly: you can control which parts of which jobs the secrets are actually exposed to. But it's easy to forget to use these knobs, and that's a fundamental problem of GHA.
(I am a Miri maintainer and author of the patch mentioned in the blog post. I view this patch as defense in depth, similar to stack canaries and ASLR: it's always a bug if you're relying on this 2nd layer of defense, but it's still useful to have.)
linkdd | a day ago
The point is that it's not specific to GHA. Jenkins, Gitlab CI, CircleCI, etc... All have the same "vulnerability" of "if you leak secrets, secrets are leaked".
ralfj | a day ago
That's bad, but it doesn't make it any more feasible to guarantee that the secrets don't end up somewhere in the target folder.
For example, this build script also stores the entire environment somewhere. Build script outputs end up in
targetand that may get cached. It would be a never-ending cat-and-mouse game to try and find all those cases. Which just goes to show that it's the wrong game you play. If you rely on the entire build process keeping the env vars secret, you already lost.linkdd | a day ago
And "storing the entire environment" is a good idea since when?
If the Rust ecosystem is so badly disciplined that tracking bad behavior is a "cat-and-mouse" game that is already lost, maybe I should avoid Rust then.
ralfj | 18 hours ago
If you are trusting the C/Python/NPM/... ecosystem to never store the environment anywhere, I don't think I can help you but I certainly don't think you'll be avoiding such issues. Security works with clear security boundaries, and if you pull the entire build system into that boundary you're holding it wrong.
When given the choice between an approach that structurally prevents leaks, and one that depends on dozens of tools to respect a fuzzy undocumented boundary, if you are picking the latter, you are not picking security. This is not Rust specific, it's just basic security engineering. It's basically the same as the principle of least privilege -- don't give your secrets to processes that don't need them. Your proposal for how to deal with the problem is like telling people "just don't write UB in your C code. Just be careful." We all know how well that works.
But you can of course choose to ignore decades of accumulated experience in how to design secure systems and then blame others if things go wrong. shrug
linkdd | 18 hours ago
Free shot at C :D
And a complete inaccurate take at what I am saying. You are blaming a CI platform for doing its job when a tool you developed is behaving badly (aka: storing sensitive information in a folder that is known to be cached without a second thought).
I don't. That's not what I've been saying. But when their ecosystem behaves badly, they don't blame the CI platform that has been running the bad agent.
Which is what Miri has been doing, not CI platforms.
Again, you are blaming the CI platform for doing its job (running code, and caching stuff). But yes, I'm the one ignoring decades of accumulated experience and blaming others...
EDIT: I personally avoid storing sensitive information in environment variables because, as said by another comment, those are public. It's already a leak. But here, Miri made it worse by storing that information in a place that will obviously be cached by all CI platforms, because it's in their design to do that.
ralfj | 16 hours ago
I am blaming a CI platform for not doing its job properly wrt managing the secrets of their users.
Miri hasn't pulled anything into anything. CI platforms decided that everything they run is now a security barrier. That's a terrible decision, and I won't let the blame be shifted anywhere else.
It's not the caching that's wrong. It's giving inputs to processes whose cache will be made public. This can be avoided in multiple sensible ways (separate caches, not exposing the secrets before the cache is created, ...). But exposing secrets to the entire CI job is a gross violation if the principle of least privilege and puts unreasonable and insufficiently specified expectations on the build pipeline.
Anyway, we haven't added new information for a while here, we just repeated ourselves. Our positions are clear to each other, I think. I cannot imagine any argument you might bring up that would convince me that your stance is reasonable, and it seems like the same is true the other way around. It's unlikely we're going to make progress by repeating ourselves a few more times.
linkdd | 16 hours ago
CI platforms are not secret management softwares, it's not their job to manage secrets. Their job is to run code. They are essentially "Remote Code Execution" platforms. It's in their design. You should not run untrusted code, with sensitive information.
The point I'm making is not "CI is safe, Rust ecosystem is not". CI is obviously not safe, their design isn't intended to be safe, RCE will never be safe. The question I'm asking is why should a build system need a compete copy of all environment variables, especially knowing that it will be run on a sensitive environment such as CI. Blaming the CI platform when doing obviously wrong things is like yelling at the weather because you peed in the wind.
ralfj | 15 hours ago
People put secrets into them. Most of them even have an explicit concept of secret variables. That makes them secret management softwares. They are aware of that and do things like filtering secrets from the logs, which is helpful but nowhere near enough.
Please don't move goalposts. We're not talking about the RCE part, we're talking about the secret management part. There is more than one form of security.
kryptiskt | 16 hours ago
It's not limited to Rust and it's not limited to intentional behaviour of programs, all processes get the environment, and it can easily leak by accident (program crashes, leaves a core dump).
fanf | a day ago
I was taught in the 1990s by unix greybeards that environment variables must not be used for sensitive data because they are effectively public: many flavours of unix expose them to all processes via
ps, and there are many ways for them to leak much further. Hence programs of that era use files or named pipes (which go through the kernel’s access controls) to provide access to secrets: X, ssh, gpg, etc. So it has been weird to see Heroku and so many subsequent designs depending on such an unsafe mechanism for security-critical information.koala | a day ago
However, I think
/procdata being world-readable is a Linux-y thing which I've often seen as somewhat undesirable (I don't like other users in a shared system seeing which files I'm opening).IIRC, for a while I was working on OVH VPS that had some kernel patch applied that made a lot of
/procnon-world readable. It was confusing, but perhaps it's the right idea?(No matter what, the current situation is that
/procis world-readable and we should behave accordingly.)hawski | a day ago
With envvars the problem is also that they are inherited and almost no one cleans them - either correctly by removing all but safe or with hope by removing "unsafe".
The prevalent idiom is to expose envvars to some kind of build script (probably shell) and then everything (sometimes every build step) inherits them in the build process which is silly.
koala | a day ago
Yeah, I have to admit that sometimes I build an empty environment for subprocesses in Python, but it's kinda tedious and I don't always do it.
I've played a bit with
sops decryptand using the Bitwarden CLI for stuff, but not so much either.linkdd | a day ago
I usually pass through environment variables the path to a file containing the secret, and that path is on a tmpfs (for Docker, it's usually in
/secretsor/run/secrets) and injected via a Secret Manager such as Vault.pflanze | a day ago
What I don't get is: as I understood from their description, the way to exfiltrate the secret from the cache is by force-pushing code that reads them while it runs in CI (and then prints to logs, sends over the network, or something).
But doesn't every CI run have those env variables set anyway, cached or not? (That's how miri picked them up to write them to the cache, after all?) So, can't the force-pushed code just read the env vars directly? As I understand, the only additional thing they get by reading the miri cache is env values from previous runs? Fair enough, this is an information leak (from an older to a newer CI run) that should be fixed; but usually the current env vars are most likely to be valuable. Thus almost every CI setup that cares about this leak (i.e. has sensitive variables) will continue to expose them?
hjvt | a day ago
I think the idea is that you, say, publish a release, and that workflow includes running miri, then someone can create a pr and get the same cache as your release workflow.
pflanze | a day ago
Aha, makes sense (I wasn't aware different workflows can share the same cache, interesting?).
PS. if that's true, then can't it also happen that the cache is filled in the PR workflow and executed in the release workflow (thus again giving the attacker access to the release secrets by way of positing hijacked binaries in the cache)?
[OP] peter | a day ago
From the post:
mqudsi | a day ago
Don’t store your secrets in environment variables. Don’t say you weren’t warned.
More helpfully: our cross-platform multi-language, open SecureStore protocol, cli, and libraries makes it easy to avoid doing that, storing secrets encrypted in git: https://github.com/neosmart/securestore-rs
Available for all major operating systems, with libs for rust, .NET, go, php, python, js/ts, and more. Contributions and ports to more languages/libs are welcome!