SRE/Sysadmin/DevOps/Whatever here, while blog didn't talk about doing anything difficult but setting ENVVAR standards, I will point out all replacements are just as frustrating especially when talking about secrets.
Anything involving vaults where Application reaches out to specific secret vault like Hashicorp Vault/OpenBao/Secrets Manager quickly becomes massive vendor lock in where replacement is very difficult due to library replacement and makes vault uptime extremely important. This puts Ops in extremely difficult place when it becomes time to upgrade or do maintenance.
Config files have problem of you have secrets, how do you get them into config file since config files are generally kept in public systems? Most of the time it's some form of either "Template replacement by privileged system before handing it to application" or "Entire Config File gets loaded into secret vault and passed into application". Templating can be error prone and loading entire config files into secret manager is frustrating as well since someone could screw up the load.
Speaking of config files, since most systems are running containers, and unless you are at Ops discipline company, these config files are never in the right place, it becomes error prone for Ops to screw up the mounting. Also, whatever format you use, JSON/YAML/TOML is ripe for some weird config file bug to emerge like Norway problem in YAML.
Getting secrets from Kubernetes Secrets API I've seen done but lock in again.
I will say I've seen Subprocess thing bite people but I've seen less and less subprocess generation these days. Most teams go with message bus type system instead of sub processing since it's more robust and allows independent scaling.
Interesting read. Another interesting fact is that `setenv()` is fundamentally broken on POSIX, and should essentially never be called in library code. In application code, it should be called only in the absence of any alternative, and certainly before any threads have started.
The reason is that `getenv()` hands out raw pointers to the variables, so overwriting a variable using `setenv()` is impossible to guard against. Treat with extreme caution.
I think it's worth specifically calling out that the right way to set environment variables is in execve() only, as communication across an exec() is the precise niche for which they are the right tool.
IIRC the Solaris implementation of getenv()/setenv() leaks memory, because they want it to still be POSIX-compatible and the POSIX API has a "obey the spec, don't leak memory, be thread-safe -- pick 2" thing going on and you can't actually have a bug-free solution that keeps compatibility with decades of Unix applications.
Arguably that's less harmful than the thread-safety issues on Linux, but there's no perfect solution here as long as POSIX stays what it is.
Environment variables are often used to pass secrets around. But, despite its ubiquity, I believe that's a bad practice:
- On Linux systems, any user process can inspect any other process of that same user for it's environment variables. We can argue about threat model but, especially for a developer's system, there are A LOT of processes running as the same user as the developer.
- IMO, this has become an even more pronounced problem with the popularity of running non-containerized LLM agents in the same user space as the developer's primary OS user. It's a secret exfiltration exploiter's dream.
- Environment variables are usually passed down to other spawned processes instead of being contained to the primary process which is often the only one that needs it.
- Systemd exposes unit environment variables to all system clients through DBUS and warns against using environment variables for secrets[1]. I believe this means non-root users have access to environment variables set for root-only units/services. I could be wrong, I haven't tested it yet. But if this is the case, I bet that's a HUGE surprise to many system admins.
I think ephemeral file sharing between a secret managing process (e.g. 1Password's op cli tool) and the process that needs the secrets (flask, terraform, etc.) is likely the only solution that keeps secrets out of files and also out of environment variables. This is how Systemd's credentials system works. But it's far from widely supported.
Any good solutions for passing secrets around that don't involve environment variables or regular plain text files?
Edit: I think 1Password's op client has a good start in that each new "session" requires my authorization. So I can enable that tool for a cli sessions where I need some secrets but a rogue process that just tries to use the `op` binary isn't going to get to piggyback on that authorization. I'd get a new popup. But this is only step 1. Step 2 is...how to share that secret with the process that needs it and we are now back to the discussion above.
> Environment variables are often used to pass secrets around. But, despite its ubiquity, I believe that's a bad practice:
I think environment variables are recommended to pass configuration parameters, and also secrets, in containerized applications managed by container orchestration systems.
By design, other processes cannot inspect what environment variables are running in a container.
Also, environment variables are passed to child processes because, by design, the goal is to run child processes in the same environment (i.e., same values) as the parent process, or with minor tweaks. Also, the process that spawns child processes is the one responsible for set it's environment variables, which means it already has at least read access to those secrets.
All in all I think all your concerns are based on specious reasoning, but I'll gladly discuss them in finer detail.
I haven't seen much language support for it, though. On one part maybe because it's Linux only.
People that write in Rust (and maybe Go, depends how easy FFI is) should give it a try.
I wanted for a time to get some support for it in PHP, since wrapping a C function should be easy, but the thought of having to also modify php-fpm put a dent in that. I can't and don't want to hack on C code.
In practice it'd be great if the process manager spawn children after opening a secret file descriptor, and pass those on. Not in visible memory, not in /proc/*/environ
Since at least 2012, environment variables have been at least as secure as ordinary memory:
commit b409e578d9a4ec95913e06d8fea2a33f1754ea69
Author: Cong Wang <xiyou.wangcong@gmail.com>
Date: Thu May 31 16:26:17 2012 -0700
proc: clean up /proc/<pid>/environ handling
You can't read another process's environment unless you can ptrace-read the process, and if you can ptrace-read the process you know all its secrets anyway.
I've never tried to use ptrace-read to read secrets from a running process's memory so I can't comment on that part.
I had always assumed getting secrets from a running process' memory was non-trivial and/or required root permissions but maybe that's a bad assumption.
However, reading a process' environment is as trivial as:
`cat /proc/123456/environ`
as long as it's ran by the same user. No ptrace-read required.
Point to a file?
E.g. `CONFIG_PATH=/etc/myapp/config.ini /opt/myapp`
That being said, I still use env vars and don't plan on stopping. I just haven't (yet?) seen any exploits or threat models relating to it that would keep me up at night.
> On Linux systems, any user process can inspect any other process of that same user for it's environment variables. We can argue about threat model but, especially for a developer's system, there are A LOT of processes running as the same user as the developer.
This is a very good point I'd never realised! I'm not sure how you get around it, though, as if that program can even find a credential and decrypt a file, if it runs as the user then everything else can go find that credential as well.
I added a note to my original post about how 1Password's cli tool work. Access to the binary doesn't mean automatic access to any authorized session. I think that's a good start. In my case, I have it tied into my fingerprint reader, so if something want's access to a secret using `op` I get a prompt to authorize. If I haven't just called `op` myself, I know it's a suspect request.
Linux security model is pretty broken without namespaces. systemd has some bells and whistles that help but if you want something better than environment variables you're naturally going to reach for cgroups.
cgroups aren't relevant here, I think. Not sure if that was just a typo, since you did mention namespaces in the first sentence. PID and user namespaces in particular are relevant here.
--passphrase-fd was always the correct thing to do (gpg has had it for ages), but support for that is very meager. I mean, you can’t even kubectl login using environment variables, you pretty much have to pass tokens through command-line arguments and we’ve known that’s terrible for around forty-five years.
I get anxiety whenever I need to set an environment var on Linux. There are (somewhat distro-specific) ways that work properly, but the usual procedures you find online stops working once you reboot (or close the terminal I think?). They should add a simple env var GUI like Windows has that just works, and isn't terminal-specific. Windows has the annoyance of needing to restart the the terminal (or open a new one) for changes to take effect, but works well other than that.
> but the usual procedures you find online stops working once you reboot (or close the terminal I think?)
The environment isn't persistent between sessions. That means you need to make the change in a way that runs on every new session (login or new terminal window).
Depending on how your system is configured:
.bash_profile gets run once on every login
.bashrc gets run once on every non-login new session (i.e. a new terminal window)
It's typical, if using these files, to do something like this:
if [ -f ~/.bashrc ]; then
source ~/.bashrc
fi
in the .bash_profile file, putting most other things in .bashrc, such that you don't have to worry about the distinction.
If you're not even using bash or bash-likes at all, but instead something like Zsh, fish, etc you'll need to set things the way they want to be set there.
> They should add a simple env var GUI like Windows has that just works, and isn't terminal-specific
This doesn't exist in linux, because there isn't "one place" that all possible terminals draw from. Conceivably, it's possible to write a GUI tool that reads your .bashrc file, looks for anything that resembles a variable, parses the bash code to look for ways it is set, and then present a GUI tool that could update it in common cases, but... it's way easier to just write the change in a text editor.
Funny, as a primarily linux user the windows behaviour irks me to no end, it's the cause of so many recurrent problems on end-user machines with so many apps that pollute environment.. and then you wonder why something doesn't work and then it turns out that for some reason you're using $SOFTWARE from c:\Perl64\bin instead of its proper place
Eh, they are just more command line arguments, ones that go on the left side of the command instead of the right. I guess an alternative is something the Windows registry, but I'm not seeing that as a great improvement since it's less direct.
We've tried the other extreme: IBM mainframe OS only had environment variables (JCL DD statements).
...and people have to find special ways to set them. I didn't even know that pam_env existed until a recent security vulnerability announcement. It's never come to my attention before, I don't think I've ever seen it utilized. I've now made a runbook item to disable it. It's a shame that "configuration" includes undoing the fetishes and helpfulness [0][1] of others.
pam-config -d --env
[0] Crowded elevator atrium. Multiple elevators running. Elevator wants to close, another one is coming (oh! I heard it "ding"!). Somebody is holding the elevator which wants to depart and trying to wave me in. Why doesn't somebody push them out?
[1] I'm at a stop sign. Some complete idiot is trying to turn left onto the street I'm leaving and waving me to turn left in front of them. Fuck no! I turn in front of you, somebody rearends you, you fly forward into me: my fault! You should be able to make this turn, if you can't go around the block! [2]
[2] I could go out of my way and turn right. Or I can just wait and see what happens.
> POSIX-specified utilities use uppercase envvars, but that’s not prescriptive for your programs. Quite the contrary: you’re encouraged to use lowercase for your envvars so they don’t collide with the standard tools.
> But in reality, not many applications use lowercase. The proper etiquette in software development is to use ALL_UPPERCASE
I always prefer lower case for env variables in scripts. Thanks for pointing out that it can help reduce clashes with standard tools.
Change your browser settings. You can tell the browser what font you prefer to use, and set minimum font sizes. You can also turn off remote fonts, so that pages can only use fonts you already have. Then you can just uninstall any fonts you don’t like or that you cannot read easily.
We're almost rid of dotfiles in $HOME. Can we please also get rid of the abuse of environment variables for configuration? There is no reason your program needs to muck up the environment table of every process by defining MY_PROGRAM_API_KEY instead of taking a command line argument or reading a configuration file. It's not "secure" just because it's not on the command line. And it will mess up for users because it's nigh impossible to ensure you have the same environment variables over all login types. The variable might be there when you run locally, but not in an ssh session or cron. Some ephemeral configuration such as LD_LIBRARY_PATH and PWD is difficult to handle without environment variables, but those are rare exceptions and not the norm.
Contribute to a clean environment! Don't use environment variables for configuration! </rant>
SRE/Sysadmin/DevOps/Whatever here, while blog didn't talk about doing anything difficult but setting ENVVAR standards, I will point out all replacements are just as frustrating especially when talking about secrets.
Anything involving vaults where Application reaches out to specific secret vault like Hashicorp Vault/OpenBao/Secrets Manager quickly becomes massive vendor lock in where replacement is very difficult due to library replacement and makes vault uptime extremely important. This puts Ops in extremely difficult place when it becomes time to upgrade or do maintenance.
Config files have problem of you have secrets, how do you get them into config file since config files are generally kept in public systems? Most of the time it's some form of either "Template replacement by privileged system before handing it to application" or "Entire Config File gets loaded into secret vault and passed into application". Templating can be error prone and loading entire config files into secret manager is frustrating as well since someone could screw up the load.
Speaking of config files, since most systems are running containers, and unless you are at Ops discipline company, these config files are never in the right place, it becomes error prone for Ops to screw up the mounting. Also, whatever format you use, JSON/YAML/TOML is ripe for some weird config file bug to emerge like Norway problem in YAML.
Getting secrets from Kubernetes Secrets API I've seen done but lock in again.
I will say I've seen Subprocess thing bite people but I've seen less and less subprocess generation these days. Most teams go with message bus type system instead of sub processing since it's more robust and allows independent scaling.
Interesting read. Another interesting fact is that `setenv()` is fundamentally broken on POSIX, and should essentially never be called in library code. In application code, it should be called only in the absence of any alternative, and certainly before any threads have started.
The reason is that `getenv()` hands out raw pointers to the variables, so overwriting a variable using `setenv()` is impossible to guard against. Treat with extreme caution.
I think it's worth specifically calling out that the right way to set environment variables is in execve() only, as communication across an exec() is the precise niche for which they are the right tool.
AFAIK Solaris solved that problem but Linux refuses to copy their solution.
Their solution was that setenv leaks memory rather than overwriting in place.
FreeBSD does the same. See here for discussion: https://freebsd-current.freebsd.narkive.com/NwqZQDWm/fix-for...
IIRC the Solaris implementation of getenv()/setenv() leaks memory, because they want it to still be POSIX-compatible and the POSIX API has a "obey the spec, don't leak memory, be thread-safe -- pick 2" thing going on and you can't actually have a bug-free solution that keeps compatibility with decades of Unix applications.
Arguably that's less harmful than the thread-safety issues on Linux, but there's no perfect solution here as long as POSIX stays what it is.
Link?
Environment variables are often used to pass secrets around. But, despite its ubiquity, I believe that's a bad practice:
- On Linux systems, any user process can inspect any other process of that same user for it's environment variables. We can argue about threat model but, especially for a developer's system, there are A LOT of processes running as the same user as the developer.
- IMO, this has become an even more pronounced problem with the popularity of running non-containerized LLM agents in the same user space as the developer's primary OS user. It's a secret exfiltration exploiter's dream.
- Environment variables are usually passed down to other spawned processes instead of being contained to the primary process which is often the only one that needs it.
- Systemd exposes unit environment variables to all system clients through DBUS and warns against using environment variables for secrets[1]. I believe this means non-root users have access to environment variables set for root-only units/services. I could be wrong, I haven't tested it yet. But if this is the case, I bet that's a HUGE surprise to many system admins.
I think ephemeral file sharing between a secret managing process (e.g. 1Password's op cli tool) and the process that needs the secrets (flask, terraform, etc.) is likely the only solution that keeps secrets out of files and also out of environment variables. This is how Systemd's credentials system works. But it's far from widely supported.
Any good solutions for passing secrets around that don't involve environment variables or regular plain text files?
Edit: I think 1Password's op client has a good start in that each new "session" requires my authorization. So I can enable that tool for a cli sessions where I need some secrets but a rogue process that just tries to use the `op` binary isn't going to get to piggyback on that authorization. I'd get a new popup. But this is only step 1. Step 2 is...how to share that secret with the process that needs it and we are now back to the discussion above.
1: https://www.freedesktop.org/software/systemd/man/latest/syst...
> Environment variables are often used to pass secrets around. But, despite its ubiquity, I believe that's a bad practice:
I think environment variables are recommended to pass configuration parameters, and also secrets, in containerized applications managed by container orchestration systems.
By design, other processes cannot inspect what environment variables are running in a container.
Also, environment variables are passed to child processes because, by design, the goal is to run child processes in the same environment (i.e., same values) as the parent process, or with minor tweaks. Also, the process that spawns child processes is the one responsible for set it's environment variables, which means it already has at least read access to those secrets.
All in all I think all your concerns are based on specious reasoning, but I'll gladly discuss them in finer detail.
> Any good solutions for passing secrets around that don't involve environment variables or regular plain text files?
memfd_secret comes to mind https://man7.org/linux/man-pages/man2/memfd_secret.2.html
I haven't seen much language support for it, though. On one part maybe because it's Linux only.
People that write in Rust (and maybe Go, depends how easy FFI is) should give it a try.
I wanted for a time to get some support for it in PHP, since wrapping a C function should be easy, but the thought of having to also modify php-fpm put a dent in that. I can't and don't want to hack on C code.
In practice it'd be great if the process manager spawn children after opening a secret file descriptor, and pass those on. Not in visible memory, not in /proc/*/environ
For go, it's available in /x/sys/unix: https://pkg.go.dev/golang.org/x/sys/unix#MemfdSecret
Since at least 2012, environment variables have been at least as secure as ordinary memory:
You can't read another process's environment unless you can ptrace-read the process, and if you can ptrace-read the process you know all its secrets anyway.cmdline is a different story.
I've never tried to use ptrace-read to read secrets from a running process's memory so I can't comment on that part.
I had always assumed getting secrets from a running process' memory was non-trivial and/or required root permissions but maybe that's a bad assumption.
However, reading a process' environment is as trivial as:
`cat /proc/123456/environ`
as long as it's ran by the same user. No ptrace-read required.
Any good cross-platform and easy ways to share secrets without using environment variables?
Point to a file? E.g. `CONFIG_PATH=/etc/myapp/config.ini /opt/myapp`
That being said, I still use env vars and don't plan on stopping. I just haven't (yet?) seen any exploits or threat models relating to it that would keep me up at night.
> On Linux systems, any user process can inspect any other process of that same user for it's environment variables. We can argue about threat model but, especially for a developer's system, there are A LOT of processes running as the same user as the developer.
This is a very good point I'd never realised! I'm not sure how you get around it, though, as if that program can even find a credential and decrypt a file, if it runs as the user then everything else can go find that credential as well.
I added a note to my original post about how 1Password's cli tool work. Access to the binary doesn't mean automatic access to any authorized session. I think that's a good start. In my case, I have it tied into my fingerprint reader, so if something want's access to a secret using `op` I get a prompt to authorize. If I haven't just called `op` myself, I know it's a suspect request.
Not an answer, but I do wish there was a low level primitive and a corresponding high level language construct to pass around secrets.
Something like: my_secret = create_secret(value)
Then ideally it's an opaque value from that point on
Linux security model is pretty broken without namespaces. systemd has some bells and whistles that help but if you want something better than environment variables you're naturally going to reach for cgroups.
Linux's namespaces are not a security mechanism. Please do not use them as one.
I've said exactly the same thing on HN myself. However for this problem what do you suggest? Using namespaces is better than using no namespaces.
cgroups aren't relevant here, I think. Not sure if that was just a typo, since you did mention namespaces in the first sentence. PID and user namespaces in particular are relevant here.
Yeah seems you're right, I thought namespaces came under the cgroups umbrella.
I've used keyring, but ultimately does not solve the issue of every application running under the same user.
https://pypi.org/project/keyring/
--passphrase-fd was always the correct thing to do (gpg has had it for ages), but support for that is very meager. I mean, you can’t even kubectl login using environment variables, you pretty much have to pass tokens through command-line arguments and we’ve known that’s terrible for around forty-five years.
If are like most users you have secrets stored in read-protected files in ~/.ssh. There's nothing wrong with that.
I get anxiety whenever I need to set an environment var on Linux. There are (somewhat distro-specific) ways that work properly, but the usual procedures you find online stops working once you reboot (or close the terminal I think?). They should add a simple env var GUI like Windows has that just works, and isn't terminal-specific. Windows has the annoyance of needing to restart the the terminal (or open a new one) for changes to take effect, but works well other than that.
> but the usual procedures you find online stops working once you reboot (or close the terminal I think?)
The environment isn't persistent between sessions. That means you need to make the change in a way that runs on every new session (login or new terminal window).
Depending on how your system is configured:
.bash_profile gets run once on every login
.bashrc gets run once on every non-login new session (i.e. a new terminal window)
It's typical, if using these files, to do something like this:
in the .bash_profile file, putting most other things in .bashrc, such that you don't have to worry about the distinction.If you're not even using bash or bash-likes at all, but instead something like Zsh, fish, etc you'll need to set things the way they want to be set there.
> They should add a simple env var GUI like Windows has that just works, and isn't terminal-specific
This doesn't exist in linux, because there isn't "one place" that all possible terminals draw from. Conceivably, it's possible to write a GUI tool that reads your .bashrc file, looks for anything that resembles a variable, parses the bash code to look for ways it is set, and then present a GUI tool that could update it in common cases, but... it's way easier to just write the change in a text editor.
Funny, as a primarily linux user the windows behaviour irks me to no end, it's the cause of so many recurrent problems on end-user machines with so many apps that pollute environment.. and then you wonder why something doesn't work and then it turns out that for some reason you're using $SOFTWARE from c:\Perl64\bin instead of its proper place
Eh, they are just more command line arguments, ones that go on the left side of the command instead of the right. I guess an alternative is something the Windows registry, but I'm not seeing that as a great improvement since it's less direct.
We've tried the other extreme: IBM mainframe OS only had environment variables (JCL DD statements).
I gave up on environment variables long ago. Now my compilers read a dmd.conf file that is in the same directory as the compiler executable.
...and people have to find special ways to set them. I didn't even know that pam_env existed until a recent security vulnerability announcement. It's never come to my attention before, I don't think I've ever seen it utilized. I've now made a runbook item to disable it. It's a shame that "configuration" includes undoing the fetishes and helpfulness [0][1] of others.
[0] Crowded elevator atrium. Multiple elevators running. Elevator wants to close, another one is coming (oh! I heard it "ding"!). Somebody is holding the elevator which wants to depart and trying to wave me in. Why doesn't somebody push them out?[1] I'm at a stop sign. Some complete idiot is trying to turn left onto the street I'm leaving and waving me to turn left in front of them. Fuck no! I turn in front of you, somebody rearends you, you fly forward into me: my fault! You should be able to make this turn, if you can't go around the block! [2]
[2] I could go out of my way and turn right. Or I can just wait and see what happens.
> POSIX-specified utilities use uppercase envvars, but that’s not prescriptive for your programs. Quite the contrary: you’re encouraged to use lowercase for your envvars so they don’t collide with the standard tools.
> But in reality, not many applications use lowercase. The proper etiquette in software development is to use ALL_UPPERCASE
I always prefer lower case for env variables in scripts. Thanks for pointing out that it can help reduce clashes with standard tools.
Another legacy mess:
It's absolutely crazy that this isn't a dynamically resizable vector.It is, there’s just a limit set by the kernel for the number of pages the command line as a whole can occupy or something like that.
No, it's still an artificial limit, even if you can change it (usually after your command has failed).
That was a hard to read font.
Huh... Guess it's what you're used to.
First thing that struck me about the site is how beautiful I found it. I even inspected the font: Iosevka Web, apparently.
Change your browser settings. You can tell the browser what font you prefer to use, and set minimum font sizes. You can also turn off remote fonts, so that pages can only use fonts you already have. Then you can just uninstall any fonts you don’t like or that you cannot read easily.
We're almost rid of dotfiles in $HOME. Can we please also get rid of the abuse of environment variables for configuration? There is no reason your program needs to muck up the environment table of every process by defining MY_PROGRAM_API_KEY instead of taking a command line argument or reading a configuration file. It's not "secure" just because it's not on the command line. And it will mess up for users because it's nigh impossible to ensure you have the same environment variables over all login types. The variable might be there when you run locally, but not in an ssh session or cron. Some ephemeral configuration such as LD_LIBRARY_PATH and PWD is difficult to handle without environment variables, but those are rare exceptions and not the norm.
Contribute to a clean environment! Don't use environment variables for configuration! </rant>
This problem is also removed by just not exporting the variables. Also you can just pass an envp to execvE.