How to use .env variables in package.json
I was working on setting up a new project while following a tutorial. This tutorial started to introduce a multi-step docker setup shell script file .sh
. My preference is to break up these types of scripts so that each step can be run individually in the case of a partial setup/failure. This is an approach that I’ve seen and some of my co-workers use as a standard. But is it the best/only option?
The problem
It’s common to need access to your environment variables in multiple places. Your code, your build scripts, and your pipeline may all need access. It’s not always simple to propagate these variables from one end to the other. There’s sometimes a fair amount of exporting/importing and sometimes it can require some level of duplication.
Example setup script with a series of steps and some variables:
#!/bin/bashset -eSERVER=”my_database_server”;
PW=”mysecretpassword”;
DB=”my_database”;echo “echo stop & remove old docker [$SERVER] and starting new fresh instance of [$SERVER]”;(docker kill $SERVER || :) && \
(docker rm $SERVER || :) && \
docker run — name $SERVER -e POSTGRES_PASSWORD=$PW \
-e PGPASSWORD=$PW \
-p 5432:5432 \
-d postgres# wait for pg to startecho “sleep wait for pg-server [$SERVER] to start”;SLEEP 3;# create the dbecho “CREATE DATABASE $DB ENCODING ‘UTF-8’;” | docker exec -i $SERVER psql -U postgresecho “\l” | docker exec -i $SERVER psql -U postgres
Yes, I could do what has been done before. I could break each step out into a separate shell script file .sh
. But most of my projects don’t need that many scripts and the simpler I can make things, the better.
This got me thinking:
- Do I need external script files? Is this the most valuable approach? Does having more files just make it harder to find where things are happening?
- How would it be possible to skip writing these files at all?
- How can we have a single source of secret variables that are used everywhere?
The solution
Dotenv
A well known and widely used tool that makes getting environment variables easily loaded. Dotenv is built into many of the tools/bundlers/frameworks that you may already be using. And it has a series of plugins/extensions that make it even more useful. Like dotenv-safe, dotenv-extend, and dotenv-cli, which make the development experience smoother and more robust.
Cross-env/var
Admittedly, these two are not the same tool or interchangeable. But they are both so useful that they’re worth mentioning.
Cross-env makes setting environment variables work across platforms.
Cross-var makes substituting environment variables work across platforms. We’ll be using this in our example.
With our powers combined…
By using dotenv
and cross-var
together, we’re able to read in whichever .env
files we want, or consume existing environment variables (from cli, bash_profile, CI, etc) and then easily substitute them into our package.json
scripts and it works across development platforms!
Dependencies being used in this example:
"cross-var": "1.1.0",
"dotenv-cli": "3.0.0",
An example .env
file:
DOCKER_NAME="my-project"
POSTGRES_HOST=127.0.0.1
POSTGRES_PORT=5432
POSTGRES_DATABASE="my-db"
POSTGRES_USERNAME="foo"
POSTGRES_PASSWORD="bar"
Recreating the docker scripts in our package.json
:
"docker:kill": "dotenv cross-var docker kill %DOCKER_NAME%","docker:remove": "dotenv cross-var docker rm %DOCKER_NAME%","docker:run": "dotenv -- cross-var docker run --name %DOCKER_NAME% -e POSTGRES_USER=%POSTGRES_USER% -e POSTGRES_PASSWORD=%POSTGRES_PASSWORD% -p 5432:5432 -d postgres","docker:setup": "dotenv -- cross-var \"echo CREATE DATABASE $DB ENCODING ‘UTF-8’; | docker exec -i %DOCKER_NAME% psql -d %POSTGRES_DATABASE% -U %POSTGRES_USER%\"",
For my project, I was able to simplify this further by utilizing more features/flags with docker:
"docker:kill": "dotenv cross-var docker kill %DOCKER_NAME%","docker:run": "dotenv -- cross-var docker run --rm --name %DOCKER_NAME% -e POSTGRES_DB=%POSTGRES_DATABASE% -e POSTGRES_USER=%POSTGRES_USERNAME% -e POSTGRES_PASSWORD=%POSTGRES_PASSWORD% -p 5432:5432 -d postgres",
This is in no way specific to docker. You can use this with any of your scripts if it makes sense for your project.
Break It Down
Combine dotenv
, dotenv-cli
, and cross-var
.
Simple command:
dotenv cross-var foobar %MY_VARIABLE%
Commands with flag(s):
dotenv -- cross-var foobar -F --b %MY_VARIABLE%dotenv -e .env.staging -- cross-var foobar -F --b %MY_VARIABLE%
Complex command:
dotenv -- cross-var \"echo \\l; | foobar -F --b %MY_VARIABLE%\"
You’ll want to take note of three features that become necessary:
- All variables need to be wrapped with
%
, like:%MY_VARIABLE%
. This prevents the shell from substituting the variable before our script runs --
comes fromdotenv-cli
and separates the internal command from thedotenv
call so it knows which flags belong todotenv
and which belong to the internal command\"...\"
comes fromcross-var
and wraps more complicated commands
Notes
Why aren’t we using the $MY_VARIABLE
syntax? That was the initial approach, however, on MacOS, the shell would automatically substitute the variable before our script executed (before .env
was loaded). I then also tried wrapping variables with single quotes (ex: '$MY_VARIABLE'
) to stop substitution, however this would introduce quotes and would occasionally break script functionality. That’s why we opt to use the Windows syntax (%MY_VARIABLE
), since cross-var
does a replace for this format as well, and the auto-substitution doesn’t appear to occur on Windows.
Testing
This has been tested on Windows 10 with Git Bash, Command Prompt, and PowerShell and on MacOSX with Terminal and Z Shell. But there are a lot of platforms and shells available. If you find an issue, please leave a comment!
Thoughts/Next Steps
There will always be a use case for larger build/CI/CD systems where this solution is not a good fit.
In the age of extreme simplicity when it comes to build systems, like Parcel, it’s nice to have an approach to handling secrets that feels consistently easy. When you can keep things simple, why not?
Is this a common enough of a scenario to add this feature to dotenv-cli
? Or should a new tool (dotenv-var
?) be developed to simplify the combo into a single command?
Produced at GenUI
This approach was conceived at GenUI, a software development firm in Seattle, WA, USA. Interested in knowing more, starting a new project or working with us? Come check us out at https://www.genui.co/