Docker v2 Beta can't handle env file without "=" after variable name

[potiuk]

Description

When you specify --env-file filename.env for the new docker-compose and the env file contains just variable names without "=" afterwards, the docker-compose command fails with "Can't separate key from value"

Steps to reproduce the issue:

1. Create _docker.env file with this content in a directory where you have a docker-compose file:

VARIABLE

2. Run docker-compose up --env-file ${PATH}/_docker.env

Describe the results you received:

The command fails with Can't separate key from value error

Describe the results you expected:

• It should not fail
• If VARIABLE is not defined in host, it should not be present in the container
• If VARIABLE is defined in host, it should be set in the container to the same value

Additional information you deem important (e.g. issue happens only occasionally):

Seems that adding = after the variable fixes the problem:

VARIABLE=

This issue was first discovered in apache/airflow#16949 and workarounded in apache/airflow#16950 by @oyarushe for Apache Airflow.

**Output of docker-compose --version: ***

v2.0.0-beta.6

Output of docker version:

Docker Desktop: 3.5.2 on MacOS

[potiuk]

This workaround with adding = also has proven to break regular docker --env-file command. Previously, the same env file without = could be used for both docker-compose an docker but when you use the same pattern for docker env-file those variables get EMPTY:

VARIABLE=

[potiuk]

Because of this docker-compose and docker different behaviour this is a major issue.

[ndeloof]

need to check how https://pypi.org/project/python-dotenv/ and https://github.com/joho/godotenv differ for this specific syntax, and propose a fix

[ndeloof]

python-dotenv explicitely consider lack of an equal sign as "value: none"
https://github.com/theskumar/python-dotenv/blob/master/src/dotenv/parser.py#L156-L160

ruby's dotenv parser don't declare the equal sign as optional
https://github.com/bkeepers/dotenv/blob/c237d6d6291c898d8affb290b510c7aac49aed71/lib/dotenv/parser.rb#L19

Based on this I'm not sure godotenv would accept a PR to ignore a missing separator?

I also need to tell I don't really understand how usefull name without a value is in a .env file, as it would then be resolved from os.Env IIUC but compose variables are already resolved by os.Env so this has no actual effect. If anyone can give me some more detail on the way they rely on this...

[potiuk]

This is all about passing variables to INSIDE the container, not compose variables.

[potiuk]

We heavily rely on it in Apache Airflow: https://github.com/apache/airflow/blob/main/scripts/ci/docker-compose/_docker.env
We run a "development" tool called breeze which prepares set of environment variables based on the input and then passes all those variables to inside of the Airflow container.

We alse have some scenarios (mostly CI-based) when we use the same airflow image and run a specific command - but we do not need docker compose for that as we only need to spin that single image.

In the first case our docker-compose has this:
https://github.com/apache/airflow/blob/1960e37952efc08ad33328ff511a846f2376ad4f/scripts/ci/docker-compose/base.yml#L28

In the second case we pass the same env file to a docker run command via "extra args":
https://github.com/apache/airflow/blob/1960e37952efc08ad33328ff511a846f2376ad4f/scripts/ci/libraries/_initialization.sh#L274

All those are based on values already set by previous steps in the script (based on the user flags) or environment variables set in CI

For example here is how they are set from user input:

https://github.com/apache/airflow/blob/1960e37952efc08ad33328ff511a846f2376ad4f/breeze#L792

And here we set a number of those env vars in CI: https://github.com/apache/airflow/blob/1960e37952efc08ad33328ff511a846f2376ad4f/.github/workflows/ci.yml#L31

[potiuk]

For now I had to actually warn the users to disable v2 :( until the problem is solved:

https://github.com/apache/airflow/blob/1960e37952efc08ad33328ff511a846f2376ad4f/breeze#L594

Happy to help testing it when it is fixed :)

[ndeloof]

not very pleasant, but a possible short term workaround is to use FOO=${FOO} syntax in your dotEnv file to make it explicit value is inherited from local environment

[potiuk]

Not sure if that would work for docker case, but yeah. Worth checking! thanks for the suggestion.

[potiuk]

Yep. works in both cases! Thanks for the suggestion @ndeloof - I will workaround it this way for now!

UPDATE: I must have been doing something wrong. The workaround does not work with docker's --env-file :(

[potiuk]

Easy fix: sed 's/\(^[^#=]*\)$/\1=${\1}/' -i _docker.env :)

[ndeloof]

fixing joho/godotenv won't be trivial, as we can't just consider FOO (no equal sign) to be equivalent to FOO=${FOO}. with the latter, is FOO env variable is not set, container is set with FOO variable set to an empty string. While it should have FOO variable unset.

[potiuk]

fixing joho/godotenv won't be trivial, as we can't just consider FOO (no equal sign) to be equivalent to FOO=${FOO}. with the latter, is FOO env variable is not set, container is set with FOO variable set to an empty string. While it should have FOO variable unset.

That's correct. That's how it worked so far

[potiuk]

BTW. I am fine with Airflow with using FOO=${FOO} because I deliberately made a choice of not relying on variable being set/unset but on its value. So even binary variables for me have "true"/"false" value and check is != "true" - precisely for the reason.

[potiuk]

Actualy I just double-checked and tried it again and it DOES NOT work in docker :(. So it should be fixed.

The error here https://github.com/apache/airflow/pull/17052/checks?check_run_id=3087380066 was caused by PACKAGE_FORMAT got the "unexpected" value of "${PACKAGE_FORMAT}". Since PACKAGE_FORMAT value is set in that step to "wheel", the only explanation I find is that the the docker used does not resolve those values as env variables (and actually docker documentation does not even mention that it can do it). I just double-checked it and well... yeah:

export PACKAGE_FORMAT=wheel
[jarek:~/code/airflow] ± docker run -it  --env-file scripts/ci/docker-compose/_docker.env apache/airflow:2.1.2  bash
airflow@352e19876bd7:/opt/airflow$ set | grep PACKAGE_FORMAT
PACKAGE_FORMAT='${PACKAGE_FORMAT}'

[potiuk]

Seems that we have very similar issues for Linux v2 just released