Environment variables defined in tasks.json::tasks.options.env do not overwrite existing environment variables

[zfields]

Code: 1.22.2
macOS: 10.13.3

When I create a task, I want to provide a very specific environment (which is why I bother filling out the JSON values). However, if an environment variable is pre-existing, it will not be overwritten in the execution environment of the task.

I am extremely surprised by this behavior. I discovered this by not being able to find a library I was building with a task when specifying DYLD_LIBRARY_PATH.

A very simple test is to update ~/.bash_profile to include

export FOO=bar

Then attempt to override it with tasks.json

{
    // See https://go.microsoft.com/fwlink/?LinkId=733558
    // for the documentation about the tasks.json format
    "version": "2.0.0",
    "tasks": [
        {
            "label": "diagnostic",
            "type": "shell",
            "options": {
                "cwd": "${workspaceFolder}/beam",
                "env": {
                    "FOO": "baz",
                }
            },
            "command": "printenv",
            "problemMatcher": []
        }
    ]
}

Output:

...
FOO=bar
...

A "task" appears to execute . ~/.bash_profile to set it's environment. Since it is already creating a new environment for each execution, it would be nice if I could override the values as I see fit, by specifying the values in tasks.options.env.

[dbaeumer]

Since this is a shell task the bash decides to source ~/.bash_profile. This is how shells work and there is not much to my knowledge that VS Code can do to avoid this.

The shell started has the -c option so the shell source /.bashrc which normally has

# If not running interactively, don't do anything
case $- in
    *i*) ;;
      *) return;;
esac

at its being. So every instructions you have after that are not executed if the -i option is not present. So I recommend that you move the environment variables into ./bashrc.

OK to close

[zfields]

NOT ok to close.

How is VSCode setting the environment variables I request in tasks.options.env?

[dbaeumer]

It is setting the environment for the launched shell process. Tasks don't start login shells so the ./bash_profile shouldn't be executed.

[zfields]

Do you have a Mac to reproduce this with? Use diagnostic task I pasted above. It will reflect any variables specified in ~/.bash_profile.

To reproduce:

0. Use a Mac?
1. Set an environment variable in ~/.bash_profile
2. Modify the environment variable by using the shell in VSCode (optional)
3. Add it with an alternate value to the diagnostic task.
4. Execute the diagnostic task (you will observe the value of the environment variable you set in ~/.bash_profile.)
5. Change the value in ~/.bash_profile, and watch it propagate to value printed in the diagnostic task.

The value from ~/.bash_profile will display when you run the diagnostic task (regardless of any value set on the task itself).

[dbaeumer]

@zfields I understand the problem and it is independent of the OS at the end. Since VS Code doesn't want to temper with what shells do when executed you can't override any variables that are later on sourced from somewhere else.

I tested this and ~/.bash_profile is not sourced when running a task for me. What happens when you simply start bash without --login. Will it source ~/.bash_profile

[zfields]

I think we are talking past each other, maybe we should take two steps back...

I have no desire to use a shell. In fact, I ONLY want to use VSCode.

However, when running a "shell task" in VSCode (i.e. diagnostic from above), I happened to notice VSCode picks up environment variables that are only set in my ~/.bash_profile. If you are not able to reproduce this on your non-Apple device, then perhaps it IS platform specific.

I think we need to make sure we are experiencing the same behavior, before we can attempt to solve this problem.

[dbaeumer]

I tried this on Apple and Linux and in both cases the ~/.bash_profile was not source for me unless I used a login shell. This is why I ask you what happens for you when you run bash from a different shell or even form a terminal already running bash without --login. Will the code from ~/.bash_profile be executed.

[zfields]

I'm sorry, I am completely confused.

I am not trying (and do not wish) to use any shell whatsoever, I am only trying to use the "tasks" detailed by tasks.json from the VSCode UI, on a Mac.

Your directives about running bash with different parameters and using different shells make no sense to me - unless those are options that need to be entered into the tasks.json file itself.

Can you deliberately spell out what you mean by "this" in the following sentence?

I tried this on Apple and Linux...

Meanwhile, I will try to shutdown my machine completely. Only open VSCode and no other applications, and see if the environment variables listed in ~/.bash_profile appear when I run the diagonostic "task" I have provided to you above.

Futhermore, I WILL NOT be opening a shell (deliberately), unless it is done so on my behalf, without my knowing, by VSCode or macOS.

[zfields]

Just to confirm...

1. I cold booted my Mac

2. Opened VSCode to a new folder on the system

3. Added a new task "diagnostic", based on the "Others" template

   {
       // See https://go.microsoft.com/fwlink/?LinkId=733558
       // for the documentation about the tasks.json format
       "version": "2.0.0",
       "tasks": [
           {
               "label": "diagnostic",
               "type": "shell",
               "command": "printenv"
           }
       ]
   }
   

4. I execute the task from the "Tasks >> Run Task..." menu

1. In the output appearing in the "TERMINAL" tab of VSCode UI, I can see environment variables only declared in ~/.bash_profile

How is this happening, if executing a task from the UI in VSCode is not loading environment varialbes from ~/.bash_profile?

[dbaeumer]

If you create a tasks of type shell then VS Code starts a shell to execute the command. In our case printenv. The shell used is the one configured as the integrated shell for VS Code using the setting "terminal.integrated.shell.osx". Did you configure any shell args for the integrated shell ?

[zfields]

No, I wasn't aware the setting existed, so any behavior is the default behavior of VSCode. Following on, does it stand to reason that VSCode does in fact consume ~/.bash_profile?

If so, are you able to inject the environment variables I have supplied in tasks.options.env, AFTER it consumes ~/.bash_profile? If you can, it would truly be helpful when constructing an environment in which to execute a task (the crux of the issue at hand).

[zfields]

I haven't heard anything in a few days, and the issue is still marked as "needs more info". I just wanted to make sure you are not waiting on me.

[dbaeumer]

Sorry, we are in end game. If you want to inject something into the shell the best is to build a command that does both. For example 'set X="abc" & myCommand'.

What happens when you open an integrated terminal and execute printenv in it. Does it source things from the profile as well.

[zfields]

What happens when you open an integrated terminal and execute printenv in it. Does it source things from the profile as well.

Yes.

Sorry, we are in end game. If you want to inject something into the shell the best is to build a command that does both. For example 'set X="abc" & myCommand'.

What is the point of specifying tasks.options.env, if putting everything in the command string is the only reliable way to ensure the environment variables are getting set?

It seems like I have identified a bug for you, and you have identified a solution. However, you are suggesting that I hack it in place, instead of VSCode correctly assembling the command string by prepending the setting of environment variables to the specified command.

To use your example, VSCode is taking a command string from me, and also has all the environment variables I wish to have set.

{
...
  "options": {
    "env": {
      "X": "abc"
    }
  },
  "command": "myCommand",
...
}

Why isn't VSCode capable of constructing the following string on my behalf, before passing myCommand to its internal command line API?

set X="abc" && myCommand

Note the "double ampersand" to ensure they run synchronously and complete successfully

[hellboy81]

What is the best way to set new env var and or overwrite existing PATH var required only for test build on machine without admin rights?

.NET classic is used

[dbaeumer]

Yes.

This is really strange since values from the profile should only be sourced when --login is specified and the rc files usually have interactive check at the beginning. Have you configured anything special on our machine.

We don't inject something into the shell to keep it consistent with the process type were we can't do such tricks and rely on what the process sees when executed.

I will keep the item open to consider it as a feature request to treat this different for shells

[dbaeumer]

@hellboy81 usually you should be able to overwrite the PATH by specifying the "env" in the "options". Is this not working for you?

[zfields]

Have you configured anything special on our machine.

Did you mean "your" machine? If so, then no, I have not.

... a feature request to treat this different for shells

Do you mean "shell tasks"? Shell tasks are the only thing I am addressing in this issue.

[zfields]

Based on my experience thus far, I would not expect @hellboy81 to be able to overwrite an existing environment variable (i.e. PATH), using "env" in the "options"; the namesake of this issue.

[dbaeumer]

I meant your machine. I read a little bit more about it and it looks like under MacOS it depends on which terminal is used whether the ./bash_profile is source or nor for non login shells. The Terminal.app doesn't follow the convention of only sourcing ./bash_profile for login shells. I tested it with the Terminal.app and there I see it source every time :-(

[zfields]

I've lost your train of thought. Can you take a moment to explain what the results of your Terminal.app test mean in the context of "shell tasks" in VSCode and how it affects this issue?

[dbaeumer]

Sorry for the short answer. Normally bash defines that ~/.bash_profile should only be considered when the shell / terminal is a login shell (for example when you login over ssh into a machine). However the standard Mac Terminal application doesn't stick to this rule and considers the content of ~/.bash_profile for all started shell (hence the behavior you are seeing). When I retested this today I used the standard Mac Terminal as was able to reproduce the behavior you are seeing.

[dbaeumer]

@zfields thanks a lot for testing this. I will look into why some of the behavior is unexpected.

Regarding 2.) this is expected in the sense that things from ~/.bashrc should not influence shells started with -c (which tasks do). If you want a different behavior you need to remove

case $- in
    *i*) ;;
      *) return;;
esac

from the beginning however then exports from the ~/.bashrc will always win.

Can you let me know in terms of variables which once you want to define were and which once should win?

[zfields]

My expectation is as follows. If I am defining environment variables in tasks.options.env, then I would expect those environment variables to be available to the task, regardless. In other words, tasks.options.env should override everything else.

Specifically, DYLD_LIBRARY_PATH is very important to me. Several of my workflows require my company's release libraries (i.e. export DYLD_LIBRARY_PATH=/Users/zfields/lib set in ~/.bash*). However, when I'm in a development environment I will need to override the release libraries with debug versions .../debug/lib so I can step through and debug.

It would be great if I could simply override DYLD_LIBRARY_PATH in the task, so I can develop and debug without modifying my file system.

[dbaeumer]

This should actually work by having the default DYLD_LIBRARY_PATH value in the ~/.bashrc and then override in the env setting of the task. I will give it a try on a Mac to verify it is indeed working.

[giuseppegalano]

Can this be related to Runtime Protections?

https://developer.apple.com/library/content/documentation/Security/Conceptual/System_Integrity_Protection_Guide/RuntimeProtections/RuntimeProtections.html

BTW the one mentioned by @zfields is the desired feature for most vscode users: it would be great if I could simply override DYLD_LIBRARY_PATH in the task, so I can develop and debug without modifying my file system.

[dbaeumer]

In this case the easiest is for now to have a task command line that sets DYLD_LIBRARY_PATH to the desired value.

[dbaeumer]

Some more digging here with our Mac expert and here is a workaround for now: in the task that wants to override the environment variable add the following:

			"options": {
				"shell": {
					"executable": "bash",
					"args": [
						"-c"
					]
				}
			}

Let me know if this fixes the problem for you. If so I can make a change in VS Code itself that doesn't make this necessary anymore.

[zfields]

Task:

{
    "label": "__diagnostic",
    "type": "shell",
    "options": {
        "cwd": "${workspaceFolder}/beam",
        "env": {
            "COVERAGE": "1",
            "DEBUG": "1",
            "DEBUG_FLAG": "-debug",
            "DOLBY": "1",
            "PATH": "foo",
            "DYLD_LIBRARY_PATH": "/Users/zfields/lib/:/Users/zfields/debug/lib/"
        },
        "shell": {
            "executable": "bash",
            "args": [
                "-c"
            ]
        }
    },
    "command": "echo BAZ=$BAZ; echo COVERAGE=$COVERAGE; echo DEBUG=$DEBUG; echo DEBUG_FLAG=$DEBUG_FLAG; echo DOLBY=$DOLBY; echo DYLD_LIBRARY_PATH=$DYLD_LIBRARY_PATH; echo FOO=$FOO; echo PATH=$PATH; echo pwd=$(pwd)",
    "problemMatcher": []
}

Results in:

> Executing task: echo BAZ=$BAZ; echo COVERAGE=$COVERAGE; echo DEBUG=$DEBUG; echo DEBUG_FLAG=$DEBUG_FLAG; echo DOLBY=$DOLBY; echo DYLD_LIBRARY_PATH=$DYLD_LIBRARY_PATH; echo FOO=$FOO; echo PATH=$PATH; echo pwd=$(pwd) <

execvp(3) failed.: No such file or directory
The terminal process terminated with exit code: 1

Terminal will be reused by tasks, press any key to close it.

If I remove the shell object from the JSON, then the task runs without exception

BAZ=boz
COVERAGE=1
DEBUG=1
DEBUG_FLAG=-debug
DOLBY=1
DYLD_LIBRARY_PATH=
FOO=bar
PATH=/Users/zfields/.pyenv/shims:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:foo
pwd=/Users/zfields/dev/beam

[dbaeumer]

@zfields thanks. The error comes from the fact that VS Code couldn't find bash. Can you make it an absolute path and retry.

And can you check if this then lets you replace the FOO variable. It did work for us in that setup. With this change there is no need to have a ~/.bashrc and source it from the profile as recommended in #47985 (comment)

[zfields]

$ which bash
/bin/bash

I modified __diagnostic in tasks.json accordingly

...
                "shell": {
                    "executable": "/bin/bash",
                    "args": [
                        "-c"
                    ]
                }
...

Result:

BAZ=bat
COVERAGE=1
DEBUG=1
DEBUG_FLAG=-debug
DOLBY=1
DYLD_LIBRARY_PATH=
FOO=bar
PATH=foo
pwd=/Users/zfields/dev/beam

Observations:

1. BAZ is picked up from .bash_profile
2. FOO is picked up from .bashrc
3. I can overwrite both BAZ and FOO from __diagnostic 👍
4. PATH can now be overridden correctly 👍
5. Neither BAZ nor FOO will reflect a change in the filesystem (previously BAZ would update - now behaves like reset in bash, but not like . ~\.bash_profile)
6. DYLD_LIBRARY_PATH cannot be overridden 👎 👎 👎

[dbaeumer]

Actually that puzzles me. You can override BAZ and FOO but not DYLD_LIBRARY_PATH. I thought you set it in .bash_profile like you set BAZ. Can you think of any different between them.

With the workaround of specifying the shell there is actually no need anymore to have a .bashrc. If I understood you correctly you didn't have one before either. So you can move everything back into .bash_profile if you want.

Just to check: does the .bashrc you have have the following snippet at the top

case $- in
    *i*) ;;
      *) return;;
esac

[zfields]

I double checked and match exactly on the .bashrc (I originally copy/pasted it from you earlier).

I have remerged ~/.bashrc and ~/.bash_profile. I'm rebooting now and I will let you know what happens.

--- After reboot ---

There is no difference between the way I specify DYLD_LIBRARY_PATH and BAZ. They are line items right next to each other in ~/.bash_profile and they both show up in the terminal correctly.

Still missing DYLD_LIBRARY_PATH...

In VSCode (__diagnostic):

BAZ=bat
COVERAGE=1
DEBUG=1
DEBUG_FLAG=-debug
DOLBY=1
DYLD_LIBRARY_PATH=
FOO=bar
PATH=foo
pwd=/Users/zfields/dev/uplynk_source/uplynk/beam

In VSCode (TERMINAL):

$ echo BAZ=$BAZ; echo DYLD_LIBRARY_PATH=$DYLD_LIBRARY_PATH
BAZ=bat
DYLD_LIBRARY_PATH=/Users/zfields/.lib/:

NOTE: VSCode is now version 1.23.1, and C/C++ Extension is version 0.17.3

[dbaeumer]

If you don't override it in the env will the task print the value correctly that is specified in .bash_profile?

[zfields]

No, it doesn't.

Removing the override:

> Executing task: echo BAZ=$BAZ; echo COVERAGE=$COVERAGE; echo DEBUG=$DEBUG; echo DEBUG_FLAG=$DEBUG_FLAG; echo DOLBY=$DOLBY; echo DYLD_LIBRARY_PATH=$DYLD_LIBRARY_PATH; echo FOO=$FOO; echo PATH=$PATH; echo pwd=$(pwd) <

BAZ=bat
COVERAGE=1
DEBUG=1
DEBUG_FLAG=-debug
DOLBY=1
DYLD_LIBRARY_PATH=
FOO=bar
PATH=foo
pwd=/Users/zfields/dev/beam

Terminal will be reused by tasks, press any key to close it.

However, if I press "Enter" to close the task terminal and fall back to the terminal in VSCode, then I copy/paste the command string from above and get these results:

$ echo BAZ=$BAZ; echo COVERAGE=$COVERAGE; echo DEBUG=$DEBUG; echo DEBUG_FLAG=$DEBUG_FLAG; echo DOLBY=$DOLBY; echo DYLD_LIBRARY_PATH=$DYLD_LIBRARY_PATH; echo FOO=$FOO; echo PATH=$PATH;echo pwd=$(pwd)
BAZ=bat
COVERAGE=
DEBUG=
DEBUG_FLAG=
DOLBY=
DYLD_LIBRARY_PATH=/Users/zfields/.lib:
FOO=bar
PATH=/Users/zfields/.pyenv/shims:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
pwd=/Users/zfields/dev/beam

[dbaeumer]

I will test this again on a Mac. I have to admit that I have no idea what makes DYLD_LIBRARY_PATH special on the Mac. However I did find articles on the web saying that you shouldn't temper with this env var and use otool -L instead. But I am not an expert when it comes to dynamic link libraries and Mac.

[Glavin001]

May not be 💯% related to the above, although I stumbled upon this and found options.shell recommended.
Using the options.shell within the task did not work for me to load the ~/.bashrc or ~./bash_profile.

This is what worked for me:

        {
            "label": "my-task",
            "type": "shell",
            "command": "bash --login -c 'run-command'",

Hope it helps someone else! 🎉

[jwmurray]

Should this work the same on Windows?

I have the following task:

{
            "label": "test"
            ,"type": "shell"
            ,"options": {
                "env": {
                    "message": "hello"
                }
            }
            
            ,"windows": {
                
                "command": "echo $message"
            }
            ,"group": {
                "kind": "build",
                "isDefault": true
            }
            ,"presentation": {
                "panel": "new",
            }
}

and get the following output:

> Executing task: echo $message <


Press any key to close the terminal.

Not sure why $message is not expanded to the environment variable that is being set.

[zfields]

@jwmurray Please open a separate issue for your experience, as it is not directly related to this issue. This issue is still open / under investigation, and it took forever for the bug listed above to be recognized and confirmed by Microsoft. I'm sorry to shut you down, but there has been enough confusion on this thread.

@dbaeumer It has been several months since you were going to test again on your Mac, and I am curious if you have been able to land on a solution.

[dbaeumer]

I will reactive my Mac and will have a look beginning of next week.

[fumfi]

This issue and the handling of "env" options in tasks.json is kind of confusing.
I am developing on Windows and Linux (and prototyping for the latter on WSL) and adding to what @zfields reported, I have created this tasks.json.

{
    "version": "2.0.0",
    "tasks": [
        {
            "label": "check (bash)",
            "type": "shell",
            "windows": {
                "options": {
                    "shell": {
                        "executable": "bash.exe",
                        "args": [ "-c" ]
                    }
                }
            },
            "linux" : {
                "options": {
                    "shell": {
                        "executable": "bash",
                        "args": [ "-c" ]
                    }
                }
            },
            "options": {
                "env": {
                    "BAR" : "BAZ"
                },
            },
            "command": "echo env{FOO}=${env:FOO} env{BAR}=${env:BAR} var{BAR}=$BAR",
            "problemMatcher": []
        },
        {
            "label": "check (cmd)",
            "type": "shell",
            "windows": {
                "options": {
                    "shell": {
                        "executable": "cmd.exe",
                        "args" : [ "/C" ]
                    }
                }
            },
            "options": {
                "env": {
                    "BAR" : "BAZ"
                }
            },
            "command": "echo env{FOO}=${env:FOO} env{BAR}=${env:BAR} var{BAR}=%BAR%",
            "problemMatcher": []
        }
    ]
}

Results (without FOO or BAR set/exported in the parent shell that executes code)

1. Windows cmd.exe env{FOO}= env{BAR}= var{BAR}=BAZ
2. WSL bash.exe env{FOO}= env{BAR}= var{BAR}=
3. Linux/Mac OS bash env{FOO}= env{BAR}= var{BAR}=BAZ

Results (with set/exported FOO=foo BAR=bar in the parent shell that executes code)

1. Windows cmd.exe env{FOO}=foo env{BAR}=bar var{BAR}=BAZ
2. WSL bash.exe env{FOO}=foo env{BAR}=bar var{BAR}=
3. Linux/Mac OS bash env{FOO}=foo env{BAR}=bar var{BAR}=BAZ

I am assuming that Mac OS behaves like Linux.
Update 2019/02/13: MacOS behaves like Linux.

Tested with Visual Studio Code 1.31.

Bottom line is:

• ${env:...} only expands environment variables that were set in the parent shell that ran code. It doesn't expand variables set in the tasks.json env options.
• %var% expands env variables from the tasks.json options when running cmd.exe. So does $var for real Linux bash. But on WSL, $var doesn't expand env variables from the tasks.json options.
• %var% (cmd.exe) and $var (Linux bash) may expand to values different from ${env:var} if var was set in code's parent shell.

Is this a bug or by design??

FWIW, I would prefer if ${env:var} would also expand the env options of the tasks.json file, superseding everything that was set in the environment of code's parent shell. If ${env:var} is the standard way to expand environment (and env options) variables, the task's command may be unified across platforms because there's no need for $var vs. %var%.

If not a bug, would that be a feature request?

[zmmille2]

Since you are defining the value in the tasks.json, you already know the expanded value and you could easily just include the expanded value in the task command.
#72323 (comment)

This deserves reconsideration. Yes, I know the value that's going to be expanded, but I'd like to not check this value into source code.

I'd really like the ability to run tasks with arguments passed from launch.json, but pre-defining my environment or arguments in tasks.json is a non-starter for me. I need a .env or local.settings.json or equivalent file that I can read from and .gitignore.

[DBJDBJ]

microsoft/vscode-cpptools#6450 (comment)

microsoft/vscode-cpptools#6450 (comment)

[alexr00]

Generally speaking, env vars set in task options will be set in the task shell and will overwrite existing ones of the same name. Closing as that's what this bug is about.

[zfields]

I can confirm this works as expected now, and the point of this original issue has been resolved to my satisfaction.

Thank you!

[DBJDBJ]

https://dbj.org/how-to-build-software-and-still-have-some-time-for-programming/