Add .wp-env.json support to wp-env

[noisysocks]

Adds support to wp-env for .wp-env.json files as described in #17724 (comment).

Specification

.wp-env.json supports three fields:

Field	Type	Default	Description
"core"	string or null	null	The WordPress installation to use. If null is specified, wp-env will use the latest production release of WordPress.
"plugins"	string[]	[]	A list of plugins to install and activate in the environment.
"themes"	string[]	[]	A list of themes to install in the environment. The first theme in the list will be activated.

Several types of strings can be passed into these fields:

Type	Format	Example(s)
Relative path	.<path> or ~<path>	"./a/directory", "../a/directory", "~/a/directory"
Absolute path	/<path> or <letter>:\<path>	"/a/directory", "C:\\a\\directory"
GitHub repository	<owner>/<repo>[#<ref>]	"WordPress/WordPress", "WordPress/gutenberg#master"

Remote sources will be downloaded into a temporary directory located in ~/.wp-env.

Examples

Latest production WordPress + current directory as a plugin

This is useful for plugin development.

{
  "core": null,
  "plugins": [
    "."
  ]
}

Latest development WordPress + current directory as a plugin

This is useful for plugin development when upstream Core changes need to be tested.

{
  "core": "WordPress/WordPress#master",
  "plugins": [
    "."
  ]
}

Local wordpress-develop + current directory as a plugin

This is useful for working on plugins and WordPress Core at the same time.

{
  "core": "../wordpress-develop/build",
  "plugins": [
    "."
  ]
}

A complete testing environment

This is useful for integration testing: that is, testing how old versions of WordPress and different combinations of plugins and themes impact each other.

{
  "core": "WordPress/WordPress#5.2.0",
  "plugins": [
    "WordPress/wp-lazy-loading",
    "WordPress/classic-editor",
  ],
  "themes": [
    "WordPress/theme-experiments"
  ]
}

Testing

1. Check out this branch
2. cd to a directory that contains a WordPress installation, a plugin, a theme, or a .wp-env.json file
3. Run /path/to/gutenberg/packages/env/bin/wp-env start

Questions

• Is the wp-env reset command useful? It has some problems, namely that it requires wp-env start to have been run at least once. Since we’re bumping the major version here it might be a good chance to replace wp-env-reset with e.g. wp-env destroy which destroys all containers and volumes so that one can run wp-env start from a clean slate. This would be useful for troubleshooting.
• What is the tests-wordpress-phpunit container used for? I could not find where this is used and so have left it out for now.

[epiqueras]

Amazing work here!

I think you need to import types to the files where they are used for them to work.

Is the wp-env reset command useful? It has some problems, namely that it requires wp-env start to have been run at least once. Since we’re bumping the major version here it might be a good chance to replace wp-env-reset with e.g. wp-env destroy which destroys all containers and volumes so that one can run wp-env start from a clean slate. This would be useful for troubleshooting.

We don't have a wp-env reset but a wp-env destroy sounds very useful. Perhaps we should add it in a follow-up?

What is the tests-wordpress-phpunit container used for? I could not find where this is used and so have left it out for now.

It was used for PHP unit tests but we held off on it due to flakiness.

[epiqueras]

This needs a dev note, right?

[noisysocks]

I noted that this breaks backwards compatibility in packages/env/CHANGELOG.md but a agree dev note would be good, too. If nothing else it will help publicise that this tool can be used for core development.

[epiqueras]

👍

[noahtallen]

I get this error running it locally (in a WordPress plugin with and also without a config file)

~/source/wp-calypso/apps/full-site-editing/full-site-editing-plugin(try/wordpress-env-for-tests) » ~/source/gutenberg/packages/env/bin/wp-env start                     noah.allen@Noahs-MacBook-Pro
internal/util.js:214
    throw new errors.TypeError('ERR_INVALID_ARG_TYPE', 'original', 'function');
    ^

TypeError [ERR_INVALID_ARG_TYPE]: The "original" argument must be of type function
    at Object.promisify (internal/util.js:214:11)
    at Object.<anonymous> (/Users/noah.allen/source/gutenberg/packages/env/lib/detect-directory-type.js:15:23)
    at Module._compile (module.js:653:30)
    at Object.Module._extensions..js (module.js:664:10)
    at Module.load (module.js:566:32)
    at tryModuleLoad (module.js:506:12)
    at Function.Module._load (module.js:498:3)
    at Module.require (module.js:597:17)
    at require (internal/module.js:11:18)
    at Object.<anonymous> (/Users/noah.allen/source/gutenberg/packages/env/lib/config.js:13:29)

[epiqueras]

You are running an old node version.

[noahtallen]

Thanks, @epiqueras! Spot on. Turns out I had to rm -rf node_modules and reinstall everything with the correct node version to get it to work (neither nvm use && npm i nor nvm use && npm rebuild fixed the issue.)

[epiqueras]

Glad I could help. That’s probably due to some custom post install scripts.

[noahtallen]

I'm now getting the following output, which leads me to believe that the custom port number is not working quite right. I'm not sure if it was entirely working before though, as I had similar issues with @wordpress/scripts env with custom port numbers. (Speaking of, it'd be great to add a custom port number to .wp-env.json!)

WP_ENV_PORT=3333 ./packages/env/bin/wp-env start                                                                                              

✖ 31911d623e75f345e9ed328b9f48cff6_mysql_1 is up-to-date
Recreating 31911d623e75f345e9ed328b9f48cff6_wordpress_1 ... 
Starting 31911d623e75f345e9ed328b9f48cff6_tests-wordpress_1 ... 
Starting 31911d623e75f345e9ed328b9f48cff6_tests-wordpress_1 ... error

Recreating 31911d623e75f345e9ed328b9f48cff6_wordpress_1     ... done1 (2e501b716d5fdfed3e33bad99337702ef97588c9cabb94003ce8735b5e85d28f): Bind for 0.0.0.0:8889 failed: port is already allocated

ERROR: for tests-wordpress  Cannot start service tests-wordpress: driver failed programming external connectivity on endpoint 31911d623e75f345e9ed328b9f48cff6_tests-wordpress_1 (2e501b716d5fdfed3e33bad99337702ef97588c9cabb94003ce8735b5e85d28f): Bind for 0.0.0.0:8889 failed: port is already allocated
Encountered errors while bringing up the project.



 {
  exitCode: 1,
  err: '31911d623e75f345e9ed328b9f48cff6_mysql_1 is up-to-date\n' +
    'Recreating 31911d623e75f345e9ed328b9f48cff6_wordpress_1 ... \r\n' +
    'Starting 31911d623e75f345e9ed328b9f48cff6_tests-wordpress_1 ... \r\n' +
    'Host is already in use by another container\n' +
    '\u001b[1A\u001b[2K\rStarting 31911d623e75f345e9ed328b9f48cff6_tests-wordpress_1 ... \u001b[31merror\u001b[0m\r\u001b[1B\n' +
    'ERROR: for 31911d623e75f345e9ed328b9f48cff6_tests-wordpress_1  Cannot start service tests-wordpress: driver failed programming external connectivity on endpoint 31911d623e75f345e9ed328b9f48cff6_tests-wordpress_1 (2e501b716d5fdfed3e33bad99337702ef97588c9cabb94003ce8735b5e85d28f): Bind for 0.0.0.0:8889 failed: port is already allocated\n' +
    '\u001b[2A\u001b[2K\rRecreating 31911d623e75f345e9ed328b9f48cff6_wordpress_1     ... \u001b[32mdone\u001b[0m\r\u001b[2B\n' +
    'ERROR: for tests-wordpress  Cannot start service tests-wordpress: driver failed programming external connectivity on endpoint 31911d623e75f345e9ed328b9f48cff6_tests-wordpress_1 (2e501b716d5fdfed3e33bad99337702ef97588c9cabb94003ce8735b5e85d28f): Bind for 0.0.0.0:8889 failed: port is already allocated\n' +
    'Encountered errors while bringing up the project.\n',
  out: ''
} 

To remedy

• Stop and remove the conflicting docker container
• Run the command again with a custom port number (i.e. make an environment for the Gutenberg repo)
• The port number works correctly but localhost:8889 redirects to the custom port as well.

If I then try to add a separate instance with a different port number, it also fails. So it seems that even if you specify a custom port number, something under the hood still allocates the default port (8889) for use by the original container. And then when you try to do a second wp-env instance, it thinks that 8889 has been used by something, so it also fails.

Here's the output after 1. stopping and removing all docker images. 2. creating a wp-env environment with the custom port 3333. 3. Running the command again in a different plugin with a different port number (9999):

✖ 58d3a7453e296b6fce00337462f0c1c5_mysql_1 is up-to-date
Creating 58d3a7453e296b6fce00337462f0c1c5_tests-wordpress_1 ... 
Creating 58d3a7453e296b6fce00337462f0c1c5_tests-wordpress_1 ... error
Host is already in use by another container

ERROR: for 58d3a7453e296b6fce00337462f0c1c5_tests-wordpress_1  Cannot start service tests-wordpress: driver failed programming external connectivity on endpoint 58d3a7453e296b6fce00337462f0c1c5_tesCreating 58d3a7453e296b6fce00337462f0c1c5_wordpress_1       ... done

ERROR: for tests-wordpress  Cannot start service tests-wordpress: driver failed programming external connectivity on endpoint 58d3a7453e296b6fce00337462f0c1c5_tests-wordpress_1 (c3fdce977ec32954b3af0c8d638173bcb1e9c47c3915e8d5c47a2d04957f2ad6): Bind for 0.0.0.0:8889 failed: port is already allocated
Encountered errors while bringing up the project.



 {
  exitCode: 1,
  err: '58d3a7453e296b6fce00337462f0c1c5_mysql_1 is up-to-date\n' +
    'Creating 58d3a7453e296b6fce00337462f0c1c5_tests-wordpress_1 ... \r\n' +
    'Creating 58d3a7453e296b6fce00337462f0c1c5_wordpress_1       ... \r\n' +
    'Host is already in use by another container\n' +
    '\u001b[2A\u001b[2K\rCreating 58d3a7453e296b6fce00337462f0c1c5_tests-wordpress_1 ... \u001b[31merror\u001b[0m\r\u001b[2B\n' +
    'ERROR: for 58d3a7453e296b6fce00337462f0c1c5_tests-wordpress_1  Cannot start service tests-wordpress: driver failed programming external connectivity on endpoint 58d3a7453e296b6fce00337462f0c1c5_tests-wordpress_1 (c3fdce977ec32954b3af0c8d638173bcb1e9c47c3915e8d5c47a2d04957f2ad6): Bind for 0.0.0.0:8889 failed: port is already allocated\n' +
    '\u001b[1A\u001b[2K\rCreating 58d3a7453e296b6fce00337462f0c1c5_wordpress_1       ... \u001b[32mdone\u001b[0m\r\u001b[1B\n' +
    'ERROR: for tests-wordpress  Cannot start service tests-wordpress: driver failed programming external connectivity on endpoint 58d3a7453e296b6fce00337462f0c1c5_tests-wordpress_1 (c3fdce977ec32954b3af0c8d638173bcb1e9c47c3915e8d5c47a2d04957f2ad6): Bind for 0.0.0.0:8889 failed: port is already allocated\n' +
    'Encountered errors while bringing up the project.\n',
  out: ''
} 

[epiqueras]

You are not specifying a custom port for the tests container so that one conflicts across environment installs.

[noisysocks]

Thanks for the quick feedback @enriquesanchez @epiqueras! (oops) I've rebased this PR, addressed all of your inline comments, and fixed an issue where both the development and tests environments pointed to the same database.

I think you need to import types to the files where they are used for them to work.

👍 done.

We don't have a wp-env reset but a wp-env destroy sounds very useful. Perhaps we should add it in a follow-up?

Sorry, I meant to say: Is wp-env clean useful? It has some shortfalls, and since I'm bumping the major version here, it's maybe a good opportunity to remove wp-env clean in lieu of something like wp-env destroy.

It was used for PHP unit tests but we held off on it due to flakiness.

👍 I'll leave it removed for now then.

[epiqueras]

Haha, you got the wrong Enrique.

Sorry, I meant to say: Is wp-env clean useful? It has some shortfalls, and since I'm bumping the major version here, it's maybe a good opportunity to remove wp-env clean in lieu of something like wp-env destroy.

I see clean being used to reset a testing environment and destroy being used when things break as a last troubleshooting resort so both can be useful.

I think this is good to merge now. Its a kind of PR that needs a lot of testing in the wild and we can always iterate before the next release. The foundation is strong, thanks for the work!

[noisysocks]

Haha, you got the wrong Enrique.

I think I've done this a dozen times now 😂

[noahtallen]

This is looking really awesome! I hope the issue I noted isn't from yet another node problem :)

I was just thinking about this in the context of @wordpress/scripts. Since we no longer automatically link the directory from which you run the script, we can't really tell what the "main" plugin is for the environment anymore. Perhaps this won't be an issue in most cases, but if I want to run phpunit/phpcs on just my own plugin (perhaps in a CI environment), we'd need to be able to specify that somehow.

[epiqueras]

I was just thinking about this in the context of @wordpress/scripts. Since we no longer automatically link the directory from which you run the script, we can't really tell what the "main" plugin is for the environment anymore. Perhaps this won't be an issue in most cases, but if I want to run phpunit/phpcs on just my own plugin (perhaps in a CI environment), we'd need to be able to specify that somehow.

This doesn't change any of that. When you run phpunit/phpcs you specify a path in the host project.

[noisysocks]

Since we no longer automatically link the directory from which you run the script, we can't really tell what the "main" plugin is for the environment anymore.

If you take a look at the generated docker-compose.yml (cat ~/.wp-env/*/docker-compose.yml) you can see that the composer container is still binded to the directory that .wp-env.json was located in. We could borrow this approach for phpunit, phpcs, or any other containers that need a concept of main plugin should they ever come up. (Right now it's just composer as far as I can see.)

[noisysocks]

Thanks @tellthemachines for testing this on Windows! 😍

[simison]

Did you consider using Docker's (and Docker-compose's)'s native support for "env files"?

I would suggest defining defaults in one file and then letting optional file (or environment constants!) to overrun that.

That might solve issues with Linux and furthermore, be more familiar with anyone already used to Docker setups. Less code at our end, too so easier to maintain. Less our custom-layer specific issues (you'd be still left with Docker & Docker compose specific issues of course).

The file format would be different but that's more or less subjective if it's worse or better. Wider support for ways to input values (via environment variables & CLI arguments) is a great bonus.

• https://docs.docker.com/compose/env-file/
• https://docs.docker.com/compose/compose-file/#env_file
• https://docs.docker.com/compose/environment-variables/#the-env-file

Thoughts?

We have an example setup of something like this in Jetpack:

• https://github.com/Automattic/jetpack/blob/9fad80b4e01573c0400ad0ee4eb831be5b046ce5/docker/default.env
• https://github.com/Automattic/jetpack/blob/9fad80b4e01573c0400ad0ee4eb831be5b046ce5/docker/docker-compose.yml#L18-L20

[ockham]

Thanks @simison for the suggestion! I wasn't aware of (or had forgotten about) Docker env files, but if they (maybe together with a static docker-compose.override.yml -- rather than a dynamically generated one) can serve a similar purpose as .wp-env.json, I'd much prefer them.

Generally, if there's an existing tool and config format, I think it's preferrable to use that, rather than defining a new format.

Relying on an existing format means

• less maintenance burden for us
• less figuring out of obscure bugs
• less design needed to figure out edge cases
• less domain-specific config formats to learn for developers using our software

[epiqueras]

This goes against the goal of wp-env being minimal in its configuration. Docker-compose files are behemoths with colossal documentation. It's not a burden we want to put on users.

The Linux bug is simply a lack of permissions to write in the user folder.

[noahtallen]

Docker env file

We did briefly discuss adding support for environment variable support here: #20158 (comment). You can see some more reasoning for why we want to expose a minimal amount of configuration. Essentially, the goal is to be able to run wp-env start from any plugin and it just works without having to set anything up.

Additionally, if you want to use wp-env start to start and stop your docker compose files, you could always add the .env and/or docker-compose.override.yml in the location where wp-env puts the configuration (~/.wp-env/hash/docker-compose.yml) and Docker would pick up those files when it runs. It wouldn't be officially supported, but it could be a workaround.

but if they (maybe together with a static docker-compose.override.yml -- rather than a dynamically generated one) can serve a similar purpose as .wp-env.json, I'd much prefer them.

There would be nothing stopping anyone from setting up their WordPress configuration in this way. :) It isn't even that much work; the basic config is pretty lightweight. If you want to use docker-compose for lots of power 💪 , that's absolutely fine. But that's not quite the goal of wp-env. If you don't want to use wp-env, you don't have to, right?

I will say that wp-env gives us some stuff that docker compose does not:

• Download plugin/theme/wordpress source code from GitHub
• Can link local dev sources (plugins/themes/wordpress) to the instance without any headache
• More in the future :)

Linux issues

Back to the original issue (filesystems permissions in Linux), I wonder if we could allow users to specify the download location for wp-env source via an environment variable. If set, then wp-env would use that to download sourcecode and configurations to a different location. It seems like the type of setting that would be valuable "per user"

[epiqueras]

100% agreed.

[aduth]

I believe the single quotes here are interpreted verbatim by WP-CLI and are causing the Site Title of the default installation to incorrectly include them:

Depending whether it needs to be quoted to account for spaces in the name, it may need double quotes instead to work as expected?

Suggested change

	--title='${ config.name }'
	--title="${ config.name }"

Suggested change

	--title='${ config.name }'
	--title=${ config.name }

[noisysocks]

Thanks! Fixed in #20520.

[mewt]

Hey I would to report there is an error install

`sudo npm install -g @wordpress/env
npm WARN deprecated [email protected]: request has been deprecated, see request/request#3142
^T[ ......] - extract:json-schema: sill extract [email protected] extracted to /usr/lib/node_modules/.staging/js/usr/bin/wp-env -> /usr/lib/node_modules/@wordpress/env/bin/wp-env

[email protected] install /usr/lib/node_modules/@wordpress/env/node_modules/nodegit
node lifecycleScripts/preinstall && node lifecycleScripts/install

[nodegit] Running pre-install script
[nodegit] Running install script
node-pre-gyp
WARN Using request for node-pre-gyp https download
node-pre-gyp
WARN Pre-built binaries not installable for [email protected] and [email protected] (node-v83 ABI, glibc) (falling back to source compile with node-gyp)
node-pre-gyp WARN Hit error EACCES: permission denied, mkdir '/usr/lib/node_modules/@wordpress/env/node_modules/nodegit/build'
gyp
WARN
EACCES current user ("nobody") does not have permission to access the dev dir "/root/.cache/node-gyp/14.2.0"
gyp
WARN EACCES attempting to reinstall using temporary dev dir "/usr/lib/node_modules/@wordpress/env/node_modules/nodegit/.node-gyp"
gyp
WARN install got an error, rolling back install
gyp
WARN install got an error, rolling back install
gyp
ERR! configure error
gyp ERR!
stack Error: EACCES: permission denied, mkdir '/usr/lib/node_modules/@wordpress/env/node_modules/nodegit/.node-gyp'
gyp
ERR! System Linux 5.6.12-1-MANJARO
gyp ERR! command "/usr/bin/node" "/usr/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "configure" "--fallback-to-build" "--module=/usr/lib/node_modules/@wordpress/env/node_modules/nodegit/build/Release/nodegit.node" "--module_name=nodegit" "--module_path=/usr/lib/node_modules/@wordpress/env/node_modules/nodegit/build/Release" "--napi_version=6" "--node_abi_napi=napi" "--napi_build_version=0" "--node_napi_label=node-v83"
gyp ERR! cwd
/usr/lib/node_modules/@wordpress/env/node_modules/nodegit
gyp ERR! node -v v14.2.0
gyp ERR! node-gyp -v v5.1.0
gyp
ERR! not ok
node-pre-gyp
ERR! build error
node-pre-gyp ERR! stack Error: Failed to execute '/usr/bin/node /usr/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js configure --fallback-to-build --module=/usr/lib/node_modules/@wordpress/env/node_modules/nodegit/build/Release/nodegit.node --module_name=nodegit --module_path=/usr/lib/node_modules/@wordpress/env/node_modules/nodegit/build/Release --napi_version=6 --node_abi_napi=napi --napi_build_version=0 --node_napi_label=node-v83' (1)
node-pre-gyp ERR! stack at ChildProcess. (/usr/lib/node_modules/@wordpress/env/node_modules/node-pre-gyp/lib/util/compile.js:83:29)
node-pre-gyp ERR! stack at ChildProcess.emit (events.js:315:20)
node-pre-gyp ERR! stack at maybeClose (internal/child_process.js:1051:16)
node-pre-gyp ERR! stack at Process.ChildProcess._handle.onexit (internal/child_process.js:287:5)
node-pre-gyp
ERR! System Linux 5.6.12-1-MANJARO
node-pre-gyp ERR! command "/usr/bin/node" "/usr/lib/node_modules/@wordpress/env/node_modules/.bin/node-pre-gyp" "install" "--fallback-to-build"
node-pre-gyp ERR! cwd /usr/lib/node_modules/@wordpress/env/node_modules/nodegit
node-pre-gyp ERR! node -v
v14.2.0
node-pre-gyp ERR! node-pre-gyp -v v0.13.0
node-pre-gyp ERR! not ok
Failed to execute '/usr/bin/node /usr/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js configure --fallback-to-build --module=/usr/lib/node_modules/@wordpress/env/node_modules/nodegit/build/Release/nodegit.node --module_name=nodegit --module_path=/usr/lib/node_modules/@wordpress/env/node_modules/nodegit/build/Release --napi_version=6 --node_abi_napi=napi --napi_build_version=0 --node_napi_label=node-v83' (1)
[nodegit] ERROR - Could not finish install
[nodegit] ERROR - finished with error code: 1
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] install: node lifecycleScripts/preinstall && node lifecycleScripts/install
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR! /root/.npm/_logs/2020-05-24T04_51_37_658Z-debug.log
`

[aduth]

@mewt It's best to create a new issue than to comment on a merged pull request, as an issue would be easier to track. In your case it appears it could be a permission issue stemming from how the internal nodegit module is built from source (see related nodegit/nodegit#606, nodegit/nodegit#1537).

You shouldn't need nor want to use sudo to install the package. I'd suggest to test to see if that helps. Otherwise I'd recommend to see if a search of the issues in the upstream nodegit repository could resolve your issue, else create a new issue.