Functional api test unable to load install-config-mysql database credentials

[baatractive]

Preconditions (*)

1. Magento CLI 2.3.5
2. WSL (Windows Sub-System For Linux)

Steps to reproduce (*)

1. Follow the steps as per the documentation: https://devdocs.magento.com/guides/v2.4/get-started/web-api-functional-testing.html#howto and then try running the command: ../../../vendor/bin/phpunit --config phpunit_rest.xml

Expected result (*)

1. It should load the expected install-config-mysql.php file for database connection but it rather tries to connect with the default connection settings. For some reason the default connection is also throwing an error with this command. However, I could successfully run integration tests and also my default is fully functional when try to upgrade or clear any cache.

Actual result (*)

---

Please provide Severity assessment for the Issue as Reporter. This information will help during Confirmation and Issue triage processes.

• Severity: S0 - Affects critical data or functionality and leaves users without workaround.
• Severity: S1 - Affects critical data or functionality and forces users to employ a workaround.
• Severity: S2 - Affects non-critical data or functionality and forces users to employ a workaround.
• Severity: S3 - Affects non-critical data or functionality and does not force users to employ a workaround.
• Severity: S4 - Affects aesthetics, professional look and feel, “quality” or “usability”.

[m2-assistant]

Hi @baatractive. Thank you for your report.
To help us process this issue please make sure that you provided the following information:

• Summary of the issue
• Information on your environment
• Steps to reproduce
• Expected and actual results

Please make sure that the issue is reproducible on the vanilla Magento instance following Steps to reproduce. To deploy vanilla Magento instance on our environment, please, add a comment to the issue:

@magento give me 2.4-develop instance - upcoming 2.4.x release

For more details, please, review the Magento Contributor Assistant documentation.

Please, add a comment to assign the issue: @magento I am working on this

---

• Join Magento Community Engineering Slack and ask your questions in #github channel.

⚠️ According to the Magento Contribution requirements, all issues must go through the Community Contributions Triage process. Community Contributions Triage is a public meeting.

🕙 You can find the schedule on the Magento Community Calendar page.

📞 The triage of issues happens in the queue order. If you want to speed up the delivery of your contribution, please join the Community Contributions Triage session to discuss the appropriate ticket.

🎥 You can find the recording of the previous Community Contributions Triage on the Magento Youtube Channel

✏️ Feel free to post questions/proposals/feedback related to the Community Contributions Triage process to the corresponding Slack Channel

[baatractive]

@magento give me 2.4-develop instance

[magento-deployment-service]

Hi @baatractive. Thank you for your request. I'm working on Magento instance for you.

[magento-deployment-service]

Hi @baatractive, here is your Magento Instance: https://b6af6eaf68e0b9d60502403379a6271e-2-4-develop.instances.magento-community.engineering
Admin access: https://b6af6eaf68e0b9d60502403379a6271e-2-4-develop.instances.magento-community.engineering/admin_86ae
Login: a323e93b
Password: 2d210d2b68fd

[baatractive]

The API functional test uses app/etc/env.php but rather dev/tests/functional-api/config/install-mysql-config.php. I changed the default credentials to api functional ones and the tests started working.

[gwharton]

#27213

[baatractive]

@gwharton Yes that seems a similar issue and why is it closed? Was it resolved?

[gwharton]

Lack of interest from Magento team.

[gwharton]

I'll take a look at this later today as I almost had a fix working when I last looked at this.

[baatractive]

@gwharton That would be great help, thanks.

[mrtuvn]

as i remembered you need create seperate db tmp-mysql for test magento_integration_tests. It's seperate service just like apache/nginx

[mrtuvn]

elasticsearch seem required too
#30995 < = check this

[gwharton]

OK, ive had a look at the API Functional test code and how it works and it exposes a fundamental issue with the way the API Functional tests work.

Lets consider the integration tests first. When they are run, they use the install-mysql-config.php and the test framework installs a bastardised runtime of magento into dev/tests/integration/tmp/sandbox---------- and creates a separate database as defined in install-mysql-config.php. The tests are all run in the context of this sandbox test environment so as to not affect any data in the main development database. This works fine and is well understood.

Now, the API Functional tests. Well, these seem to follow the same approach, as there is a install-mysql-config.php file detailing a separate database however the bootstrap functions in the api functional tests operate differently to the integration tests. The test framework seems to have been implemented by a mixture of people who don't understand the difference between api functional and integration tests, and people that have hacked it to appear to work. Let me explain.

So, the difference between the two test types : whereas the integration tests run in their own sandbox, you cannot do this for the api functional tests so easily. The API Functional tests are essentially very simple REST/SOAP/GRAPHQL clients which makes a call to a Magento HTTP(s) endpoint. Had the test framework been configured to work in the same way as the integration tests with its own sandbox using details in install-mysql-config.php, then it would be ONLY the REST/SOAP/GRAPHQL clients that run in this sandbox. You would have a sandboxed database in existence for the test client, which then makes a call to a HTTP(s) endpoint somewhere else. but where??? There is only one HTTP(s) based Magento endpoint and that is the one configured in app/etc/env.php. We do not have a separate webserver setup that could serve the sandboxed version of the api-functional test installation.

I believe at this point the Magento team may well have just thrown their hands in the air and decided that the API functional tests should not run in a sandbox environment, and adjusted the test framework bootstrap code to operate on the main database. They did leave in a couple of nasty features though

• if the <const name="TESTS_CLEANUP" value="disabled"/> setting is enabled in phpunit.xml then the system attempts to uninstall the system defined in app/etc/env.php
• if the <const name="TESTS_MAGENTO_INSTALLATION" value="enabled"/> setting is enabled in phpunit.xml then the system reinstalls into the main database defined in app/etc/env.php

So, what do we do. The current API functional tests intentionally ignore the install-mysql-config.php data and use the app/etc/env.php details instead. The tests operate on the HTTP(s) endpoint configured in app/etc/env.php with the result being the tests are performed on the main development database and not a sandboxed version. People assume it uses a sandbox database but it doesn't, it uses whichever database is configured behind the HTTP(s) endpoint. Their development database will be being polluted with data from the API Functional tests, if not cleaned up properly by the test.

It is fairly straight forward to adjust the test framework to use the install-mysql-config.php settings and generate a sandbox for the REST/SOAP/GRAPHQL client to run in, but we would also need to either adjust the main app/etc/env.php to temporarily use the sandbox details from install-mysql-config.php for the duration of the test, or create a separate web server config that points at the sandbox installation.

[baatractive]

@gwharton I see, I investigated this issue too but thought I might be wrong because they already provide install-mysql-config.php which should clearly work the same as for integration tests but I was right as their configuration was not correct. It truncated my env database and then I re-uploaded the dump.

I am fairly new to Magento so not quite sure for any alternative approcah especially as you suggested: create a web server config that points at the sandbox installation. Could you give me any hint for this as how to achieve it?

[baatractive]

@mrtuvn There's already a database for the integration which is working properly but the current issue is with the Functional Api tests.

[gwharton]

A little bit more research that confirms the fundamental broken'ness of the API Functional tests.

If the following settings are set in phpunit.xml

<const name="TESTS_CLEANUP" value="disabled"/>
<const name="TESTS_MAGENTO_INSTALLATION" value="disabled"/>

Then running the API-Functional tests will run them against the Magento HTTP(s) endpoint as defined in the setting <const name="TESTS_BASE_URL" value="http://localhost:8080/"/>. It will not attempt to uninstall or reinstall the database.

if either of the TESTS_CLEANUP (ununstalls magento) or TESTS_MAGENTO_INSTALLATION (installs magento) are set to enabled, then because of this bug, then the database defined in app/etc/env.php is DROPPED and then the database defined in install-config-mysql.php is installed. This is probably not what you want and will most likely mess up your development node database.

So, two ways to run API Functional tests......

Run the API Functional Tests against Magento configured to use your development database.

• Leave your app/etc/env.php alone.
• No need to set anything in install-config-mysql.php. It is not used in this configuration.
• Set TESTS_CLEANUP to disabled in phpunit.xml
• Set TESTS_MAGENTO_INSTALLATION to disabled in phpunit.xml
• Set TESTS_BASE_URL, TESTS_WEBSERVICE_USER, TESTS_WEBSERVICE_APIKEY in phpunit.xml to match the credentials on your development node. TESTS_BASE_URL is the web address of your dev node installation.
• Run the API-Functional test. The tests will be run against your development node.
• Be aware it may pollute your dev database with test data.

Run the API Functional Tests against Magento configured to use a sandbox database.

• Prior to running the test. Update your app/etc/env.php to point to your sandbox database.
• Set TESTS_CLEANUP to enabled in phpunit.xml
• Set TESTS_MAGENTO_INSTALLATION to enabled in phpunit.xml
• Set TESTS_BASE_URL, TESTS_WEBSERVICE_USER, TESTS_WEBSERVICE_APIKEY in phpunit.xml to match the credentials as set in install-config-mysql.php. TESTS_BASE_URL is the web address of your dev node installation.
• Run the API-Functional test.
• It will uninstall the database defined in app/etc/env.php (Which should point to your sandbox right!).
• It will reinstall the database defined in install-config-mysql.php (the items in this file are directly transposed into command line arguments in the setup:install command so adjust to suit, but this should be the command to install magento into your sandbox database).
• Because you updated app/etc/env.php (your development node), the Magento HTTP(s) endpoint that is used in your test will act on your sandbox database.
• At the end of the test, revert the app/etc/env.php back to your original developer settings.

[gwharton]

Hi, @ihor-sviziev I wondered if you were aware of the issues with the API-Functional test framework as detailed in my post above. Can you copy anyone in, in the test team that may need to be aware of this. Its been in this broken/working only by chance state for a while now, and I am sure because of the bug, regularly wipes peoples development databases when they first try to run the intergration tests, and pollutes their development database with test data as the test framework uses the wrong database.

I raised #27213 on it a while ago but got no traction.

[baatractive]

@gwharton I see, thanks for the explanation. However, I did update etc/env.php to use a sandbox version but I don't really like this approach either. I would rather find an alternate solution to override the default database keys with the sandbox one. I hope to get it done somehow without going back & forth. Thanks :)

[gwharton]

you could have two env.php files and use a symbolic link to switch between them. It could be triggered automatically by a test runner script. Lets see if Magento core comes back with any comments. I agree its not ideal.

[m2-assistant]

Hi @engcom-Delta. Thank you for working on this issue.
In order to make sure that issue has enough information and ready for development, please read and check the following instruction: 👇

• 1. Verify that issue has all the required information. (Preconditions, Steps to reproduce, Expected result, Actual result).

  Details

  If the issue has a valid description, the label Issue: Format is valid will be added to the issue automatically. Please, edit issue description if needed, until label Issue: Format is valid appears.

• 2. Verify that issue has a meaningful description and provides enough information to reproduce the issue. If the report is valid, add Issue: Clear Description label to the issue by yourself.

• 3. Add Component: XXXXX label(s) to the ticket, indicating the components it may be related to.

• 4. Verify that the issue is reproducible on 2.4-develop branch

  Details

  - Add the comment @magento give me 2.4-develop instance to deploy test instance on Magento infrastructure.
  - If the issue is reproducible on 2.4-develop branch, please, add the label Reproduced on 2.4.x.
  - If the issue is not reproducible, add your comment that issue is not reproducible and close the issue and stop verification process here!

• 5. Add label Issue: Confirmed once verification is complete.

• 6. Make sure that automatic system confirms that report has been added to the backlog.

[m2-assistant]

Hi @engcom-Echo. Thank you for working on this issue.
In order to make sure that issue has enough information and ready for development, please read and check the following instruction: 👇

• 1. Verify that issue has all the required information. (Preconditions, Steps to reproduce, Expected result, Actual result).

  Details

  If the issue has a valid description, the label Issue: Format is valid will be added to the issue automatically. Please, edit issue description if needed, until label Issue: Format is valid appears.

• 2. Verify that issue has a meaningful description and provides enough information to reproduce the issue. If the report is valid, add Issue: Clear Description label to the issue by yourself.

• 3. Add Component: XXXXX label(s) to the ticket, indicating the components it may be related to.

• 4. Verify that the issue is reproducible on 2.4-develop branch

  Details

  - Add the comment @magento give me 2.4-develop instance to deploy test instance on Magento infrastructure.
  - If the issue is reproducible on 2.4-develop branch, please, add the label Reproduced on 2.4.x.
  - If the issue is not reproducible, add your comment that issue is not reproducible and close the issue and stop verification process here!

• 5. Add label Issue: Confirmed once verification is complete.

• 6. Make sure that automatic system confirms that report has been added to the backlog.

[m2-assistant]

Hi @engcom-Lima. Thank you for working on this issue.
In order to make sure that issue has enough information and ready for development, please read and check the following instruction: 👇

• 1. Verify that issue has all the required information. (Preconditions, Steps to reproduce, Expected result, Actual result).

  Details

  If the issue has a valid description, the label Issue: Format is valid will be added to the issue automatically. Please, edit issue description if needed, until label Issue: Format is valid appears.

• 2. Verify that issue has a meaningful description and provides enough information to reproduce the issue. If the report is valid, add Issue: Clear Description label to the issue by yourself.

• 3. Add Component: XXXXX label(s) to the ticket, indicating the components it may be related to.

• 4. Verify that the issue is reproducible on 2.4-develop branch

  Details

  - Add the comment @magento give me 2.4-develop instance to deploy test instance on Magento infrastructure.
  - If the issue is reproducible on 2.4-develop branch, please, add the label Reproduced on 2.4.x.
  - If the issue is not reproducible, add your comment that issue is not reproducible and close the issue and stop verification process here!

• 5. Add label Issue: Confirmed once verification is complete.

• 6. Make sure that automatic system confirms that report has been added to the backlog.

[engcom-Lima]

Hi @baatractive,

Thank you for reporting the issue.

I have gone through the issue description and related comments with @gwharton. I tried to replicate this issue on 2.4-develop instance with the provided steps but I was not able to reproduce it.

So for further analysis, can you please update on the below queries:

1. If this issue is happening with WSL specifically ?
2. Can you guys please check if this issue is reproducible on Magento 2.4-develop or is it specific to Magento 2.3.5 ?

It would be helpful if you guys can provide some more info so that I can try again to reproduce this.

[engcom-Lima]

Hi @baatractive,

We have noticed that this issue has not been updated for some time. Since we were not able to reproduce this so we are closing it. Please raise a fresh ticket or reopen this ticket if you need more assistance on this with answer to queries asked in #33696 (comment)

Regards