Get functional test suite working on GitHub Actions #945

colinmollenhour · 2020-05-04T22:06:04Z

Goal: get the Magento Test Framework tests working via GitHub Actions so we actually have a robust functional test suite.

colinmollenhour · 2020-05-04T23:59:54Z

Made a lot of progress but hit a wall:

https://travis-ci.org/github/OpenMage/magento-lts/jobs/683167264

PHP Fatal error: Declaration of Magento\Mtf\TestSuite\TestCase::run(?PHPUnit_Framework_TestResult $result = NULL) must be compatible with Magento\Mtf\TestSuite\TestSuite::run(?PHPUnit_Framework_TestResult $result = NULL, $filter = false, array $groups = Array, array $excludeGroups = Array, $processIsolation = false) in /home/travis/build/OpenMage/magento-lts/dev/tests/functional/vendor/magento/mtf/Magento/Mtf/TestSuite/TestCase.php on line 126
Fatal error: Declaration of Magento\Mtf\TestSuite\TestCase::run(?PHPUnit_Framework_TestResult $result = NULL) must be compatible with Magento\Mtf\TestSuite\TestSuite::run(?PHPUnit_Framework_TestResult $result = NULL, $filter = false, array $groups = Array, array $excludeGroups = Array, $processIsolation = false) in /home/travis/build/OpenMage/magento-lts/dev/tests/functional/vendor/magento/mtf/Magento/Mtf/TestSuite/TestCase.php on line 126

This code is not in our repo, though.. It is in magento/mtf.

colinmollenhour · 2020-05-05T17:00:16Z

Latest progress: https://travis-ci.org/github/OpenMage/magento-lts/builds/683476886

colinmollenhour · 2020-05-05T18:47:29Z

I got it to execute PHPUnit on PHP 7.2 with no errors but for some reason no tests were actually run..
https://travis-ci.org/github/OpenMage/magento-lts/jobs/683514414

I don't know anything about the MTF framework. Does anyone else have any clues as to how to get this to run?

colinmollenhour · 2020-05-05T21:42:00Z

Runs on PHP 7.2, 7.3 and 7.4, but times out with no details about what errors occurred...
https://travis-ci.org/github/OpenMage/magento-lts/jobs/683573694

I suppose this would need to be run locally to debug it further.. :(

tmotyl · 2020-05-18T20:22:03Z

for the record, here is the instruction for the old mtf framework for m1.
https://web.archive.org/web/20180303132243/http://devdocs.magento.com/guides/v2.0/mtf/mtf_quickstart/mtf_quickstart_config.html

If we find it useful we should fork it in openmage and adapt to suite our needs.

tmotyl · 2020-05-18T20:49:47Z

when I try to run

sh docker/run.sh

I'm getting

+ docker-compose -p mtf up -d
ERROR: 
        Can't find a suitable configuration file in this directory or any
        parent. Are you in the right directory?

        Supported filenames: docker-compose.yml, docker-compose.yaml

tmotyl · 2020-05-18T20:53:27Z

I've took the liberty to rebase the branch on top of current 1.9.4.x and add a commit with executable permissions for sh scripts

szymonbdeligo · 2020-06-18T14:49:49Z

.travis.yml

-  - '! find app/design -type f -name "*.phtml" -exec php -d error_reporting=32767 -l {} \; 2>&1 >&- | grep "^"'
+  - if [ $TEST_SUITE == "static" ]; then ! find . -not \( -path ./.phpstorm.meta.php -prune \) -not \( -path ./lib/PEAR -prune \) -not \( -path ./lib/phpseclib -prune \) -not \( -path ./lib/Zend -prune \) -type f -name "*.php" -exec php -d error_reporting=32767 -l {} \; 2>&1 >&- | grep "^"; fi
+  - if [ $TEST_SUITE == "static" ]; then ! find app/design -type f -name "*.phtml" -exec php -d error_reporting=32767 -l {} \; 2>&1 >&- | grep "^"; fi
+  - if [ $TEST_SUITE == "functional" ]; then dev/tests/functional/vendor/phpunit/phpunit/phpunit -c dev/tests/$TEST_SUITE; fi


Functional tests takes more than 20 mintes and you have to add travis_wait 30 otherwise task will fail with error: No output has been received in the last 10m0s, this potentially indicates a stalled build or something wrong with the build itself.

My sugesstion:
- if [ $TEST_SUITE == "functional" ]; then travis_wait 30 dev/tests/functional/vendor/phpunit/phpunit/phpunit -c dev/tests/$TEST_SUITE; fi

Thanks @szymonbdeligo I'll give it a go!

colinmollenhour · 2020-06-24T18:39:03Z

Tests are still failing, I think it is unable to login for some reason. I changed the password to match but may have missed something...

https://travis-ci.org/github/OpenMage/magento-lts/jobs/700745572

fballiano · 2022-09-29T17:03:44Z

it's been more than 2 years without movement on this PR, I'll close it but can always be reopened in case somebody want to continue it.

github-actions · 2023-01-10T22:12:27Z

Unit Test Results

7 tests 7 ✔️ 0s ⏱️
1 suites 0 💤
1 files 0 ❌

Results for commit b5df99e.

♻️ This comment has been updated with latest results.

sreichel · 2023-01-11T04:04:23Z

@colinmollenhour follow your commits ... absolutly awesome ❤️

colinmollenhour · 2023-01-11T06:11:41Z

Sooo close... https://github.com/OpenMage/magento-lts/actions/runs/3890167666/jobs/6639043916#step:16:18

EDIT: The Apache test requests are failing but do not trigger an error so I think that is actually the issue.. I'm not familiar with mod_proxy_fcgi so probably did something wrong there.

colinmollenhour · 2023-01-11T06:27:49Z

Sadly I must give up on this for a while.. If anyone wants to help troubleshoot, please be my guest!

Apache is responding with 403/503 so there is some configuration issue. This probably fully explains the "Admin user cannot be logged in by curl handler!" but that error should be replaced with something more useful.
There are some deleted files from merging latest 1.9.4.x - I don't know why they were deleted or if they should have been?
It is running on PHP 7.3 currently - 8.1.14 is broken woth timezone issue - also probably upgrading PHPUnit will break some stuff and the MTF code may need to be updated to fix code broken by 7.4, 8.0, 8.1
No idea if the functional tests will still pass (but they should?), just trying to get them "running" again.. the code appears to have reached the point where it logs into the admin panel so it would seem that it is working to some degree although I don' tknow if this is actually talking to Selenium yet.

elidrissidev · 2023-01-11T10:51:14Z

Invading to try something 🕵️

elidrissidev · 2023-01-11T10:58:45Z

I think we have some progress. 503 error is substituted by 403 when requesting http://localhost/index.php

sreichel · 2023-01-11T15:53:34Z

I've no time at the moment ... can you try http://127.0.0.1/ instead?

@szymonbdeligo

commit 974fb37 Author: Mohamed ELIDRISSI <[email protected]> Date: Fri Mar 10 10:15:57 2023 +0100 Remove `chroot` and `chdir` completely commit ded6e16 Author: Mohamed ELIDRISSI <[email protected]> Date: Fri Mar 10 10:13:08 2023 +0100 Comment out `chroot` and `chdir` commit 6fd3aba Author: Sven Reichel <[email protected]> Date: Sun Jan 15 02:39:27 2023 +0100 Test No3 commit 943f5c2 Author: Sven Reichel <[email protected]> Date: Sun Jan 15 02:19:36 2023 +0100 Test No2 commit acf73d7 Author: Sven Reichel <[email protected]> Date: Sun Jan 15 01:53:48 2023 +0100 Test commit f77300c Author: Mohamed ELIDRISSI <[email protected]> Date: Wed Jan 11 11:50:43 2023 +0100 Remove ProxyPassMatch from apache site Should fallback to php7.3-fpm.conf since it's enabled AFAIK commit d633e04 Author: Colin Mollenhour <[email protected]> Date: Wed Jan 11 01:07:57 2023 -0500 Troubleshooting Selenium/Apache commit 59d6343 Author: Colin Mollenhour <[email protected]> Date: Wed Jan 11 01:02:03 2023 -0500 Troubleshooting Selenium/Apache commit df8f211 Author: Colin Mollenhour <[email protected]> Date: Tue Jan 10 18:27:56 2023 -0500 Migrate MTF from Travis to Github Actions. First attempt. commit 46e5c36 Merge: b0acd23 bd0cb12 Author: Colin Mollenhour <[email protected]> Date: Tue Jan 10 16:55:59 2023 -0500 Merge remote-tracking branch 'openmage/1.9.4.x' into functional-test-suite commit b0acd23 Author: akrzemianowski <[email protected]> Date: Mon Jun 29 16:11:44 2020 +0200 Fixed issue with admin login after forced password rehash (#1071) commit 24e1d5c Author: szymonbdeligo <[email protected]> Date: Thu Jun 25 00:40:51 2020 +0200 Fix static tests forPHP 7.4 (#1058) commit 08c79ad Author: Colin Mollenhour <[email protected]> Date: Mon Jun 22 00:57:39 2020 -0400 Fix install command to use same password as mtf. commit 58ff3c6 Author: Colin Mollenhour <[email protected]> Date: Sun Jun 21 20:44:13 2020 -0400 Add travis timeout for 30 minutes as per @szymonbdeligo. commit e544ea9 Author: Tymoteusz Motylewski <[email protected]> Date: Mon May 18 22:49:23 2020 +0200 Make script files executable commit 17a4131 Author: Colin Mollenhour <[email protected]> Date: Tue May 5 18:14:12 2020 -0400 Add script for running MTF locally via Docker. commit 8219636 Author: Colin Mollenhour <[email protected]> Date: Tue May 5 17:03:45 2020 -0400 Remove DownloaderPart2Test and DownloaderTest as they do not run and we do not support downloader. commit 4620035 Author: Colin Mollenhour <[email protected]> Date: Tue May 5 15:17:47 2020 -0400 Fix Composer install for PHP 7.4 commit 2aed58c Author: Colin Mollenhour <[email protected]> Date: Tue May 5 15:13:59 2020 -0400 Generate tests and remove filter. commit 8ff94e7 Author: Colin Mollenhour <[email protected]> Date: Tue May 5 14:35:23 2020 -0400 Try to reconcile magento/mtf updates and PHPUnit updates. commit 6eee8a0 Author: Colin Mollenhour <[email protected]> Date: Tue May 5 12:56:59 2020 -0400 Disable `mtf troubleshooting:check-all` command as it appears to only fork for Magento 2 commit 50b837e Author: Colin Mollenhour <[email protected]> Date: Mon May 4 20:13:41 2020 -0400 Try updating PHPUnit dependencies. May require PHPUnit 8. commit 2deb37d Author: Colin Mollenhour <[email protected]> Date: Mon May 4 20:02:20 2020 -0400 Trying Magento MTF rc64. commit 6ac2ed3 Author: Colin Mollenhour <[email protected]> Date: Mon May 4 19:12:08 2020 -0400 Add back troubleshooting:check-all step. commit 5095481 Author: Colin Mollenhour <[email protected]> Date: Mon May 4 19:08:10 2020 -0400 Add missing travis_acceptance.xml commit a0f4b10 Author: Colin Mollenhour <[email protected]> Date: Mon May 4 18:51:56 2020 -0400 Fix before_script.sh for functional test. Skip composer install for static test. commit b9134fe Author: Colin Mollenhour <[email protected]> Date: Mon May 4 18:37:24 2020 -0400 Fix invalid password for admin, remove unneeded static test steps. commit 01e04da Author: Colin Mollenhour <[email protected]> Date: Mon May 4 18:23:58 2020 -0400 Fix Magento install. commit 30564cc Author: Colin Mollenhour <[email protected]> Date: Mon May 4 18:09:32 2020 -0400 Shell scriupts must be executable. commit c57d92c Author: Colin Mollenhour <[email protected]> Date: Mon May 4 18:04:19 2020 -0400 First attempt to get functional tests working using magento2 travis files. Conflicts: dev/tests/functional/lib/Magento/Mtf/App/State/AbstractState.php dev/tests/functional/lib/Magento/Mtf/App/State/State1.php dev/tests/functional/lib/Magento/Mtf/Client/Driver/Selenium/Driver.php dev/tests/functional/lib/Magento/Mtf/Client/Element/SimpleElement.php dev/tests/functional/lib/Magento/Mtf/Util/Generate/Page.php

colinmollenhour · 2023-04-04T19:25:58Z

Rebased onto main. These files were deleted in upstream, I am not sure why or if it is ok?

Unmerged paths:
  (use "git restore --staged <file>..." to unstage)
  (use "git add/rm <file>..." as appropriate to mark resolution)
        deleted by them: dev/tests/functional/lib/Magento/Mtf/App/State/AbstractState.php
        deleted by them: dev/tests/functional/lib/Magento/Mtf/App/State/State1.php
        deleted by them: dev/tests/functional/lib/Magento/Mtf/Client/Driver/Selenium/Driver.php
        deleted by them: dev/tests/functional/lib/Magento/Mtf/Client/Element/SimpleElement.php
        deleted by them: dev/tests/functional/lib/Magento/Mtf/Util/Generate/Page.php

fballiano · 2024-02-21T23:03:44Z

question: does it still make sense in 2024 to keep this open?

Flyingmana · 2024-02-21T23:58:49Z

getting the functional testsuite to work again would at least increase our coverage a lot lot.
The alternative would be to build up or own for all these cases

colinmollenhour · 2024-02-24T01:17:45Z

I'm on the fence.. It would be great to leverage the existing work but at the same time it seems kinda ancient and klunky.. I'd much rather use something like Cypress (I'm sure there are many other good choices) but we'd be starting from scratch. As far as my existing work on this PR, if it was for nothing that is ok - no sunk cost bias here.

I may know a company that would be willing to sponsor the development of a modern test suite if it could be done fairly fast (weeks or months, not years). Is anyone interested in that opportunity?

mattdavenport · 2024-08-21T19:47:54Z

Any objections to closing this PR?

sreichel · 2024-08-22T03:29:39Z

Cypress lgtm but its not free. What alternatives we have? Selenium? ...?

mattdavenport · 2024-08-23T19:29:44Z

Cypress has a free MIT-licensed version. I think it's when you get into parallel pipeline running and UI features that it's paid. There are also some ways around this as well, but the open source version has been sufficient for us.

colinmollenhour · 2024-08-23T19:35:37Z

There is also Playwright. From what I've seen, Cypress and Playwright are the best choices in my opinion. The free version of Cypress is definitely sufficient for our purposes.

sreichel · 2024-09-10T02:57:30Z

I may know a company that would be willing to sponsor the development of a modern test suite if it could be done fairly fast (weeks or months, not years). Is anyone interested in that opportunity?

@colinmollenhour, I'd go for it.

Not weeks, not years, but months ...

colinmollenhour changed the title ~~WIP: Travis and MFT~~ Get functional test suite working on Travis May 4, 2020

sreichel added the environment label May 11, 2020

tmotyl force-pushed the functional-test-suite branch from 7c35cee to e544ea9 Compare May 18, 2020 20:52

szymonbdeligo reviewed Jun 18, 2020

View reviewed changes

github-actions bot added Component: Admin Relates to Mage_Admin Component: Eav Relates to Mage_Eav labels Jul 24, 2020

fballiano closed this Sep 29, 2022

colinmollenhour reopened this Jan 10, 2023

github-actions bot removed Component: Eav Relates to Mage_Eav Component: Admin Relates to Mage_Admin labels Jan 10, 2023

colinmollenhour changed the title ~~Get functional test suite working on Travis~~ Get functional test suite working on GitHub Actions Jan 10, 2023

This comment has been minimized.

Sign in to view

colinmollenhour force-pushed the functional-test-suite branch from c8c9cbd to df8f211 Compare January 11, 2023 05:48

colinmollenhour added the help wanted label Jan 11, 2023

github-actions bot added Component: Core Relates to Mage_Core Component: Adminhtml Relates to Mage_Adminhtml Template : admin Relates to admin template labels Jan 15, 2023

colinmollenhour force-pushed the functional-test-suite branch from 974fb37 to b5df99e Compare April 4, 2023 19:24

colinmollenhour changed the base branch from 1.9.4.x to main April 4, 2023 19:24

colinmollenhour added the Rebased: RFC-0002 This PR has been rebased onto either 'main' or 'next' according to RFC-0002 label Apr 4, 2023

sreichel mentioned this pull request Aug 22, 2024

Removed MTF #4161

Merged

colinmollenhour closed this Aug 23, 2024

colinmollenhour deleted the functional-test-suite branch September 9, 2024 13:16

sreichel mentioned this pull request Sep 27, 2024

Removed docker setup #4166

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Get functional test suite working on GitHub Actions #945

Get functional test suite working on GitHub Actions #945

colinmollenhour commented May 4, 2020 •

edited

Loading

colinmollenhour commented May 4, 2020 •

edited

Loading

colinmollenhour commented May 5, 2020

colinmollenhour commented May 5, 2020

colinmollenhour commented May 5, 2020

tmotyl commented May 18, 2020

tmotyl commented May 18, 2020

tmotyl commented May 18, 2020

szymonbdeligo Jun 18, 2020

colinmollenhour Jun 22, 2020

colinmollenhour commented Jun 24, 2020

fballiano commented Sep 29, 2022

This comment has been minimized.

github-actions bot commented Jan 10, 2023 •

edited

Loading

sreichel commented Jan 11, 2023 •

edited

Loading

colinmollenhour commented Jan 11, 2023 •

edited

Loading

colinmollenhour commented Jan 11, 2023

elidrissidev commented Jan 11, 2023

elidrissidev commented Jan 11, 2023

sreichel commented Jan 11, 2023

colinmollenhour commented Apr 4, 2023

fballiano commented Feb 21, 2024

Flyingmana commented Feb 21, 2024

colinmollenhour commented Feb 24, 2024

mattdavenport commented Aug 21, 2024

sreichel commented Aug 22, 2024

mattdavenport commented Aug 23, 2024

colinmollenhour commented Aug 23, 2024

sreichel commented Sep 10, 2024

Get functional test suite working on GitHub Actions #945

Get functional test suite working on GitHub Actions #945

Conversation

colinmollenhour commented May 4, 2020 • edited Loading

colinmollenhour commented May 4, 2020 • edited Loading

colinmollenhour commented May 5, 2020

colinmollenhour commented May 5, 2020

colinmollenhour commented May 5, 2020

tmotyl commented May 18, 2020

tmotyl commented May 18, 2020

tmotyl commented May 18, 2020

szymonbdeligo Jun 18, 2020

Choose a reason for hiding this comment

colinmollenhour Jun 22, 2020

Choose a reason for hiding this comment

colinmollenhour commented Jun 24, 2020

fballiano commented Sep 29, 2022

This comment has been minimized.

github-actions bot commented Jan 10, 2023 • edited Loading

Unit Test Results

sreichel commented Jan 11, 2023 • edited Loading

colinmollenhour commented Jan 11, 2023 • edited Loading

colinmollenhour commented Jan 11, 2023

elidrissidev commented Jan 11, 2023

elidrissidev commented Jan 11, 2023

sreichel commented Jan 11, 2023

colinmollenhour commented Apr 4, 2023

fballiano commented Feb 21, 2024

Flyingmana commented Feb 21, 2024

colinmollenhour commented Feb 24, 2024

mattdavenport commented Aug 21, 2024

sreichel commented Aug 22, 2024

mattdavenport commented Aug 23, 2024

colinmollenhour commented Aug 23, 2024

sreichel commented Sep 10, 2024

colinmollenhour commented May 4, 2020 •

edited

Loading

colinmollenhour commented May 4, 2020 •

edited

Loading

github-actions bot commented Jan 10, 2023 •

edited

Loading

sreichel commented Jan 11, 2023 •

edited

Loading

colinmollenhour commented Jan 11, 2023 •

edited

Loading