pbTests: Use 'homemade' Solaris box #2405

Willsparker · 2021-12-07T16:08:35Z

Ref: https://adoptium.slack.com/archives/C53GHCXL4/p1637309425055700

The Solaris box we were using didn't have any of the X11 libraries that is required to build a JDK. Therefore, with the help of @sxa and @gdams , I've made a new vagrant box; which I have installed to every vagrant* user on the VPC machine. This box has a lot more libraries located in /usr/openwin/, so, fingers-crossed it means we can actually build a JDK on there!

Creating this box has also allowed for it to be setup how we want it. Therefore, the compiler is pre-installed on the box so we don't have to do that whenever creating the VM, we have a newer version of SSH on there, so we don't have to change the ssh KexAlgorithm, and the VBox Guest Additions are up to date.

The .box file that was used to add the box to the machine, is located on the infra-ibmcloud-vagrant-x64-1 machine, at /home/will/solaris10_homemade.box, in case any additional config needs to be put in. The PR is in draft as I want to get some working JDK builds before merging it :-)

Please note; I've taken the adoptopenjdkSol10.vm.hostname argument out of the Vagrantfile, as this was causing the /etc/hostname file to be deleted. Deleting this file causes the network interface configuration command to fail, due to sudo's unable to resolve host issue. The hostname that's been set on the box is adoptSolaris10, and I haven't seen any additional errors come up from removing the argument.

Willsparker · 2021-12-14T14:26:08Z

As an update, as I found whilst looking at #2411 , the new box does have /usr/xpg4/bin, it's just not in the path. I've updated the vagrant boxes on the VPC machine to have /usr/xpg4/bin at the front of the path (by exporting PATH in /etc/profile). This was done as it most accurately reflects the older box.

Willsparker · 2021-12-14T15:56:56Z

FYI, the latest commit will increase the timeout for the vagrant box. By default the timeout is 5 minutes, however apparently it is taking a bit longer on the VPC machine. If it still doesn't boot after 10 minutes, something is probably very wrong. Latest run: VPC-1352

…0_homemade

Willsparker · 2021-12-16T08:58:45Z

Okay, I've managed to get a JDK build in VPC-1355 . This managed to successfully build JDK8/HS ; however, the tests weren't able to run, I believe due to the problem that is mentioned in #2210 . I also noticed that in the actual tests, were grep error messages:

19:55:23 /export/home/local/vagrant/testLocation/aqa-tests/TKG/scripts/getTestenvProperties.sh --repo_dir /export/home/local/vagrant/testLocation/aqa-tests/openj9 --output_file /export/home/local/vagrant/testLocation/aqa-tests/testenv/testenv.properties --repo_name OPENJ9
19:55:23 Check sha in /export/home/local/vagrant/testLocation/aqa-tests/openj9 and store the info in /export/home/local/vagrant/testLocation/aqa-tests/testenv/testenv.properties
19:55:23 grep: illegal option -- P
19:55:23 grep: illegal option -- o
19:55:23 Usage: grep -hblcnsviw pattern file . . .

Also, I'm not 100% certain the long boot times fo rthe VM are fixed. I was unable to recreate the issue on my machine, nor directly using the VPC machine, and it seems the boot time varies wildly, wherein it took >10 minutes to boot in VPC-1353, and 3 minutes to boot in VPC-1355

Willsparker · 2021-12-22T11:22:37Z

Testing running the tests, locally, I've determined a few things: we need to install ant, as that isn't in the playbooks for Solaris. However, having done that, I'm getting this problem when running the ant -f scripts/build-tools.xml ...etc command:

Buildfile: /export/home/local/vagrant/testLocation/aqa-tests/TKG/scripts/build_tools.xml
  [taskdef] Could not load definitions from resource net/sf/antcontrib/antlib.xml. It could not be found.

Googling has told me that this means ant-contrib.jar is not in the classpath. The lib is in /usr/local/apache-ant-1.10.5/lib/ant-contrib.jar, but apparently that isn't picked up.

In addition, some kind of perl Digest SHA package needs to be installed, according to this error:

getDependentLibs:
     [exec] Can't locate Digest/SHA.pm in @INC (@INC contains: /usr/perl5/5.8.4/lib/i86pc-solaris-64int /usr/perl5/5.8.4/lib /usr/perl5/site_perl/5.8.4/i86pc-solaris-64int /usr/perl5/site_perl/5.8.4 /usr/perl5/site_perl /usr/perl5/vendor_perl/5.8.4/i86pc-solaris-64int /usr/perl5/vendor_perl/5.8.4 /usr/perl5/vendor_perl .) at /export/home/local/vagrant/testLocation/aqa-tests/TKG/../TKG/scripts/getDependencies.pl line 19.
     [exec] BEGIN failed--compilation aborted at /export/home/local/vagrant/testLocation/aqa-tests/TKG/../TKG/scripts/getDependencies.pl line 19.

On something like RHEL, it'd be perl-Digest-SHA.x86_64, presumably there's a Solaris equivalent.

For reference, the perl version that's being used is 5.8.4, however pkguitl -i perl shows CSWperl-5.10.1 is installed. May have better luck using the newer one?

EDIT:
Yes! The newer one worked. I had to add /opt/csw/bin/ to the front of the path however. Nice. I'll have to put a special case in for Solaris into testJDK.sh.

Willsparker · 2021-12-22T11:37:54Z

Carrying on from that, it seems we need to install a newer version of ant:

BUILD SUCCESSFUL
Total time: 10 seconds
0
/export/home/local/vagrant/openjdk-build/workspace/build/src/build/solaris-x86_64-normal-server-release/images/jdk8u322-b05/bin/java -cp ./bin/TestKitGen.jar org.openj9.envInfo.EnvDetector
System.getProperty('java.vm.name')=OpenJDK 64-Bit Server VM

System.getProperty('java.vendor')=Temurin

Warning: cannot determine vendor, use System.getProperty('java.vendor')=Temurin directly.

System.getProperty('os.name')=SunOS

System.getProperty('os.arch')=amd64

System.getProperty('java.fullversion')=null

System.getProperty('sun.arch.data.model')=64

Error: required antVersion: 1.9.6. Installed version: 1.8.4
make: *** [makefile:73: envDetect] Error 1

The test scripts do carry on, beyond that point, however it skips the test:

===============================================
Running test MBCS_Tests_pref_ja_JP_linux_0 ...
===============================================
MBCS_Tests_pref_ja_JP_linux_0 Start Time: Wed Dec 22 11:35:29 2021 Epoch Time (ms): 1640172929744
Skipped due to jvm options ( ) and/or platform requirements (os.linux) => MBCS_Tests_pref_ja_JP_linux_0_SKIPPED
MBCS_Tests_pref_ja_JP_linux_0 Finish Time: Wed Dec 22 11:35:29 2021 Epoch Time (ms): 1640172929762
make[5]: Leaving directory '/export/home/local/vagrant/testLocation/aqa-tests/functional/MBCS_Tests/pref'
make[4]: Leaving directory '/export/home/local/vagrant/testLocation/aqa-tests/functional/MBCS_Tests'
make[3]: Leaving directory '/export/home/local/vagrant/testLocation/aqa-tests/functional'
make[2]: Leaving directory '/export/home/local/vagrant/testLocation/aqa-tests'
cd "/export/home/local/vagrant/testLocation/aqa-tests/TKG/../TKG/scripts"; \
perl "resultsSum.pl" --failuremk="/export/home/local/vagrant/testLocation/aqa-tests/TKG/../TKG/failedtargets.mk" --resultFile="/export/home/local/vagrant/testLocation/aqa-tests/TKG/../TKG/output_16401729292614/TestTargetResult" --platFile="/export/home/local/vagrant/testLocation/aqa-tests/TKG/../TKG/resources/buildPlatformMap.properties" --diagnostic=failure --jdkVersion=8 --jdkImpl=hotspot --jdkVendor="Temurin" --spec=sunos_x86-64 --buildList=functional --customTarget= --testTarget=MBCS_Tests_pref_ja_JP_linux_0 --tapPath=/export/home/local/vagrant/testLocation/aqa-tests/TKG/../TKG/output_16401729292614/ --tapName=


TEST TARGETS SUMMARY
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
TOTAL: 1   EXECUTED: 0   PASSED: 0   FAILED: 0   DISABLED: 0   SKIPPED: 1
ALL TESTS PASSED
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

…0_homemade

sxa · 2021-12-24T14:17:08Z

Hmmm I would have thought we'd be pulling ant 1.10.5 into /usr/local/ like we do elsewhere ... Unless we've got two versions installed and it's picking up the wrong one by default - which seems the most likely option and would explain why it didn't find ant-contrib properly too.

Willsparker · 2022-01-04T09:56:24Z

Ah yes, I've found it; you were right. Installing it via PKGUtil just puts it in /opt/csw/bin. Alright, so I need to put /usr/local/bin into the path on Solaris as it isn't there at all. I'm going to put this in as another 'special case' in testJDK.sh; the alternative is to put a task somewhere that adds /usr/local/bin into the path for Solaris, however I wouldn't want to change a person's environment that much, without putting the adoptopenjdk/jenkins tag on the task - which VPC skips anyway.

Willsparker · 2022-01-04T11:26:43Z

Testing that locally, (i.e. adding to the path /usr/local/bin), It seemed to work well, however it still skipped the test after having compiled all of of them:

===============================================
Running test MBCS_Tests_pref_ja_JP_linux_0 ...
===============================================
MBCS_Tests_pref_ja_JP_linux_0 Start Time: Tue Jan  4 11:20:37 2022 Epoch Time (ms): 1641295237474
Skipped due to jvm options ( ) and/or platform requirements (os.linux) => MBCS_Tests_pref_ja_JP_linux_0_SKIPPED
MBCS_Tests_pref_ja_JP_linux_0 Finish Time: Tue Jan  4 11:20:37 2022 Epoch Time (ms): 1641295237499
make[5]: Leaving directory '/export/home/local/vagrant/testLocation/aqa-tests/functional/MBCS_Tests/pref'

...

TEST TARGETS SUMMARY
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
TOTAL: 1   EXECUTED: 0   PASSED: 0   FAILED: 0   DISABLED: 0   SKIPPED: 1
ALL TESTS PASSED
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

AFAIK, no special options have been put on the JVM, so I guess the test is skipping due to the platform. There doesn't appear to be any Solaris / SunOS mentioned in the /export/home/local/vagrant/testLocation/aqa-tests/functional/MBCS_Tests/pref, so I assume there isn't a Solaris equivalent.

Willsparker · 2022-01-05T11:03:24Z

Ref: The latest committ. As said by @sophia-guo , Solaris can't run the MBCS_Tests_pref_ja_JP_linux_0 test. (See: Slack comment) . Instead I have made the JDK just run the _jdk_math test. While this isn't ideal as it doesn't test the perl requirements, it does test the JDK in some regard, and only took ~4 minutes locally, so it should suffice. (Though if anybody can think of any better tests, it can be changed).

Theoretically, with this, the VPC Solaris run should now work - VPC Run: VPC-1632.

Willsparker · 2022-01-06T09:04:36Z

It worked (!!!). Ready for review.

sxa

Nothing too controversial in here - just the result of a lot of debugging to figure out why things were failing :-)

I'd suggest we want some docs on the process of setting up the Vagrant machine somewhere too, but other than one quoting suggestion I'm ok for this to go in :-)

ansible/pbTestScripts/testJDK.sh

sxa

Let's go back to a simpler test instead of the whole jdk_math suite

ansible/pbTestScripts/testJDK.sh

sxa

LGTM

Willsparker added 2 commits December 7, 2021 15:52

pbTests: Add in homemade Solaris Vagrant box

6a85c8d

Remove old Vagrantfile

3b4ada1

Willsparker self-assigned this Dec 7, 2021

github-actions bot added pbTests Vagrant labels Dec 7, 2021

Willsparker mentioned this pull request Dec 14, 2021

pbTests: Replace gsub in awk command #2411

Merged

5 tasks

Increase boot timeout to Sol10 VF

fec3e1c

Willsparker added 2 commits December 15, 2021 13:43

Merge branch 'master' of github.com:adoptium/infrastructure into Sol1…

7ea10ba

…0_homemade

vagrant: increase Sol10 VF to use 5GB Mem

e27ba89

Willsparker added 2 commits December 22, 2021 11:40

Put in special case for testJDK.sh

d14bad1

Merge branch 'master' of github.com:adoptium/infrastructure into Sol1…

30c024b

…0_homemade

testJDK: Run a different test for Solaris

0d9dca7

Willsparker marked this pull request as ready for review January 6, 2022 09:04

Willsparker requested review from gdams, Haroon-Khel and sxa January 6, 2022 09:04

sxa approved these changes Jan 6, 2022

View reviewed changes

ansible/pbTestScripts/testJDK.sh Outdated Show resolved Hide resolved

sxa requested changes Jan 6, 2022

View reviewed changes

ansible/pbTestScripts/testJDK.sh Outdated Show resolved Hide resolved

Revert to using MachineInfo test

712ae49

Willsparker force-pushed the Sol10_homemade branch from a84ba1d to 712ae49 Compare January 6, 2022 12:36

sxa approved these changes Jan 6, 2022

View reviewed changes

Willsparker merged commit 7690e5b into adoptium:master Jan 6, 2022

This was referenced Jan 10, 2022

unixPB: Install gcc 10.3 into /usr/local/gcc10 #2420

Merged

GHActions: Solaris10 is failing #2425

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pbTests: Use 'homemade' Solaris box #2405

pbTests: Use 'homemade' Solaris box #2405

Willsparker commented Dec 7, 2021

Willsparker commented Dec 14, 2021

Willsparker commented Dec 14, 2021

Willsparker commented Dec 16, 2021

Willsparker commented Dec 22, 2021 •

edited

Loading

Willsparker commented Dec 22, 2021

sxa commented Dec 24, 2021 •

edited

Loading

Willsparker commented Jan 4, 2022

Willsparker commented Jan 4, 2022 •

edited

Loading

Willsparker commented Jan 5, 2022 •

edited

Loading

Willsparker commented Jan 6, 2022

sxa left a comment

sxa left a comment

sxa left a comment

pbTests: Use 'homemade' Solaris box #2405

pbTests: Use 'homemade' Solaris box #2405

Conversation

Willsparker commented Dec 7, 2021

Willsparker commented Dec 14, 2021

Willsparker commented Dec 14, 2021

Willsparker commented Dec 16, 2021

Willsparker commented Dec 22, 2021 • edited Loading

Willsparker commented Dec 22, 2021

sxa commented Dec 24, 2021 • edited Loading

Willsparker commented Jan 4, 2022

Willsparker commented Jan 4, 2022 • edited Loading

Willsparker commented Jan 5, 2022 • edited Loading

Willsparker commented Jan 6, 2022

sxa left a comment

Choose a reason for hiding this comment

sxa left a comment

Choose a reason for hiding this comment

sxa left a comment

Choose a reason for hiding this comment

Willsparker commented Dec 22, 2021 •

edited

Loading

sxa commented Dec 24, 2021 •

edited

Loading

Willsparker commented Jan 4, 2022 •

edited

Loading

Willsparker commented Jan 5, 2022 •

edited

Loading