Feature Test Summary: MicroProfile Reactive Streams Operators 1.0

[Emily-Jiang]

**1) Describe the test strategy & approach for this feature, and describe how the approach verifies the functions delivered by this feature.

All testing is automated and fully integrated with the Simplicity Test Infrastructure for reporting
anything less than 100% pass:

Testing occurs in 4 layers of automated test suites:

Unit Testing:

Unit tests are stored in
com.ibm.ws.microprofile.reactive.streams.operators/test/src/com/ibm/ws/microprofile/reactive/streams/test
and are run automatically as part of building the implementation bundle or by running

./gradlew com.ibm.ws.microprofile.reactive.streams.operators:test

There are currently 8 unit test classes with 25 tests that have been peer reviewed
at https://github.com/OpenLiberty/open-liberty/pull/6617#pullrequestreview-206221908
as a valid UT suite at. The purpose of this suite is to check a basic implementation and
integration are operational.

The Reactive Streams .Org TCK

This is a highly mature, comprehensive and stable test suite which lives at:
https://github.com/reactive-streams/reactive-streams-jvm/tree/master/tck
There are 251 automated tests that provide a thorough verification that the implementation
is compliant with the Reactive Streams TCK.

The Reactive Streams .Org TCK is run as part of executing:

./gradlew com.ibm.ws.microprofile.reactive.streams.operators_fat_tck:buildandrun

This test suite is run automatically as part of the main build verification process.

The MicroProfile Reactive Streams Operators TCK

This is a cross vendor TCK that has multiple authors and is reviewed as being
a comprehensive verification suite for the specification. It is stored at:
[https://github.com/eclipse/microprofile-reactive-streams-operators/tree/master/tck]
(https://github.com/eclipse/microprofile-reactive-streams-operators/tree/master/tck)
It contains 339 testcases (not including the embedded Reactive Streams .Org TCK)

The MicroProfile Reactive Streams Operators TCK is run by executing:

./gradlew com.ibm.ws.microprofile.reactive.streams.operators_fat_tck:buildandrun

The WAS mpReactiveStreams Feature FAT bucket

The MicroProfile Reactive Streams Operators IBM implementation in WAS
has an IBM specific FAT test bucket. This bucket aims to test the integration
of the base implementation in the Liberty environment together with some
basic testing of the function in the context of a Web Application (Servlet/CDI etc).
There are over 500 tests in the automated TCK test suites, so this test bucket
tries to concentrate on inter-component and contextual tests.

ReactiveJaxRSTest

This test reactive stream usage in the context of a JAX-RS application with
Publisher to Subscriber linking and the use of the default reactive streams
engine.

ContextPropogationTest

This test tests that the callback elements (Publishers/Processors/Subscribers and
operators' callbacks and lambdas) are all run with contexts that are as expected
according to the MicroProfile Concurrently Specification.

Tests using the stream completion CompletableFuture

PostStreamCompletionCapturedThreadContextCleared
PostStreamCompletionCapturedThreadContextProgated
PostStreamCompletionCapturedThreadContextUnchanged

Tests using a Default Engine for CDI Propogation

Contexts tested are:

• @ApplicationScoped
• @SessionScoped
• @ConversationScoped
• @RequestScoped

We test that the correct (different/same) bean is injectable at different stages of
the pipeline.

ConcurrentWorkTest

We test that it is possible to run stream methods with appropiate concurrency

StreamContextViaThreadContextProviderSPI

This explores using the MicroProfile Concurrency SPI to provide some interesting
custom 'stream' contexts down the stream alongside application data (for example
for tracing or workflow identifier).

Error Messages

Explore rxjava, smallrye and IBM messages are all OK.

Running the FAT bucket

The FAT bucket runs by default with each build and
is also integrated into the standard Simplicity framework and can be run with

./gradlew com.ibm.ws.microprofile.reactive.streams.operators_fat:buildandrun

SVT Testing

The SVT Test Team (Brian P. Hanczaryk et al @hanczaryk ) are already testing this feature in their stress and load tests, for example 10 million server requests @ 4K per second to raise this defect:

https://github.ibm.com/was-liberty/WS-CD-Open/issues/15521

SVT Objective: Test a Liberty Collective topology leveraging mpReactiveStreams-1.0 with the ProShop application under load for 72 consecutive hours

##Dependencies
VM needs
VM #1-2 (controller/liberty server) : (2 CPU / 4 GB RAM)
VM #3 (SQLServer 2016) : (4 CPU / 16 GB RAM)
VM #4-5 (RPT/jmeter) : (2 CPU / 4 GB RAM)

##Scenario test files

ProShop app updated during Test Case SVT: APSC_DeveloperExperience_mpReactiveStreams1.0_ProShop_Tooling (#15591)

WebSphere documentation
The documentation listed should be verified throughout the testcase execution with a final verification in section 3.

##WebSphere topology
VM #1-2 (controller/liberty server) : (2 CPU / 4 GB RAM)
VM #3 (SQLServer 2016) : (4 CPU / 16 GB RAM)
VM #4-5 (RPT/jibe) : (2 CPU / 4 GB RAM)

##Installation/Configuration Instructions
Machine #1-2

Install lWAS on machine
Create controller
a) ./server create myController
b) ./collective create myController --keystorePassowrd=<>
c) copy command window output into server.xml as instructed
d) ./server start myController
Create liberty servers onVMs #2
a) ./server create myMemberX
Join liberty servers to the collective
a) ./collective join myMemberX --host=<> --port=<> --user=<> --password=<> --keystorePassword=<>
Edit values for each myMemberX's server.xml with the result from collective join , unique port info, and cluster info.

Copy ProShop.war into app directory for each member.
Restore/Create ProShop SQL Server 2016 for use with ProShop app stress
Ensure that ProShop is functional.
Execution Instructions

Setup RPT/JMeter/Jibe on VM #4 & #5 if not already done.
Start RPT and capture test recording for ProShop at http://<ihs_host>:80/ProShop/
(i.e. for mpReactiveStreams-1.0 test, hit http://<ihs_host>:80/ProShop/ReactiveStreamsTestServlet)
Create a performance test schedule to drive app load and start stress run
Run stress against ProShop.war to push a baseline cpu load.
While the stress run takes place :
Check logs for exceptions/failures
Check ffdc logs for serviceability defects

After 72 hours run of app load and admin load, stop RPT & Jibe stress.

##Final Verification Instructions
After 72 hours of RPT run has completed.

Stop the RPT run
Check logs for exceptions/failures
Check ffdc logs for serviceability defects
Zip logs and upload to GSA space

See https://github.ibm.com/was-liberty/WS-CD-Open/issues/15599 for more information.

Test Repository

All of Development's tests are stored in github in the paths above suffixed onto
Open Liberty Github

The SVT ProShop test is stored at:
https://github.ibm.com/hanczar-maven/ProShop

Confidence Level

Please indicate your confidence in the testing (up to and including FAT) delivered with this feature by selecting one of these values:

The current assessment from the squad is 4. (The testing that does not run as part of the FAT suite is the long running (millions of hits) SVT style testing.)

0 - No automated testing delivered

1 - We have minimal automated coverage of the feature including golden paths. There is a relatively high risk that defects or issues could be found in this feature.

2 - We have delivered a reasonable automated coverage of the golden paths of this feature but are aware of gaps and extra testing that could be done here. Error/outlying scenarios are not really covered. There are likely risks that issues may exist in the golden paths

3 - We have delivered all automated testing we believe is needed for the golden paths of this feature and minimal coverage of the error/outlying scenarios. There is a risk when the feature is used outside the golden paths however we are confident on the golden path. Note: This may still be a valid end state for a feature... things like Beta features may well suffice at this level.

4 - We have delivered all automated testing we believe is needed for the golden paths of this feature and have good coverage of the error/outlying scenarios. While more testing of the error/outlying scenarios could be added we believe there is minimal risk here and the cost of providing these is considered higher than the benefit they would provide.

5 - We have delivered all automated testing we believe is needed for this feature. The testing covers all golden path cases as well as all the error/outlying scenarios that make sense. We are not aware of any gaps in the testing at this time. No manual testing is required to verify this feature.

[hutchig]

I will follow this process as it was at the date of this comment: https://github.ibm.com/was-liberty/WS-CD-Open/wiki/Feature-Review-(Feature-Test-Summary-Process)

[hutchig]

Any build that contains #7191 #7191 will contain 100% of the features and tests we plan to deliver for this feature. That pull request is now part of integration branch.

[hutchig]

All FATs are delivered and passing.