Skip to content

Commit

Permalink
Enables java tests to use the test helper for S3 (#788)
Browse files Browse the repository at this point in the history
Co-authored-by: Michael Graeb <[email protected]>
  • Loading branch information
TingDaoK and graebm committed Jun 3, 2024
1 parent 5f6f2d2 commit ec236e3
Show file tree
Hide file tree
Showing 4 changed files with 98 additions and 122 deletions.
59 changes: 42 additions & 17 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -194,24 +194,21 @@ static: certificate has an existing certificate-key pair that was previously imp
```

## Testing
Many tests require environment variables to be set. These environment variables are translated at runtime to system properties for use by the tests. These tests will be quietly skipped if the properties they require are not set.

Environment variables can be set like so:
```
export ENV_VARIABLE_NAME="<variable value>"
```
Many tests require that you have [set up](https://console.aws.amazon.com/iot) an AWS IoT Thing.

Partial list of environment variables:
- `AWS_TEST_MQTT311_IOT_CORE_HOST`: AWS IoT service endpoint hostname for MQTT3
- `AWS_TEST_MQTT311_IOT_CORE_RSA_CERT`: Path to the IoT thing certificate for MQTT3
- `AWS_TEST_MQTT311_IOT_CORE_RSA_KEY`: Path to the IoT thing private key for MQTT3
- `AWS_TEST_MQTT311_IOT_CORE_ECC_CERT`: Path to the IoT thing with EC-based certificate for MQTT3
- `AWS_TEST_MQTT311_IOT_CORE_ECC_KEY`: Path to the IoT thing with ECC private key for MQTT3 (The ECC key file should only contains the ECC Private Key section to working on MacOS.)
- `AWS_TEST_MQTT311_ROOT_CA`: Path to the root certificate
- `AWS_TEST_HTTP_PROXY_HOST`: Hostname of proxy
- `AWS_TEST_HTTP_PROXY_PORT`: Port of proxy
- `NETWORK_TESTS_DISABLED`: Set this if tests are running in a constrained environment where network access is not guaranteed/allowed.
Some tests require pre-configured resources and proper environment variables to be set to run properly.
These tests will be quietly skipped if the environment variables they require are not set.

### IoT tests

Many IoT related tests require that you have [set up](https://console.aws.amazon.com/iot) an AWS IoT Thing.

* Some required environment variables
* `AWS_TEST_MQTT311_IOT_CORE_HOST`: AWS IoT service endpoint hostname for MQTT3
* `AWS_TEST_MQTT311_IOT_CORE_RSA_CERT`: Path to the IoT thing certificate for MQTT3
* `AWS_TEST_MQTT311_IOT_CORE_RSA_KEY`: Path to the IoT thing private key for MQTT3
* `AWS_TEST_MQTT311_IOT_CORE_ECC_CERT`: Path to the IoT thing with EC-based certificate for MQTT3
* `AWS_TEST_MQTT311_IOT_CORE_ECC_KEY`: Path to the IoT thing with ECC private key for MQTT3 (The ECC key file should only contains the ECC Private Key section to working on MacOS.)
* `AWS_TEST_MQTT311_ROOT_CA`: Path to the root certificate

Other Environment Variables that can be set can be found in the `SetupTestProperties()` function in [CrtTestFixture.java](https://github.com/awslabs/aws-crt-java/blob/main/src/test/java/software/amazon/awssdk/crt/test/CrtTestFixture.java)

Expand All @@ -236,6 +233,34 @@ These can be set persistently via Maven settings (usually in `~/.m2/settings.xml
</settings>%
```

### Proxy Tests

Most of proxy related tests need pre-configured Proxy host to run the tests properly.

* Required environment variables:
* `AWS_TEST_HTTP_PROXY_HOST`: Hostname of proxy
* `AWS_TEST_HTTP_PROXY_PORT`: Port of proxy
* `NETWORK_TESTS_DISABLED`: Set this if tests are running in a constrained environment where network access is not guaranteed/allowed.

### S3 Tests

Most of S3 related tests require AWS credentials and a set of pre-configured S3 buckets. There is a helper script from aws-c-s3 that can be used to set up the test environment, [here](https://github.com/awslabs/aws-c-s3/tree/main/tests/test_helper).

Example to use the helper and run the S3 tests:

``` sh
cd aws-crt-java
python3 -m pip install boto3
export CRT_S3_TEST_BUCKET_NAME=<bucket_name>
python3 crt/aws-c-s3/tests/test_helper/test_helper.py init
# Run S3ClientTest. eg: mvn -Dtest=S3ClientTest test
```

more details about the helper can be found from [here](https://github.com/awslabs/aws-c-s3/blob/main/tests/test_helper/README.md).

* Required environment variable:
* `CRT_S3_TEST_BUCKET_NAME`: The basic bucket name for S3 tests.

## IDEs
* CMake is configured to export a compilation database at target/cmake-build/compile_commands.json
* CLion: Build once with maven, then import the project as a [Compilation Database Project](https://www.jetbrains.com/help/clion/compilation-database.html)
Expand Down
2 changes: 1 addition & 1 deletion crt/aws-c-s3
Submodule aws-c-s3 updated 57 files
+0 −66 benchmarks/README.md
+0 −15 benchmarks/benchmark-config.json
+0 −13 benchmarks/benchmarks-stack/benchmarks-stack/.gitignore
+0 −6 benchmarks/benchmarks-stack/benchmarks-stack/.npmignore
+0 −14 benchmarks/benchmarks-stack/benchmarks-stack/bin/benchmarks-stack.ts
+0 −15 benchmarks/benchmarks-stack/benchmarks-stack/buildspec.yml
+0 −16 benchmarks/benchmarks-stack/benchmarks-stack/cdk.json
+0 −50 benchmarks/benchmarks-stack/benchmarks-stack/deploy/benchmarks_deploy.js
+0 −7 benchmarks/benchmarks-stack/benchmarks-stack/jest.config.js
+0 −163 benchmarks/benchmarks-stack/benchmarks-stack/lib/benchmarks-stack.ts
+0 −83 benchmarks/benchmarks-stack/benchmarks-stack/lib/canary-policy-doc.json
+0 −29 benchmarks/benchmarks-stack/benchmarks-stack/lib/canary.sh
+0 −42 benchmarks/benchmarks-stack/benchmarks-stack/lib/get_p90.py
+0 −151 benchmarks/benchmarks-stack/benchmarks-stack/lib/init_instance.sh
+0 −65 benchmarks/benchmarks-stack/benchmarks-stack/lib/project_scripts/run_aws_c_s3.sh
+0 −46 benchmarks/benchmarks-stack/benchmarks-stack/lib/project_scripts/run_java_crt.sh
+0 −18 benchmarks/benchmarks-stack/benchmarks-stack/lib/run_project_template.sh
+0 −59 benchmarks/benchmarks-stack/benchmarks-stack/lib/show_instance_dashboard.sh
+0 −33 benchmarks/benchmarks-stack/benchmarks-stack/package.json
+0 −0 benchmarks/benchmarks-stack/benchmarks-stack/response.json
+0 −40 benchmarks/benchmarks-stack/benchmarks-stack/test/benchmarks.test.ts
+0 −23 benchmarks/benchmarks-stack/benchmarks-stack/tsconfig.json
+0 −14 benchmarks/dashboard-stack/.gitignore
+0 −6 benchmarks/dashboard-stack/.npmignore
+0 −15 benchmarks/dashboard-stack/bin/benchmarks.ts
+0 −12 benchmarks/dashboard-stack/cdk.json
+0 −7 benchmarks/dashboard-stack/jest.config.js
+0 −68 benchmarks/dashboard-stack/lambda/benchmarkManager.py
+0 −367 benchmarks/dashboard-stack/lib/dashboard-stack.ts
+0 −10 benchmarks/dashboard-stack/lib/policy-doc/admin-policy-doc.json
+0 −39 benchmarks/dashboard-stack/package.json
+0 −0 benchmarks/dashboard-stack/response.json
+0 −23 benchmarks/dashboard-stack/tsconfig.json
+6 −0 docs/memory_aware_request_execution.md
+27 −8 include/aws/s3/private/s3_buffer_pool.h
+11 −19 include/aws/s3/private/s3_meta_request_impl.h
+2 −2 include/aws/s3/private/s3_request_messages.h
+2 −2 include/aws/s3/private/s3_util.h
+71 −4 include/aws/s3/s3_client.h
+18 −7 source/s3_auto_ranged_put.c
+91 −15 source/s3_buffer_pool.c
+38 −9 source/s3_checksum_stream.c
+3 −2 source/s3_client.c
+2 −4 source/s3_copy_object.c
+18 −16 source/s3_endpoint_resolver/aws_s3_endpoint_resolver_partition.c
+189 −118 source/s3_meta_request.c
+39 −14 source/s3_request_messages.c
+1 −1 source/s3_util.c
+7 −2 tests/CMakeLists.txt
+13 −87 tests/s3_asyncwrite_tests.c
+142 −0 tests/s3_buffer_pool_tests.c
+158 −4 tests/s3_data_plane_tests.c
+9 −5 tests/s3_request_messages_tests.c
+3 −1 tests/s3_tester.c
+1 −0 tests/s3_tester.h
+5 −0 tests/test_helper/README.md
+14 −8 tests/test_helper/test_helper.py
Loading

0 comments on commit ec236e3

Please sign in to comment.