Upload 1747/cloud agnostic e2e tests (#532)

* define and deserialize test case object with delivery targets * Extending the tests to do some assertions against the file info response * Extending the File Copy test to check for the right location path suffix and also some other refactors * Adding environment configurations to the fileCopy test and adding space for environment patterns in the test cases * Adding in the test case path template configs for TEST environment * Updating the test configurations for each environment * Updating Processing Status tests to use expectations from test case rather than collecting info from Azure * Upading tests in File Copy and modifying v1 manifest test case file * Addressing some PR comments and refactors * Updating Readme with latest details * Updating local.properities-example with changes and minor readme updates --------- Co-authored-by: Alex de los Reyes <[email protected]> Co-authored-by: Alex de los Reyes <[email protected]>
CDCgov · Oct 23, 2024 · e9c8872 · e9c8872
1 parent eea0185
commit e9c8872
Show file tree

Hide file tree

Showing 13 changed files with 444 additions and 296 deletions.
diff --git a/tests/smoke/kotlin/README.md b/tests/smoke/kotlin/README.md
@@ -1,80 +1,113 @@
-# Upload API Smoke Tests with TestNG
-
-This is a Kotlin/Gradle project that uses the TestNG framework to automate smoke testing for the Upload API. These tests
-upload actual files and verify the functionality for metadata verification, file copy, and Processing Status API
-integration. They are intended to be run after a release to the tst, stg, or prd environments. They can also be run
-locally on a machine within the CDC network.
+# Upload API End to End Tests with TestNG
+This is a Kotlin/Gradle project that uses the TestNG framework to automate end to end testing for the Upload API. These tests upload actual files and verify the functionality for metadata verification, file copy, Processing Status API integration and healthcheck testing. They are intended to be run after a release to the `DEV`, `TEST` or `STAGE` environments or for frequent checking of functionality against any environment, including local environments.
 
 ## Local Setup
 
 The following tools need to be installed on your machine:
 
-- Java JDK 17
-- Gradle
+- [Java JDK 17](https://www.oracle.com/java/technologies/javase/jdk17-archive-downloads.html)
+- [Gradle](https://gradle.org/install/)
+
+> [!TIP]
+> For Windows users, the `gradlew` batch file in the repo can be used to execute gradle commands as long as Gradle is installed.
 
 ### Install Gradle Dependencies
 
-Next, run `gradle build` to install dependencies for this project. This also installs the TestNG dependency.  **Note
-that you may need to turn off Zscalar in order for this operation to be successful.**
+Next, run `gradle build` to install dependencies for this project. This also installs the TestNG dependency and other related dependencies for the project.  
+
+> [!NOTE]
+> You may need to turn off Zscalar in order for this operation to be successful.
 
 ### Environment Setup
 
-Next, set required environment variables. This can be done by setting local gradle properties in a `local.properties`
-file at the root level, or passing them in on the command line as Java system properties. To see a list of required
-variables, look at the `src/test/kotlin/util/EnvConfig.kt` file.
-These environment variables are how you target different DEX environments. For example, for the dev environment, all
-environment variables need to point to URLs, endpoints, and services uses by the Upload API dev environment.
+Copy `local.properites-example` to `local.properties` for the basic required environment variables at the root level of the project. 
 
-The EnvConfig class reads configuration values from a local.properties file. This setup allows us to manage environment-specific settings, like URLs and credentials.
+The following are the currently configured environment variables that can be set for the tests to run.
 
-Storage account keys can be found in our key vaults (ocio-<env>-upload-vault). 
-Example of dev key vault: `ocio-dev-upload-vault - Microsoft Azure`. Need to have `su` account for Microsoft Azure portal as a pre-requisite.
+| Environment Variable | Required? | Description                                      |
+| ---------------------| --------- | -------------------------------------------------|
+| environment          | Yes       | Environment to target (`LOCAL`, `DEV`, `TEST`, `STAGE`)
+| upload.url           | Yes       | URL of the upload API
+| ps.api.url           | Yes       | URL for Processing Status API  
+| sams.username        | No        | SAMS username for authentication (if needed)
+| sams.password        | No        | SAMS password for authentication (if needed)
 
-![img.png](img.png)
+The EnvConfig class (`src/test/kotlin/util/EnvConfig.kt`) reads configuration values from a local.properties file. This setup allows us to manage environment-specific settings, like URLs and credentials.
 
 ### Running tests
 
-This project contains a set of test suites that define the tests to be run. These suites are grouped by environment and
-are broken up by use case.
-They are run by executing the `gradle test` comment with a few gradle properties that are passed in as command line
-arguments:
+This project contains a set of test suites that define the tests to be run for different functional areas.  Tests are located in `/src/test/kotlin`.  The test files are:
+| Test File      | Purpose        |
+|----------------|----------------|
+| `FileCopy.kt ` | Testing that the upload api can accept files for different manifest configurations and can upload and transfer files as expected.
+| `Health.kt`    | Test the healthcheck endpoint
+| `Info.kt `     | Test the /info endpoint 
+| `ProcStat.kt`  | Testing processing status reports after an upload
+
+They are run by executing the `gradle test` with the option of using some command line parameters.
+
+#### Optional Parameters
 
-- `manifestFilter` - This is a required argument that allows you to select a subset of use cases you want to run the
-  tests against. It is a comma-separated list of values for each key that you want to run. Available manifests are added
-  to a file which is under resources folder in json format. By default, the tests will run for all use cases.
+- `manifestFilter` - This argument allows you to select a subset of use cases you want to run the tests against. It is a comma-separated list of values for each key that you want to run. Available manifests are added to a file which is under resources folder in json format. By default, the tests will run for all use cases.
 
 #### Examples:
+The following are some examples of test run commands.
 
-- Run all use cases:
+> [!TIP]
+> The `--tests` parameter lets you select only a specific test file to run or a specific test within a file.
 
-`gradle test`
+> [!TIP]
+> The `--rerun` command may be needed in order to force tests to rerun, otherwise tests may be skipped if there are no changes.
 
-## Data Providers for TestNG
+##### Run All Tests
+
+```
+gradle test
+```
+
+##### Run tests from a specific file
 
-This project includes a `DataProvider` utility class that supplies test data to TestNG tests. The class uses the Jackson
-library to read and filter JSON manifests based on criteria specified through system properties.
+```
+gradle test --tests "FileCopy"
+```
 
-### How It Works
+##### Run tests from a specific file and a specific test
 
-1. Loading Manifests:
+```
+gradle test --tests "FileCopy.shouldUploadFile"
+```
 
-- The `DataProvider` class reads JSON manifest files specified in the data provider methods.
+##### Run tests filtered by manifest
 
-2. Filtering Manifests:
+> [!TIP]
+> This filter is a semicolon-separated string of key-value pairs.
+> Each key can have multiple comma-separated values.
+```
+gradle test -PmanifestFilter='data_stream_id=ehdi'
+```
+```
+gradle test -PmanifestFilter='meta_destination_id=ndlp&meta_ext_source=IZGW'
+```
+```
+gradle test -PmanifestFilter='jurisdiction=AKA,CA;data_stream_route=csv,other&sender_id=CA-ABCs,IZGW'
+```
+
+## Data Providers for TestNG
 
-- A system property `manifestFilter` can be set to define filtering criteria.
-- This filter is a semicolon-separated string of key-value pairs.
-- Each key can have multiple comma-separated values.
+This project includes a `DataProvider` utility class that supplies test data to TestNG tests.  The test data being used in tests are essentially test cases that define how a single test can be repeated and validated for different cases defined within the json being used as a source for the DataProvider.
 
-### Example Usage
+### Data Provider Definitions
 
-To filter specific key-value pairs in the JSON manifests, use the `manifestFilter` system property.
-Here are the example commands to run the tests with manifest filters:
+The `dataProvider` decorator before a test defines what data provider should be used to pass in data into the test.  These are the currently defined Data Providers in `/src/test/kotlin/util/DataProvider.kt`
 
-`gradle test -PmanifestFilter='meta_destination_id=ndlp&meta_ext_source=IZGW'`
-`gradle test -PmanifestFilter='jurisdiction=AKA,CA;data_stream_route=csv,other&sender_id=CA-ABCs,IZGW'`
+| Data Provider Name                      | Associated json file                      | Description                 |
+|-----------------------------------------|-------------------------------------------|-----------------------------|
+| `versionProvider`                       | N/A                                       | Returns an array of `["v1", "v2"]` for versions
+| `validManifestAllProvider`              | `valid_manifests_v2.json`                 | All v2 manifests and path configs |
+| `validManifestV1Provider`               | `valid_manifests_v1.json`                 | All v1 manifests and configs | 
+| `invalidManifestRequiredFieldsProvider` | `invalid_manifests_required_fields.json`  | Manifests with invalid values |
+| `invalidManifestInvalidValueProvider`   | `invalid_manifests_invalid_value.json`    | Manifests with invalid fields  |
 
-We can run tests for specified manifest in a single command line argument.
 
 ### Future Improvements
 

diff --git a/tests/smoke/kotlin/local.properties-example b/tests/smoke/kotlin/local.properties-example
@@ -1,35 +1,29 @@
+#local
+environment=LOCAL
+upload.url=http://localhost:8080
+ps.api.url=http://localhost:8080
+sams.username=
+sams.password=
+
 #dev
+environment=DEV
 upload.url=https://apidev.cdc.gov
 ps.api.url=https://dex-dev-svc.cdc.gov
 sams.username=
 sams.password=
-dex.storage.connection.string=
-edav.storage.account.name=
-routing.storage.connection.string=
-azure.client.id=
-azure.client.secret=
-azure.tenant.id=
+
 
 # tst
+environment=TEST
 upload.url=https://apitst.cdc.gov
 ps.api.url=https://dex-tst-svc.cdc.gov
 sams.username=
 sams.password=
-dex.storage.connection.string=
-edav.storage.account.name=
-routing.storage.connection.string=
-azure.client.id=
-azure.client.secret=
-azure.tenant.id=
+
 
 # stg
+environment=STAGE
 upload.url=https://apistg.cdc.gov
 ps.api.url=https://dex-stg-svc.cdc.gov
 sams.username=
 sams.password=
-dex.storage.connection.string=
-edav.storage.account.name=
-routing.storage.connection.string=
-azure.client.id=
-azure.client.secret=
-azure.tenant.id=
diff --git a/tests/smoke/kotlin/src/test/kotlin/FileCopy.kt b/tests/smoke/kotlin/src/test/kotlin/FileCopy.kt
@@ -1,5 +1,3 @@
-import com.azure.identity.ClientSecretCredentialBuilder
-import com.azure.storage.blob.BlobClient
 import dex.DexUploadClient
 import org.testng.Assert
 import org.testng.ITestContext
@@ -9,35 +7,24 @@ import tus.UploadClient
 import util.*
 import util.ConfigLoader.Companion.loadUploadConfig
 import util.DataProvider
+import java.net.URLDecoder
+import java.nio.charset.StandardCharsets
+import java.time.ZonedDateTime
+import java.util.TimeZone
 import kotlin.collections.HashMap
 
-
 @Listeners(UploadIdTestListener::class)
 @Test()
 class FileCopy {
     private val testFile = TestFile.getResourceFile("10KB-test-file")
-    private val authClient = DexUploadClient(EnvConfig.UPLOAD_URL)
-    private val dexBlobClient = Azure.getBlobServiceClient(EnvConfig.DEX_STORAGE_CONNECTION_STRING)
-    private val edavBlobClient = Azure.getBlobServiceClient(
-        EnvConfig.EDAV_STORAGE_ACCOUNT_NAME,
-        ClientSecretCredentialBuilder()
-            .clientId(EnvConfig.AZURE_CLIENT_ID)
-            .clientSecret(EnvConfig.AZURE_CLIENT_SECRET)
-            .tenantId(EnvConfig.AZURE_TENANT_ID)
-            .build()
-    )
-    private val routingBlobClient = Azure.getBlobServiceClient(EnvConfig.ROUTING_STORAGE_CONNECTION_STRING)
-    private val bulkUploadsContainerClient = dexBlobClient.getBlobContainerClient(Constants.BULK_UPLOAD_CONTAINER_NAME)
-    private val edavContainerClient = edavBlobClient.getBlobContainerClient(Constants.EDAV_UPLOAD_CONTAINER_NAME)
-    private val routingContainerClient =
-        routingBlobClient.getBlobContainerClient(Constants.ROUTING_UPLOAD_CONTAINER_NAME)
+    private val dexUploadClient = DexUploadClient(EnvConfig.UPLOAD_URL)
     private lateinit var authToken: String
     private lateinit var testContext: ITestContext
     private lateinit var uploadClient: UploadClient
 
     @BeforeTest(groups = [Constants.Groups.FILE_COPY])
     fun beforeFileCopy() {
-        authToken = authClient.getToken(EnvConfig.SAMS_USERNAME, EnvConfig.SAMS_PASSWORD)
+        authToken = dexUploadClient.getToken(EnvConfig.SAMS_USERNAME, EnvConfig.SAMS_PASSWORD)
     }
 
     @BeforeMethod
@@ -51,41 +38,43 @@ class FileCopy {
         dataProvider = "validManifestAllProvider",
         dataProviderClass = DataProvider::class
     )
-    fun shouldUploadFile(manifest: HashMap<String, String>) {
-        val uid = uploadClient.uploadFile(testFile, manifest)
+    fun shouldUploadFile(case: TestCase) {
+        val uid = uploadClient.uploadFile(testFile, case.manifest)
             ?: throw TestNGException("Error uploading file ${testFile.name}")
         testContext.setAttribute("uploadId", uid)
-        Thread.sleep(2000)
-
-        // First, check bulk upload and .info file.
-        val uploadBlob = bulkUploadsContainerClient.getBlobClient("${Constants.TUS_PREFIX_DIRECTORY_NAME}/$uid")
-        val uploadInfoBlob =
-            bulkUploadsContainerClient.getBlobClient("${Constants.TUS_PREFIX_DIRECTORY_NAME}/$uid.info")
-
-        Assert.assertTrue(uploadBlob.exists())
-        Assert.assertTrue(uploadInfoBlob.exists())
-        Assert.assertEquals(uploadBlob.properties.blobSize, testFile.length())
-
-        // Next, check that the file arrived in destination storage.
-        val config = loadUploadConfig(dexBlobClient, manifest)
-        val filenameSuffix = Filename.getFilenameSuffix(config.copyConfig, uid)
-        val expectedFilename = "${
-            Metadata.getFilePrefix(config.copyConfig, manifest)
-        }${Metadata.getFilename(manifest)}${filenameSuffix}${testFile.extension}"
-        var expectedBlobClient: BlobClient?
-
-        if (config.copyConfig.targets.contains("edav")) {
-            expectedBlobClient = edavContainerClient.getBlobClient(expectedFilename)
-
-            Assert.assertNotNull(expectedBlobClient)
-            Assert.assertEquals(expectedBlobClient!!.properties.blobSize, testFile.length())
-        }
-
-        if (config.copyConfig.targets.contains("routing")) {
-            expectedBlobClient = routingContainerClient.getBlobClient(expectedFilename)
-
-            Assert.assertNotNull(expectedBlobClient)
-            Assert.assertEquals(expectedBlobClient!!.properties.blobSize, testFile.length())
+        Thread.sleep(5000)
+        val uploadInfo = dexUploadClient.getFileInfo(uid, authToken)
+
+        // Check File Info
+        val expectedBytes: Long = 10240
+        Assert.assertEquals(uploadInfo.fileInfo.sizeBytes, expectedBytes)
+
+        // Check Upload Status
+        Assert.assertEquals(uploadInfo.uploadStatus.status, "Complete", "File upload status is not 'Complete'")
+
+        // Check Deliveries
+        Assert.assertEquals(uploadInfo.deliveries?.size, case.deliveryTargets?.size, "Expected ${case.deliveryTargets?.size ?: 0 } deliveries")
+
+        val expectedDeliveryNames = case.deliveryTargets?.map{ it.name }?.sorted()
+        val actualDeliveryNames = uploadInfo.deliveries?.map{ it.name }?.sorted()
+        Assert.assertEquals(actualDeliveryNames, expectedDeliveryNames, "Actual delivery targets do not match expected targets")
+
+        val currentDateTime = ZonedDateTime.now(TimeZone.getTimeZone("GMT").toZoneId())
+        uploadInfo.deliveries?.forEach { delivery ->
+            Assert.assertEquals(delivery.status, "SUCCESS") // remove the assertion above?
+            val actualLocation = URLDecoder.decode(delivery.location, StandardCharsets.UTF_8.toString())
+            val pattern = case.deliveryTargets?.find{ it.name == delivery.name}?.pathTemplate?.get(EnvConfig.ENVIRONMENT)
+            val expectedLocation = pattern
+                ?.replace("{dataStream}", case.manifest["data_stream_id"].toString())
+                ?.replace("{route}", case.manifest["data_stream_route"].toString())
+                ?.replace("{year}", currentDateTime.year.toString() )
+                ?.replace("{month}", String.format("%02d", currentDateTime.monthValue) )
+                ?.replace("{day}", String.format("%02d", currentDateTime.dayOfMonth) )
+                ?.replace("{hour}", String.format("%02d", currentDateTime.hour) )
+                ?.replace("{filename}", case.manifest["received_filename"].toString())
+                ?.replace("{uploadId}",uid)
+            Assert.assertTrue(actualLocation.endsWith(expectedLocation.toString()), "Actual location ($actualLocation) does not end with the expected path: $expectedLocation")
+            Assert.assertEquals(delivery.issues, null)
         }
     }
 
@@ -94,37 +83,18 @@ class FileCopy {
         dataProvider = "validManifestV1Provider",
         dataProviderClass = DataProvider::class
     )
-    fun shouldTranslateMetadataGivenV1SenderManifest(manifest: HashMap<String, String>) {
-        val useCase = Metadata.getUseCaseFromManifest(manifest)
-        val dexContainerClient = dexBlobClient.getBlobContainerClient(useCase)
-        val v1Config = loadUploadConfig(dexBlobClient, manifest)
-        val v2ConfigFilename = v1Config.compatConfigFilename ?: "$useCase.json"
-        val v2Config = loadUploadConfig(dexBlobClient, v2ConfigFilename, "v2")
-        val metadataMapping = v2Config.metadataConfig.fields.filter { it.compatFieldName != null }
-            .associate { it.compatFieldName to it.fieldName }
-
-        val uid = uploadClient.uploadFile(testFile, manifest)
+    fun shouldTranslateMetadataGivenV1SenderManifest(case: TestCase) {
+        val uid = uploadClient.uploadFile(testFile, case.manifest)
             ?: throw TestNGException("Error uploading file ${testFile.name}")
         testContext.setAttribute("uploadId", uid)
-        Thread.sleep(1000)
-
-        val filenameSuffix = Filename.getFilenameSuffix(v1Config.copyConfig, uid)
-        val expectedFilename =
-            "${Metadata.getFilePrefix(v1Config.copyConfig)}${Metadata.getFilename(manifest)}$filenameSuffix${testFile.extension}"
-        val expectedBlobClient = dexContainerClient.getBlobClient(expectedFilename)
-        val blobMetadata = expectedBlobClient.properties.metadata
-
-        metadataMapping.forEach { (v1Key, v2Key) ->
-            Assert.assertTrue(
-                blobMetadata.containsKey(v2Key),
-                "Mismatch: Blob metadata does not contain expected V2 key: $v2Key which should map from V1 key: $v1Key"
-            )
-        }
-
-        metadataMapping.forEach { (v1Key, v2Key) ->
-            val v1Val = manifest[v1Key] ?: ""
-            val v2Val = blobMetadata[v2Key] ?: ""
-            Assert.assertEquals(v1Val, v2Val, "Expected V1 value: $v1Val does not match with actual V2 value: $v2Val")
-        }
+        Thread.sleep(3000)
+
+        val uploadInfo = dexUploadClient.getFileInfo(uid, authToken)
+        //Assert.assertTrue(uploadInfo.deliveries?.all { it.status == "SUCCESS" }?:false, "Not all deliveries are 'SUCCESS' - Deliveries: ${uploadInfo.deliveries}")
+        case.manifest.forEach{(manifestKey, manifestValue) -> 
+            Assert.assertEquals(uploadInfo.manifest[manifestKey], manifestValue.toString(), "Actual manifest value does not equal the expected manifest value")
+       }
+        Assert.assertNotNull(uploadInfo.manifest["dex_ingest_datetime"])
+        Assert.assertEquals(uploadInfo.manifest["upload_id"], uid, "Upload ID on the manifest is not the expected upload ID")
     }
-}
+}