Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feature: enhanced search capability #83

Open
wants to merge 31 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
36a0f32
feat(assetlibrary) enhanced mode cloudformation templates and deploy …
jonemo Jan 26, 2022
5ba40a9
refactor(assetlibrary) remove unused stack policy file
jonemo Jan 26, 2022
f8fd5c5
fix(assetlibrary) OpenSearch CFN todos
jonemo Jan 26, 2022
5fef222
docs(assetlibrary): add detail about ElasticSearchDedicatedMasterCoun…
jonemo Feb 2, 2022
68a21ed
fix(assetlibrary): expand list of accepted Neptune instance types, va…
jonemo Feb 3, 2022
d21c22b
feat(installer): Introduce "enhanced" as answer option for asset libr…
jonemo Mar 14, 2022
ff26999
feat(installer): automatically create service-linked role for OpenSearch
jonemo Mar 15, 2022
38fbf10
feat(installer): deploy cfn-enhancedsearch.yaml
jonemo Mar 15, 2022
d47998c
feat(installer): export enhancedsearch config values
jonemo Mar 16, 2022
5bda0ac
enhanced search parameters for assetlibary-client
jonemo Jan 26, 2022
df9f101
swagger for enhanced search new parameters
jonemo Jan 26, 2022
c6b6f83
feat(assetlibrary): v1 of typescript code for enhanced search
jonemo Jan 26, 2022
d93284c
docs(assetlibrary): swagger: add pagination field to search results
jonemo Mar 18, 2022
34235ab
fix(assetlibrary): remove duplication in full/enhanced inversify configs
jonemo Mar 18, 2022
b85cfa8
docs(assetlibrary): add enhanced mode to "Assetlibrary Modes" documen…
jonemo Mar 18, 2022
7d043a3
fix(assetlibrary): SearchServiceLite implements SearchService
jonemo Jan 26, 2022
38db572
fix(assetlibrary): tighten up security groups for neptune poller
jonemo Mar 20, 2022
48afa83
feat(assetlibrary): Neptune FTS now works with OpenSearch 1.0
jonemo Mar 20, 2022
318db95
WIP(assetlibrary): Reduce Neptune Poller memory, increase polling fre…
jonemo Mar 21, 2022
edd74d2
style(assetlibrary): remove leftover commented out code
jonemo Apr 6, 2022
66f4883
docs(assetlibrary): fix search modes comparison table markdown
jonemo Apr 6, 2022
31a171c
docs(assetlibrary): update remaining references to OpenSearch
jonemo Apr 6, 2022
98636fd
docs(assetlibrary): consistently use opensearch over now legacy es names
jonemo Apr 9, 2022
668f52b
fix(assetlibrary): upgrade enhanced search lambda to python3.9 runtime
jonemo Jun 13, 2022
9ac1f65
docs(assetlibrary): integration tests
jonemo Jun 16, 2022
45fe683
changelog
jonemo Jun 30, 2022
370856c
docs(assetlibrary): migrating an existing asset library deployment
jonemo Jun 30, 2022
4ee4448
docs(assetlibrary): updated deployment parameters in enhanced AL migr…
aaronatbissell Sep 28, 2022
e21f443
docs(assetlibrary): added a note about needing to reboot neptune duri…
aaronatbissell Sep 28, 2022
b46963e
fix(assetlibrary): added extra security group for neptune stream poll…
aaronatbissell Sep 28, 2022
16aabfe
fix(assetlibrary): make search dao extendable using protected functio…
aaronatbissell Sep 18, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
{
"changes": [
{
"packageName": "@cdf/assetlibrary-client",
"comment": "New \"enhanced mode\" for Asset Library",
"type": "patch"
}
],
"packageName": "@cdf/assetlibrary-client",
"email": "[email protected]"
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
{
"changes": [
{
"packageName": "@cdf/assetlibrary",
"comment": "New \"enhanced mode\" for Asset Library",
"type": "minor"
}
],
"packageName": "@cdf/assetlibrary",
"email": "[email protected]"
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
{
"changes": [
{
"packageName": "@cdf/installer",
"comment": "New \"enhanced mode\" for Asset Library",
"type": "patch"
}
],
"packageName": "@cdf/installer",
"email": "[email protected]"
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
{
"changes": [
{
"packageName": "@cdf/integration-tests",
"comment": "New \"enhanced mode\" for Asset Library",
"type": "patch"
}
],
"packageName": "@cdf/integration-tests",
"email": "[email protected]"
}
4 changes: 4 additions & 0 deletions source/docs/migration.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,10 @@

While we endeavor to always make backward compatible changes, there may be times when we need to make changes that are not backward compatible. If these changes are made at the API level then the affected modules REST API vendor mime types will be versioned supporting both new and old versions, as well as the modules minor version bumped. But if the change affect something else such as how configuration is handled, or how applications are deployed, then the major versions of the modules will be bumped with migration notes added here.

## Migrating an existing Asset Library deployment to Enhanced Search

Starting with CDF Asset Library version 6.0.10 (part of CDF version 1.0.13), a new "enhanced" mode is available for CDF Asset Library. See the section [Migrating from full mode to enhanced mode](../packages/services/assetlibrary/docs/enhanced-search.md#migrating-from-full-mode-to-enhanced-mode) for guidance on migrating to enanced mode.

## Migrating from Release <=1.0.10 to 1.0.11

### Asset Library is now optional modules
Expand Down
31 changes: 28 additions & 3 deletions source/infrastructure/install-policy-3.json
Original file line number Diff line number Diff line change
Expand Up @@ -108,12 +108,37 @@
}
}
},
{
"Sid": "OpenSearchServiceLinkedRoleCreate",
"Action": "iam:CreateServiceLinkedRole",
"Effect": "Allow",
"Resource": "arn:aws:iam::*:role/aws-service-role/es.amazonaws.com/AWSServiceRoleForAmazonOpenSearchService",
"Condition": {
"StringLike": {
"iam:AWSServiceName": [
"opensearchservice.amazonaws.com",
"es.amazonaws.com"
]
}
}
},
{
"Sid": "OpenSearchServiceLinkedRoleGet",
"Action": "iam:GetRole",
"Effect": "Allow",
"Resource": "arn:aws:iam::*:role/aws-service-role/es.amazonaws.com/AWSServiceRoleForAmazonOpenSearchService"
},
{
"Sid": "OpenSearchLegacyServiceLinkedRoleGet",
"Action": "iam:GetRole",
"Effect": "Allow",
"Resource": "arn:aws:iam::*:role/aws-service-role/es.amazonaws.com/AWSServiceRoleForAmazonElasticsearchService"
},
{
"Sid": "ECS",
"Action": "ecs:*",
"Effect": "Allow",
"Resource": "*"
}

}
]
}
}
53 changes: 31 additions & 22 deletions source/packages/integration-tests/features/assetlibrary/README.md
Original file line number Diff line number Diff line change
@@ -1,38 +1,47 @@
# Asset Library integration tests
# Asset Library Integration Tests

Note: only the _full_ mode is tested as part of the CI/CD pipeline. We need to add both the _full (witht FGAC)_ and _lite_ modes to it. But for now, these need testing manually...
After following the [general steps for setting up an environment for integration testing](../README.md), the commands below can be used to run integration tests for Asset Library in various configurations.
The tests are executed locally on your development environment and send HTTP requests to an Asset Library API deployed in an AWS account.

### Testing full-with-authz mode
The subset of tests must match the mode and configuration of the Asset Library deployment at the URL configured in `ASSETLIBRARY_BASE_URL` in `path/to/local/.env`.
For example, integration tests in the `full-with-authz` folder only pass with an Asset Library deployment that is configured with `AUTHORIZATION_ENABLED=true`.

- Run asset library with FGAC enabled (only supported in _full_ mode):
```sh
$ assetlibrary> CONFIG_LOCATION="path/to/local/.env" ASSETLIBRARY_AUTHORIZATION_ENABLED=true npm run start
```
- Run FGAC specific integration tests:
```sh
$ integration-tests> CONFIG_LOCATION="path/to/integrationtest/.env" pnpm run integration-test -- features/assetlibrary/full-with-authz/*
```
Note: Only the _full_ mode is currently tested as part of the CI/CD pipeline.

### Testing lite mode
## Testing full/enhanced mode

See the [special notes for running lite mode tests](./lite/README.md).
To run integration tests for Asset Library in _full_ or _enhanced_ mode, irrespective of `AUTHORIZATION_ENABLED` setting:

- Run asset library in _lite_ mode:
```sh
$ assetlibrary> CONFIG_LOCATION="path/to/local/.env" ASSETLIBRARY_MODE=lite npm run start
$ source/packages/integration-tests> CONFIG_LOCATION="path/to/integrationtest/.env" pnpm run integration-test -- features/assetlibrary/full/*
```
- Run FGAC specific integration tests:

## Testing full-with-authz mode

To run integration tests for Asset Library in _full_ mode, when `AUTHORIZATION_ENABLED` is set to `true`:

```sh
$ integration-tests> CONFIG_LOCATION="path/to/integrationtest/.env" pnpm run integration-test -- features/assetlibrary/lite/*
$ source/packages/integration-tests> CONFIG_LOCATION="path/to/integrationtest/.env" npm run integration-test -- features/assetlibrary/full-with-authz/*
```

### Testing full mode
These tests are an addition to, not a superset of, "Testing full/enhanced mode".

## Testing enhanced mode

To run integration tests for Asset Library in _enhanced_ mode, irrespective of `AUTHORIZATION_ENABLED` setting:

- Run asset library in the default _full_ mode (requires a Neptune tunnel of running loc)
```sh
$ assetlibrary> CONFIG_LOCATION="path/to/local/.env" npm run start
$ source/packages/integration-tests> CONFIG_LOCATION="path/to/integrationtest/.env" npm run integration-test -- features/assetlibrary/full-with-enhancedsearch/*
```
- Run FGAC specific integration tests:

These tests are an addition to, not a superset of, "Testing full/enhanced mode".

## Testing lite mode

See the [special notes for running lite mode tests](lite/README.md).

To run integration tests for Asset Library in _lite_ mode:

```sh
$ integration-tests> CONFIG_LOCATION="path/to/integrationtest/.env" pnpm run integration-test -- features/assetlibrary/full/*
$ source/packages/integration-tests> CONFIG_LOCATION="path/to/integrationtest/.env" npm run integration-test -- features/assetlibrary/lite/*
```
Loading