Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(parser): add built-in schemas #1788

Merged
merged 51 commits into from
Nov 20, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
51 commits
Select commit Hold shift + click to select a range
aa66d8c
add dynamodb schema
am29d Oct 31, 2023
64c3eb1
add alb
am29d Oct 31, 2023
c5127f9
merged feat/v2
am29d Oct 31, 2023
f0464ab
add parser to v2 build
am29d Nov 2, 2023
f353c7f
fix test
am29d Nov 2, 2023
4a66dec
add alb
am29d Nov 2, 2023
633ce9a
Merge branch 'feat/v2' into feat/praser/built-in-schemas
am29d Nov 3, 2023
6078cb0
add built-in schema
am29d Nov 10, 2023
5bcab2e
add more tests for schemas
am29d Nov 10, 2023
0e9db61
remove index export
am29d Nov 10, 2023
fefe01a
merged target
am29d Nov 13, 2023
c06989b
add cloudwatch with base64 zlip transform
am29d Nov 13, 2023
a4d3852
add throw test case
am29d Nov 13, 2023
7a2a203
formatting
am29d Nov 13, 2023
689ef45
add kafka schema
am29d Nov 13, 2023
7e0dd71
restructured tests
am29d Nov 13, 2023
067fd82
add vpc lattice and lattice v2
am29d Nov 14, 2023
04fd892
s3 event notification should extend eventbridge
am29d Nov 14, 2023
694f389
s3 sqs should extend from sqs
am29d Nov 14, 2023
1408bb9
simplify cloudwatch extract from string
am29d Nov 14, 2023
60b139b
keep message as string, instead of empty object
am29d Nov 14, 2023
1fdac46
fix detail type of eb and field names
am29d Nov 14, 2023
f6d09d2
remove duplicated entries
am29d Nov 14, 2023
3f8f4bb
fix homepage URL in readme
am29d Nov 14, 2023
ccb51e1
improved test coverage
am29d Nov 14, 2023
208a66f
key and value are always present
am29d Nov 14, 2023
d8919d3
cleanup unnecessary definitions, widen peerDep version req
am29d Nov 14, 2023
8eefbe8
Update packages/parser/src/schemas/cloudwatch.ts
am29d Nov 15, 2023
eceea76
clean up events, some fields are imaginary
am29d Nov 15, 2023
f67a8e1
fix api gw
am29d Nov 15, 2023
955d277
fix broken IP addresses in examples
am29d Nov 15, 2023
f165302
add more tests to api gw
am29d Nov 15, 2023
06d5467
fix apigw2 add more tests
am29d Nov 15, 2023
1af893a
add optional scopes to apigwv2
am29d Nov 15, 2023
a9cfa6a
add optional field back to api gw, stricter methods for vpc lattice
am29d Nov 15, 2023
2d671d0
add test for messageId refinement
am29d Nov 15, 2023
7cdcc0b
remove redundant entry
am29d Nov 15, 2023
dd39001
fix sqs
am29d Nov 15, 2023
bd0cefe
add dmarcPolicy for ses
am29d Nov 15, 2023
7b33250
added tests
am29d Nov 15, 2023
51219a8
moved cw function from kinesis, fix imports
am29d Nov 17, 2023
bbf5ac3
add parser to build step in ci
am29d Nov 17, 2023
d8a040f
use any safely here
am29d Nov 17, 2023
254ebd7
removed console logs
am29d Nov 17, 2023
e0dcf23
name, add datetime to strings
am29d Nov 17, 2023
0b6d570
narrow string to datetime
am29d Nov 17, 2023
f453246
refine to url
am29d Nov 17, 2023
455b329
imports, remove try/catch
am29d Nov 17, 2023
228217e
add .js extension to imports
am29d Nov 17, 2023
e330e51
moved comment, fixed path
am29d Nov 17, 2023
8ce4af6
rename event filename to fix events
am29d Nov 20, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/actions/cached-node-modules/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -44,5 +44,6 @@ runs:
npm run build -w packages/parameters & \
npm run build -w packages/idempotency & \
npm run build -w packages/batch & \
npm run build -w packages/parser & \
npm run build -w packages/testing
shell: bash
Original file line number Diff line number Diff line change
Expand Up @@ -25,9 +25,9 @@ jobs:
with:
nodeVersion: ${{ matrix.version }}
- name: Run linting
run: npm run lint -w packages/commons -w packages/logger -w packages/tracer -w packages/metrics -w packages/parameters -w packages/idempotency -w packages/batch
run: npm run lint -w packages/commons -w packages/logger -w packages/tracer -w packages/metrics -w packages/parameters -w packages/idempotency -w packages/batch -w packages/parser
- name: Run unit tests
run: npm t -w packages/commons -w packages/logger -w packages/tracer -w packages/metrics -w packages/parameters -w packages/idempotency -w packages/batch
run: npm t -w packages/commons -w packages/logger -w packages/tracer -w packages/metrics -w packages/parameters -w packages/idempotency -w packages/batch -w packages/parser
check-examples:
runs-on: ubuntu-latest
env:
Expand Down
3 changes: 1 addition & 2 deletions .npmignore
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
src
tests
jest.config.js
jest.config.cjs
tsconfig.json
.vscode
.github
Expand All @@ -13,7 +13,6 @@ coverage
tslint.json
tsconfig.json
MakeFile
jest.config.js
.npmignore
.eslintignore
.huskyrc.js
Expand Down
2 changes: 1 addition & 1 deletion docs/snippets/idempotency/samples/makeIdempotentJmes.json
Original file line number Diff line number Diff line change
Expand Up @@ -27,4 +27,4 @@
},
"body": "{\"user\":\"xyz\",\"productId\":\"123456789\"}",
"isBase64Encoded": false
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,4 @@
"name": "foo",
"productId": 10000
}
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,4 @@
"name": "Foo"
},
"productId": 10000
}
}
2 changes: 1 addition & 1 deletion docs/snippets/idempotency/samples/workingWithBatch.json
Original file line number Diff line number Diff line change
Expand Up @@ -23,4 +23,4 @@
"awsRegion": "us-east-2"
}
]
}
}
22 changes: 22 additions & 0 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Original file line number Diff line number Diff line change
Expand Up @@ -5,16 +5,20 @@ module.exports = {
},
runner: 'groups',
preset: 'ts-jest',
moduleNameMapper: {
'^(\\.{1,2}/.*)\\.js$': '$1',
},
transform: {
'^.+\\.ts?$': 'ts-jest',
'^.+\\.ts?$': ['ts-jest', {tsconfig: './tests/tsconfig.json'}],
am29d marked this conversation as resolved.
Show resolved Hide resolved

},
moduleFileExtensions: ['js', 'ts'],
collectCoverageFrom: ['**/src/**/*.ts', '!**/node_modules/**'],
testMatch: ['**/?(*.)+(spec|test).ts'],
roots: ['<rootDir>/src', '<rootDir>/tests'],
testPathIgnorePatterns: ['/node_modules/'],
testEnvironment: 'node',
coveragePathIgnorePatterns: ['/node_modules/', '/types/'],
coveragePathIgnorePatterns: ['/node_modules/', '/types'],
coverageThreshold: {
global: {
statements: 100,
Expand All @@ -24,5 +28,4 @@ module.exports = {
},
},
coverageReporters: ['json-summary', 'text', 'lcov'],
setupFiles: ['<rootDir>/tests/helpers/populateEnvironmentVariables.ts'],
};
46 changes: 31 additions & 15 deletions packages/parser/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -2,27 +2,45 @@
"name": "@aws-lambda-powertools/parser",
"version": "0.0.0",
"description": "The parser package for the Powertools for AWS Lambda (TypeScript) library.",
"author": {
"name": "Amazon Web Services",
"url": "https://aws.amazon.com"
},
"publishConfig": {
"access": "restricted"
},
"scripts": {
"test": "npm run test:unit",
"test:unit": "jest --group=unit --detectOpenHandles --coverage --verbose",
"jest": "jest --detectOpenHandles --coverage --verbose",
"watch": "jest --watch",
"build": "tsc --build --force",
"build:cjs": "tsc --build tsconfig.json && echo '{ \"type\": \"commonjs\" }' > lib/cjs/package.json",
"build:esm": "tsc --build tsconfig.esm.json && echo '{ \"type\": \"module\" }' > lib/esm/package.json",
"build": "npm run build:esm & npm run build:cjs",
"lint": "eslint --ext .ts,.js --no-error-on-unmatched-pattern .",
"lint-fix": "eslint --fix --ext .ts,.js --no-error-on-unmatched-pattern .",
"prebuild": "rimraf ./lib",
"prepack": "node ../../.github/scripts/release_patch_package_json.js ."
},
"author": {
"name": "Amazon Web Services",
"url": "https://aws.amazon.com"
},
"lint-staged": {
"*.{js,ts}": "npm run lint-fix"
},
"homepage": "https://github.com/aws-powertools/powertools-lambda-typescript/tree/main/packages/batch#readme",
"homepage": "https://github.com/aws-powertools/powertools-lambda-typescript/tree/main/packages/parser#readme",
"license": "MIT-0",
"main": "./lib/index.js",
"types": "./lib/index.d.ts",
"type": "module",
"exports": {
".": {
"require": {
"types": "./lib/cjs/index.d.ts",
"default": "./lib/cjs/index.js"
},
"import": {
"types": "./lib/esm/index.d.ts",
"default": "./lib/esm/index.js"
}
}
},
"main": "./lib/cjs/index.js",
"types": "./lib/cjs/index.d.ts",
"files": [
"lib"
],
Expand All @@ -42,10 +60,8 @@
"serverless",
"nodejs"
],
"publishConfig": {
"access": "restricted"
},
"devDependencies": {
"zod": "^3.22.2"

"peerDependencies": {
"zod": ">=3.x"
}
}
}
File renamed without changes.
22 changes: 22 additions & 0 deletions packages/parser/src/schemas/alb.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
import { z } from 'zod';

const AlbSchema = z.object({
httpMethod: z.string(),
path: z.string(),
body: z.string(),
isBase64Encoded: z.boolean(),
headers: z.record(z.string(), z.string()).optional(),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see that both headers and queryStringParameters are marked as optional here, while in the Python implementation they're not.

Is this to allow the multi value extension later?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, once you turn on multi header, you can't pass headers, thus examples with multi-values break. pt-py did not test this case, or I could not find it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have looked into possible options and atm there is no straight forward way to have multiple mutually exclusive key/value pairs. It's not impossible, just needs more dedicated time to go through few options and pick the best.

I'd suggest to move forward as it is now and come back later.

queryStringParameters: z.record(z.string(), z.string()).optional(),
requestContext: z.object({
elb: z.object({
targetGroupArn: z.string(),
}),
}),
});

const AlbMultiValueHeadersSchema = AlbSchema.extend({
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The schema includes also multi value query string parameters, should we name this something more generic i.e. AlbMultiValueSchema (which I don't love) or something like it?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The semantic link between the schema and the feature is in wording Multi-Value Header, thus I assume it's easier to connect the feature in the service and the schema values.

I'd suggest to keep it.

multiValueHeaders: z.record(z.string(), z.array(z.string())),
multiValueQueryStringParameters: z.record(z.string(), z.array(z.string())),
});

export { AlbSchema, AlbMultiValueHeadersSchema };
110 changes: 110 additions & 0 deletions packages/parser/src/schemas/apigw.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,110 @@
import { z } from 'zod';

const APIGatewayCert = z.object({
clientCertPem: z.string(),
subjectDN: z.string(),
issuerDN: z.string(),
serialNumber: z.string(),
validity: z.object({
notBefore: z.string(),
notAfter: z.string(),
}),
});

const APIGatewayEventIdentity = z.object({
accessKey: z.string().nullish(),
accountId: z.string().nullish(),
apiKey: z.string().nullish(),
apiKeyId: z.string().nullish(),
caller: z.string().nullish(),
cognitoAuthenticationProvider: z.string().nullish(),
cognitoAuthenticationType: z.string().nullish(),
cognitoIdentityId: z.string().nullish(),
cognitoIdentityPoolId: z.string().nullish(),
principalOrgId: z.string().nullish(),
sourceIp: z.string().ip().optional(),
user: z.string().nullish(),
userAgent: z.string().nullish(),
userArn: z.string().nullish(),
clientCert: APIGatewayCert.nullish(),
});

const APIGatewayEventRequestContext = z
.object({
accountId: z.string(),
apiId: z.string(),
authorizer: z
.object({
claims: z.record(z.string(), z.any()).nullish(),
scopes: z.array(z.string()).nullish(),
})
.nullish(),
stage: z.string(),
protocol: z.string(),
identity: APIGatewayEventIdentity,
requestId: z.string(),
requestTime: z.string(),
requestTimeEpoch: z.number(),
resourceId: z.string().nullish(),
resourcePath: z.string(),
domainName: z.string().nullish(),
domainPrefix: z.string().nullish(),
extendedRequestId: z.string().nullish(),
httpMethod: z.enum([
'GET',
'POST',
'PUT',
'PATCH',
'DELETE',
'HEAD',
'OPTIONS',
]),
path: z.string(),
connectedAt: z.number().nullish(),
connectionId: z.string().nullish(),
eventType: z.enum(['CONNECT', 'MESSAGE', 'DISCONNECT']).nullish(),
messageDirection: z.string().nullish(),
messageId: z.string().nullish(),
routeKey: z.string().nullish(),
operationName: z.string().nullish(),
})
.refine(
(input) => {
return (
!input.messageId || (input.messageId && input.eventType === 'MESSAGE')
);
},
{
message: 'messageId is available only when `eventType` is MESSAGE',
}
);

const APIGatewayProxyEventSchema = z.object({
version: z.string().optional(),
authorizationToken: z.string().optional(),
identitySource: z.string().optional(),
methodArn: z.string().optional(),
type: z.enum(['TOKEN', 'REQUEST']).optional(),
resource: z.string(),
path: z.string(),
httpMethod: z.enum([
dreamorosi marked this conversation as resolved.
Show resolved Hide resolved
'GET',
'POST',
'PUT',
'PATCH',
'DELETE',
'HEAD',
'OPTIONS',
]),
headers: z.record(z.string()).optional(),
dreamorosi marked this conversation as resolved.
Show resolved Hide resolved
queryStringParameters: z.record(z.string()).optional(),
multiValueHeaders: z.record(z.array(z.string())).optional(),
dreamorosi marked this conversation as resolved.
Show resolved Hide resolved
multiValueQueryStringParameters: z.record(z.array(z.string())).optional(),
requestContext: APIGatewayEventRequestContext,
pathParameters: z.record(z.string()).optional().nullish(),
stageVariables: z.record(z.string()).optional().nullish(),
isBase64Encoded: z.boolean().optional(),
body: z.string().optional(),
dreamorosi marked this conversation as resolved.
Show resolved Hide resolved
});

export { APIGatewayProxyEventSchema, APIGatewayCert };
Loading
Loading