-
Notifications
You must be signed in to change notification settings - Fork 7
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
CLOUDP-273211: improve error handling and documentation for postman collection generation #252
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,6 +1,7 @@ | ||
export OPENAPI_FOLDER=./openapi | ||
export TMP_FOLDER=./tmp | ||
export FULL_OPENAPI_FOLDER=../../openapi/v2/ | ||
ENV_FILE = ./local.env | ||
|
||
default: build | ||
|
||
|
@@ -44,3 +45,16 @@ compare_and_send_forks: compare_forks send_forks | |
.PHONY: clean | ||
clean: | ||
rm ./openapi/*; rm ./tmp/* | ||
|
||
# Command to load local environment variables from file | ||
load-env: | ||
@if [ -f $(ENV_FILE) ]; then \ | ||
echo "Loading environment variables from $(ENV_FILE)..."; \ | ||
for var in $$(grep -v '^#' $(ENV_FILE) | cut -d= -f1); do \ | ||
echo "Loaded: $$var"; \ | ||
done; \ | ||
export $$(grep -v '^#' $(ENV_FILE) | xargs); \ | ||
else \ | ||
echo "Error: $(ENV_FILE) file not found."; \ | ||
exit 1; \ | ||
fi | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. added for local setup of env vars from local.env file There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. can this be done by
Side note. As we discussed before, I think we have overgrown bash use cases. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I was looking to set something simple up for the rare case where we have to do local validation
I was thinking creating a separate script setting env vars was overkill when I could just put it in the Makefile since it'll probably be single use (just a little helper).
You mean an alternative scripting language? iirc we were considering using TS originally for the post-processing, but there were limitations since we'd need Bazel set up, so bash was good enough (right tool for the job on this occasion) There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. If you feel strongly I'm happy to move it to a script (or remove it altogether since we can just set the small number of env vars up manually) There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Yes but having logic for credentials processing in makefile is not best as well. I never have strong opinions. Only flagging some common pattern. Approved PR. |
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -9,10 +9,11 @@ set -euo pipefail | |
# OPENAPI_FILE_NAME - name of the openapi specification file | ||
# OPENAPI_FOLDER - folder where openapi file is saved | ||
# TMP_FOLDER - folder for temporary files during transformations | ||
# TOGGLE_USE_ENVIRONMENT_AUTH - bool for if auth variables are stored at the environment or collection level | ||
# TOGGLE_INCLUDE_BODY - bool for if generated bodies should be removed or kept | ||
# VERSION_FILE_NAME - name of the file where the current version is stored | ||
# DESCRIPTION_FILE - name for the markdown description file | ||
# TOGGLE_INCLUDE_BODY - bool for if generated bodies should be removed or kept | ||
# TOGGLE_ADD_DOCS_LINKS - updates requests with corresponding docs links | ||
# TOKEN_URL_ENV - client credentials auth path to set at the environment level, will not be set if unpopulated | ||
# BASE_URL - the default base url the Postman Collection will use | ||
######################################################### | ||
|
||
|
@@ -22,17 +23,17 @@ OPENAPI_FILE_NAME=${OPENAPI_FILE_NAME:-"atlas-api.json"} | |
OPENAPI_FOLDER=${OPENAPI_FOLDER:-"../openapi"} | ||
TMP_FOLDER=${TMP_FOLDER:-"../tmp"} | ||
VERSION_FILE_NAME=${VERSION_FILE_NAME:-"version.txt"} | ||
|
||
DESCRIPTION_FILE=${DESCRIPTION_FILE:-"../collection-description.md"} | ||
|
||
TOGGLE_USE_ENVIRONMENT_AUTH=${TOGGLE_USE_ENVIRONMENT_AUTH:-true} | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Removed this one as we will always use env var auth |
||
TOGGLE_INCLUDE_BODY=${TOGGLE_INCLUDE_BODY:-true} | ||
TOGGLE_ADD_DOCS_LINKS=${TOGGLE_ADD_DOCS_LINKS:-true} | ||
TOKEN_URL_ENV=${TOKEN_URL_ENV:-""} | ||
|
||
current_api_revision=$(<"$OPENAPI_FOLDER/$VERSION_FILE_NAME") | ||
|
||
pushd "${TMP_FOLDER}" | ||
|
||
echo "Wrapping Collection in \"collection\" tag" | ||
echo "Wrapping Collection $COLLECTION_FILE_NAME in \"collection\" tag" | ||
jq '{"collection": .}' "$COLLECTION_FILE_NAME" > intermediateCollectionWrapped.json | ||
|
||
echo "Disabling query params by default" | ||
|
@@ -47,48 +48,52 @@ jq 'del(.collection.info._postman_id)' \ | |
echo "Removing circular references" | ||
sed 's/\\"value\\": \\"<Circular reference to #[^>"]* detected>\\"//g' intermediateCollectionNoPostmanID.json > intermediateCollectionNoCircular.json | ||
|
||
echo "Updating name with version" | ||
echo "Updating name with version $current_api_revision" | ||
jq --arg api_version "$current_api_revision" \ | ||
'.collection.info.name = ("MongoDB Atlas Administration API " + $api_version)' \ | ||
intermediateCollectionNoCircular.json > intermediateCollectionWithName.json | ||
|
||
echo "Adding Collection description" | ||
echo "Adding Collection description to $DESCRIPTION_FILE" | ||
description=$(<"$DESCRIPTION_FILE") | ||
jq --arg desc "$description" \ | ||
'.collection.info.description.content = $desc' \ | ||
intermediateCollectionWithName.json > intermediateCollectionWithDescription.json | ||
|
||
echo "Updating baseUrl" | ||
echo "Adding baseUrl env $BASE_URL" | ||
jq --arg base_url "$BASE_URL" \ | ||
'.collection.variable[0].value = $base_url' \ | ||
intermediateCollectionWithDescription.json > intermediateCollectionWithBaseURL.json | ||
|
||
echo "Adding links to docs" | ||
cp intermediateCollectionWithBaseURL.json intermediateCollectionWithLinks.json | ||
if [ "$TOGGLE_ADD_DOCS_LINKS" = "true" ]; then | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Created this toggle for local testing as this part of the script is long-running |
||
echo "Adding links to docs for each request" | ||
cp intermediateCollectionWithBaseURL.json intermediateCollectionWithLinks.json | ||
|
||
# Store all paths to requests. The summary field is the same as the title in the collection | ||
paths=$(jq 'path(.. | objects | select(has("summary"))) | @sh' "$OLDPWD"/"$OPENAPI_FOLDER"/"$OPENAPI_FILE_NAME") | ||
declare -a paths_array="($paths)" | ||
# Store all paths to requests. The summary field is the same as the title in the collection | ||
paths=$(jq 'path(.. | objects | select(has("summary"))) | @sh' "$OLDPWD"/"$OPENAPI_FOLDER"/"$OPENAPI_FILE_NAME") | ||
declare -a paths_array="($paths)" | ||
|
||
for path in "${paths_array[@]}"; do | ||
declare -a single_path_array="($path)" | ||
path_json=$(jq -n '$ARGS.positional' --args "${single_path_array[@]}") | ||
for path in "${paths_array[@]}"; do | ||
declare -a single_path_array="($path)" | ||
path_json=$(jq -n '$ARGS.positional' --args "${single_path_array[@]}") | ||
|
||
# Use the path to get all the information about this request without searching | ||
requestInfo=$(jq --argjson path "$path_json" 'getpath($path)' "$OLDPWD"/"$OPENAPI_FOLDER"/"$OPENAPI_FILE_NAME") | ||
# Use the path to get all the information about this request without searching | ||
requestInfo=$(jq --argjson path "$path_json" 'getpath($path)' "$OLDPWD"/"$OPENAPI_FOLDER"/"$OPENAPI_FILE_NAME") | ||
|
||
title=$(echo "$requestInfo" | jq -r '.summary') | ||
operationId=$(echo "$requestInfo" | jq -r '.operationId') | ||
tag=$(echo "$requestInfo" | jq -r '.tags[0]' | tr " " "-") | ||
title=$(echo "$requestInfo" | jq -r '.summary') | ||
operationId=$(echo "$requestInfo" | jq -r '.operationId') | ||
tag=$(echo "$requestInfo" | jq -r '.tags[0]' | tr " " "-") | ||
|
||
url="https://mongodb.com/docs/atlas/reference/api-resources-spec/v2/#tag/${tag}/operation/$operationId" | ||
url="https://mongodb.com/docs/atlas/reference/api-resources-spec/v2/#tag/${tag}/operation/$operationId" | ||
|
||
# Search the collection for the request with the matching name. Add the link to its description | ||
jq --arg title "$title" --arg url "$url" \ | ||
'first(.collection.item[].item[].request | select(.name == $title).description.content) += "\n\nFind out more at " + $url' \ | ||
intermediateCollectionWithLinks.json > tmp.json && mv tmp.json intermediateCollectionWithLinks.json | ||
# Search the collection for the request with the matching name. Add the link to its description | ||
jq --arg title "$title" --arg url "$url" \ | ||
'first(.collection.item[].item[].request | select(.name == $title).description.content) += "\n\nFind out more at " + $url' \ | ||
intermediateCollectionWithLinks.json > tmp.json && mv tmp.json intermediateCollectionWithLinks.json | ||
|
||
done | ||
done | ||
else | ||
cp intermediateCollectionWithBaseURL.json intermediateCollectionWithLinks.json | ||
fi | ||
|
||
# Togglable features | ||
if [ "$TOGGLE_INCLUDE_BODY" = "false" ]; then | ||
|
@@ -107,11 +112,10 @@ else | |
cp intermediateCollectionWithLinks.json intermediateCollectionPostBody.json | ||
fi | ||
|
||
if [ "$TOGGLE_USE_ENVIRONMENT_AUTH" = "false" ]; then | ||
echo "Adding auth variables" | ||
jq '.collection.variable += [{"key": "digestAuthUsername", "value": "<string>"}, | ||
{"key": "digestAuthPassword", "value": "<string>"}, | ||
{"key": "realm", "value": "<string>"}]' intermediateCollectionPostBody.json > "$COLLECTION_TRANSFORMED_FILE_NAME" | ||
if [ "$TOKEN_URL_ENV" != "" ]; then | ||
echo "Adding client credentials auth url variable $TOKEN_URL_ENV" | ||
jq --arg token_url "$TOKEN_URL_ENV" '.collection.variable += [{"key": "clientCredentialsTokenUrl", "value": $token_url}]' \ | ||
intermediateCollectionPostBody.json > "$COLLECTION_TRANSFORMED_FILE_NAME" | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Added to set client credentials env var once available |
||
else | ||
cp intermediateCollectionPostBody.json "$COLLECTION_TRANSFORMED_FILE_NAME" | ||
fi | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -28,6 +28,9 @@ pushd "${OPENAPI_FOLDER}" | |
current_collection_name="MongoDB Atlas Administration API ${current_api_revision}" | ||
|
||
echo "Fetching list of current collections" | ||
echo "curl -o ${COLLECTIONS_LIST_FILE} | ||
--location 'https://api.getpostman.com/collections?workspace=${WORKSPACE_ID}' | ||
--header 'X-API-Key: **********'" | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Added logs to help with troubleshooting when observing CD runs |
||
curl --show-error --fail --silent -o "${COLLECTIONS_LIST_FILE}" \ | ||
--location "https://api.getpostman.com/collections?workspace=${WORKSPACE_ID}" \ | ||
--header "X-API-Key: ${POSTMAN_API_KEY}" | ||
|
@@ -37,15 +40,25 @@ collection_exists=$(jq '.collections | any(.name=="'"${current_collection_name}" | |
if [ "$collection_exists" = "false" ]; then | ||
# Create new collection | ||
echo "Creating new remote collection ${current_collection_name}" | ||
echo "curl -o ${COLLECTIONS_LIST_FILE} | ||
--location 'https://api.getpostman.com/collections?workspace=${WORKSPACE_ID}' | ||
--header 'Content-Type: application/json' | ||
--header 'X-API-Key: **********' | ||
--data ${collection_transformed_path}" | ||
curl --show-error --fail --retry 5 --retry-all-errors --silent \ | ||
--location "https://api.getpostman.com/collections?workspace=${WORKSPACE_ID}" \ | ||
--header "Content-Type: application/json" \ | ||
--header "X-API-Key: ${POSTMAN_API_KEY}" \ | ||
--data "@${collection_transformed_path}" | ||
--data "@${collection_transformed_path}" \ | ||
|
||
else | ||
# Find collection ID and update collection | ||
echo "Updating remote collection ${current_collection_name}" | ||
echo "curl --request PUT | ||
--location 'https://api.getpostman.com/collections/${collection_id}' | ||
--header 'Content-Type: application/json' | ||
--header 'X-API-Key: **********' | ||
--data ${collection_transformed_path}" | ||
collection_id=$(jq -r '.collections | map(select(.name=="'"${current_collection_name}"'").id)[0]' "${COLLECTIONS_LIST_FILE}") | ||
curl --show-error --fail --retry 5 --retry-all-errors --silent --request PUT \ | ||
--location "https://api.getpostman.com/collections/${collection_id}" \ | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Alternatively (if we want to export env vars):