Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix 654 Support Alternative URLs with Production/Staging/Development Hosts Options #656

Open
wants to merge 18 commits into
base: master
Choose a base branch
from

Conversation

ronilan
Copy link
Contributor

@ronilan ronilan commented May 19, 2022

Overview

This pull request enables the use of alternative s3 urls (path style) and s3-compatible backends (e.g. min.io) in conjunction with the development_host, staging_host and production_host options. It fixes #654.

The change in this pull request is non-breaking. It is fully backwards compatible with current way of defining hosts (as string) and bucket/region (as properties of binary).

This pull request comes "on top" of (i.e. includes changes from) #655 (which in turn comes "on top" of #652, which is "on top" of #651, which is "on top" of #648, #649, #650. It's pull requests all the ay down 🐢🐢🐢🐢😉).

Change

Moving forward hosts will be defined as an object with one required property endpoint and additional optional properties: bucket region and s3ForcePathStyle.

Example 1: (S3 Virtual Host URL)

{
  "binary": {
    "host": {
      "endpoint": "https://a-bucket.s3.us-east-1.amazonaws.com",
    }
  }
}

Example 2: (Path Style alternative host)

{
  "binary": {
    "host": {
      "endpoint": "https://play.min.io",
      "bucket": "node-pre-gyp-production",
      "region": "us-east-1",
      "s3ForcePathStyle": true
    }
  }
}

Example 3: (Utilizing Staging/Development)

{
  "binary": {
    "host": {
      "endpoint": "https://my-production-bucket.s3.us-east-1.amazonaws.com",
    },
    "staging_host": {
      "endpoint": "https://play.min.io",
      "bucket": "node-pre-gyp-staging",
      "region": "us-east-1",
      "s3ForcePathStyle": true
    },
  }
}
{
  "binary": {
    "host": {
      "endpoint": "https://dns.over.some.bucket.example.com",
      "bucket": "the.real.bucket.name",
      "region": "us-east-1",
      "s3ForcePathStyle": true
    },
    "staging_host": {
      "endpoint": "https://my-staging-bucket.s3.us-east-1.amazonaws.com",
    },
    "development_host": {
      "endpoint": "https://play.min.io",
      "bucket": "temp-dev-bucket",
      "region": "us-east-1",
      "s3ForcePathStyle": true
    }
  }
}

Note: the object keys are the same as the ones passed to the AWS sdk.

Backwards compatibility

Existing string definitions and existing keys are still supported.

Example 1: (S3 Virtual Host URL)

{
  "binary": {
    "host": "https://a-bucket.s3.us-east-1.amazonaws.com",
  }
}

Example 2: (Alternative S3 URL)

{
  "binary": {
    "host": "https://play.min.io",
    "bucket": "node-pre-gyp-production",
    "region": "us-east-1",
    "s3ForcePathStyle": true
  }
}

Example 3: (Utilizing Staging/Production for S3 Virtual Host URL)

{
  "binary": {
    "staging_host": "https://npg-bucket-staging.s3.us-east-1.amazonaws.com",
    "production_host": "https://npg-bucket-production.s3.us-east-1.amazonaws.com"
  }
}

Implementation

  • Implementation attempts to limit code changes. There was no refactoring done, not even in the vicinity of changed code.
  • Backwards compatibility is archived by adding standarize_config function to versioning. The function is called by validate_config (exported) and mutates the object holding the parsed package.json, so that any hosts (host, staging_host, development_host) are defined as objects with an endpoint key, rather than strings.
  • versioning.evaluate (exported) was adapted to work with object format of the hosts.

Tests

  • Added app1.3 - identical to app1 but using host, staging_host and development_host configuration with an object.
  • Testing of versioning was expanded and refactored.
  • All tests pass

@ronilan ronilan marked this pull request as ready for review May 19, 2022 20:32
@ronilan ronilan changed the title Fix 654 Support Alternative URLs with Production/Staging /Development Hosts Optios Fix 654 Support Alternative URLs with Production/Staging/Development Hosts Options May 19, 2022
lib/mock/http.js Fixed Show fixed Hide fixed
lib/mock/http.js Fixed Show fixed Hide fixed
@ronilan ronilan force-pushed the Fix-Alternative-URLs-with-Staging-Production-Option-654 branch 2 times, most recently from e87f2c2 to bbe0e41 Compare June 29, 2024 07:55
ronilan and others added 18 commits June 30, 2024 14:27
- mocking setup can be more easily used by automated tests and/or manually by developer.
- mock modules moved to lib/mock.
- mock modules are self contained and have a signature similar to that of other modules in the package.
- simplified http mock to remove no longer needed interface.
- modified http mock to support path style bucket access.
- set http mock to work with default npg-mock-bucket bucket.
- separated s3 tests from build tests.
- refactored fetch test and proxy-bcrypt test to work with refactored mock setup.
- set hosts of test apps to point to mock bucket.
- added app1.1 - identical to app1 but using production and staging binary host option.
- added app1.2 - identical to app1 but using explicit host, region, bucket options.
…is versa. Removed previous one.

Fix CodeQL errors.
- Added a GitHub Actions workflow that runs whenever there is a push to the repo.
- Workflow includes two jobs:
    - A matrix job of node versions (10, 12, 14, 16, 18) and operating systems (Linux (ubuntu), Mac and Windows (2019 Enterprise)) that runs all tests against mock and then runs s3 tests against a bucket (located at us-east-1-bucket) specified as a repo secret.
    - A matrix job of and NW.js versions (0.64.0, 0.50.2) and node versions (10, 12, ,14, 16) that runs the NW.js test script.
- Modified `scripts/test-node-webkit.sh` so that it can now accept an NW.js version as input. This allows running the script in a GitHub Actions matrix.
- Modified `test/run.util.js` so that it does not set `--msvs_version=2015` when running in GitHub Actions. This is required because current GitHub Actions runner do not support VS Studio 2015.
- Added npm script command `test:s3` to `package.json`  that runs only the s3 tests. This is required because invoking `npx tape test/s3.test.js` on windows does not work as expected.
- Modified `test/proxy-bcrypt.test.js`. Removed uneeded CI conditionals and modified download directory setup/cleanup. Latter was required due to concurrency issues with running tests on Mac, resulting in uncatchable errors during directory removal originating from `rimraf`.
- Moved logic regarding host selection to versioning where all user defined values from package.json are transformed into command options.
- Moved testing of feature from `run.test.js` to `versioning.test.js`.
- Added `development_host` option. Becomes default option for `publish` `unpublish` when present.
- Changed behavior when alternate hosts are defined. Now `production_host` acts as alias to host. Defining `staging_host` or `development_host` is enough to default `publish` and `unpublish` away from production.
- When a chain of commands that includes `publish` or `unpublish`, when host not specifically set via command line or environment variable, ALL commands in the chain default away from production.
- An invalid `s3_host` option does not result in error and is instead silently ignored.
- Change is backwards compatible with previously valid configurations.
- added a standarize_config function. function is called by validate_config and mutates the object holding the parsed package.json, so that hosts are defined as objects with endpoint key, rather than strings.
- modified versioning.evaluate to work with object format of hosts.
- added app1.3 - identical to app1 but using host, staging_host and development_host configuration with an object.
- expanded and refactored versioning testing.
- modified bucket switching script to work with new package.json format.
…ost option when package.json has only host key specified.
@ronilan ronilan force-pushed the Fix-Alternative-URLs-with-Staging-Production-Option-654 branch from bbe0e41 to 0bf0db9 Compare June 30, 2024 21:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Can't use Alternative S3 URLs with Staging/Production Option
1 participant