Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(glue): Job construct #12506

Merged
merged 67 commits into from
Sep 8, 2021
Merged

feat(glue): Job construct #12506

merged 67 commits into from
Sep 8, 2021

Commits on Jul 11, 2021

  1. feat(aws-glue): add Job construct (#12443)

    closes #12443
    
    - Add GlueVersion, WorkerType, JobCommandName enum-like classes for future-proofing these values
    - Add PythonVersion as a normal enum as it's less likely to change
    - Add Job construct
      - CloudFormation base resource is a complex resource with almost all properties being optional and allowed in specific combinations based on command, glue version and worker types
      - It supports multiple job types e.g. ETL, Streaming and Python Shell and languages e.g. Scala and Python. This requires different combinations of props
      - One possibility is to try to create different constructs for the different types but I've opted to do just one base on resembling CloudFormation which does not prevent us from introducing more specialized types later
      - Documentation is not accurate, based on integ test deployments which led me to not having any assertions on valid props combinations
        - https://docs.aws.amazon.com/glue/latest/dg/add-job.html
        - https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-jobs-job.html
      - Not exposing some CloudFormation props
        - allocatedCapacity as it's deprecated in favour of maxCapacity
        - logUri which is not used and is reserved for future use
    humanzz committed Jul 11, 2021
    Configuration menu
    Copy the full SHA
    c158a05 View commit details
    Browse the repository at this point in the history
  2. support job's event rules and rule-based metrics

    - glue job does not emit success or failure metrics to cloudwatch metrics. Instead, it emits events to cloudwatch events
    - add JobEventState enum for known job states in https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/EventTypes.html#glue-event-types
    - add utility methods to create event rules and cloudwatch metrics based on those roles supporting the most common use cases; success, failure and timeout
    humanzz committed Jul 11, 2021
    Configuration menu
    Copy the full SHA
    9ea48b6 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    638129c View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    b99dbce View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    06b4d55 View commit details
    Browse the repository at this point in the history

Commits on Jul 27, 2021

  1. address some comments

    - introduce JobBase to contain common logic between Job and Import
      - JobBase now handles CloudWatch Event Rules and Rule-based metrics methods
    - address the rename comments
    - use the private constructor + static of() pattern where relevant
    - Make JobProps.glueVersion required
    humanzz committed Jul 27, 2021
    Configuration menu
    Copy the full SHA
    6bcdad8 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    e0eb941 View commit details
    Browse the repository at this point in the history
  3. improve docs

    humanzz committed Jul 27, 2021
    Configuration menu
    Copy the full SHA
    b2d866f View commit details
    Browse the repository at this point in the history

Commits on Jul 28, 2021

  1. Configuration menu
    Copy the full SHA
    29ff157 View commit details
    Browse the repository at this point in the history

Commits on Jul 29, 2021

  1. Configuration menu
    Copy the full SHA
    c615c8e View commit details
    Browse the repository at this point in the history

Commits on Jul 30, 2021

  1. Configuration menu
    Copy the full SHA
    c31d7e4 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    f05afa7 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    5a46dc5 View commit details
    Browse the repository at this point in the history
  4. address more comments

    humanzz committed Jul 30, 2021
    Configuration menu
    Copy the full SHA
    f59d688 View commit details
    Browse the repository at this point in the history

Commits on Aug 2, 2021

  1. Configuration menu
    Copy the full SHA
    fba208d View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    6032807 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    8b74200 View commit details
    Browse the repository at this point in the history
  4. drop JobSpecialArgumentNames

    humanzz committed Aug 2, 2021
    Configuration menu
    Copy the full SHA
    98c19df View commit details
    Browse the repository at this point in the history

Commits on Aug 11, 2021

  1. introduce JobExecutable and refactor accordingly

    - introduce JobExecutable
    - drop JobCommand
    - rename JobCommandName to JobType
    humanzz committed Aug 11, 2021
    Configuration menu
    Copy the full SHA
    933ebcd View commit details
    Browse the repository at this point in the history

Commits on Aug 12, 2021

  1. Configuration menu
    Copy the full SHA
    6b3d31e View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    2f02798 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    52ae192 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    fe08089 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    55b5ee4 View commit details
    Browse the repository at this point in the history

Commits on Aug 13, 2021

  1. Configuration menu
    Copy the full SHA
    3235ae7 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    ee74a8a View commit details
    Browse the repository at this point in the history

Commits on Aug 20, 2021

  1. Configuration menu
    Copy the full SHA
    1367813 View commit details
    Browse the repository at this point in the history

Commits on Aug 24, 2021

  1. Configuration menu
    Copy the full SHA
    c3c9e80 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    0ff9d01 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    cd27ef7 View commit details
    Browse the repository at this point in the history

Commits on Aug 30, 2021

  1. Configuration menu
    Copy the full SHA
    4353f22 View commit details
    Browse the repository at this point in the history
  2. address smaller comments

    humanzz committed Aug 30, 2021
    Configuration menu
    Copy the full SHA
    fb7e376 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    818be70 View commit details
    Browse the repository at this point in the history

Commits on Aug 31, 2021

  1. address metric comments

    humanzz committed Aug 31, 2021
    Configuration menu
    Copy the full SHA
    7072a6f View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    e860f4d View commit details
    Browse the repository at this point in the history
  3. test glue.Code

    humanzz committed Aug 31, 2021
    Configuration menu
    Copy the full SHA
    1068229 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    bf896fa View commit details
    Browse the repository at this point in the history
  5. address some comments

    Co-authored-by: Ben Chaimberg <[email protected]>
    humanzz and BenChaimberg authored Aug 31, 2021
    Configuration menu
    Copy the full SHA
    9f5f85a View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    0f587c9 View commit details
    Browse the repository at this point in the history
  7. address comments

    - move AssetCode directory check to constructor
    - rename SparkUIConfig to SparkUILoggingLocation
    - rename filter to quiet
    - rename jobStateRule to onStateChange and make it public
    - check no reserved arguments used
    - rename JobProps.numberOfWorkers to JobProps.workerCount
    - move JobExecutableConfig to the bottom of the file
    - rename PythonJobExecutableProps to PythonSparkJobExecutableProps
    humanzz committed Aug 31, 2021
    Configuration menu
    Copy the full SHA
    87dee59 View commit details
    Browse the repository at this point in the history
  8. refactor JobExecutableProps

    humanzz committed Aug 31, 2021
    Configuration menu
    Copy the full SHA
    0ded0f2 View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    82c1d98 View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    c81e736 View commit details
    Browse the repository at this point in the history

Commits on Sep 1, 2021

  1. apply suggestions from comments

    Co-authored-by: Ben Chaimberg <[email protected]>
    humanzz and BenChaimberg authored Sep 1, 2021
    Configuration menu
    Copy the full SHA
    8ce0fe8 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    2ed79e1 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    1eb9c20 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    16e3265 View commit details
    Browse the repository at this point in the history
  5. update README

    humanzz committed Sep 1, 2021
    Configuration menu
    Copy the full SHA
    7df0c1f View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    8884ad7 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    13b03e8 View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    2b5f47e View commit details
    Browse the repository at this point in the history

Commits on Sep 3, 2021

  1. tweak tests

    humanzz committed Sep 3, 2021
    Configuration menu
    Copy the full SHA
    bc82d60 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    550b919 View commit details
    Browse the repository at this point in the history
  3. tweak tests #2

    humanzz committed Sep 3, 2021
    Configuration menu
    Copy the full SHA
    70b3e24 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    a500cf2 View commit details
    Browse the repository at this point in the history

Commits on Sep 8, 2021

  1. remove role from IJob

    BenChaimberg committed Sep 8, 2021
    Configuration menu
    Copy the full SHA
    32ba2ae View commit details
    Browse the repository at this point in the history
  2. address some comments

    - update README
    - use const instead of let and beforeEach where relevant
    - assert on stack assets metadata
    - add test case of 2 jobs reusing asset
    - other minor things
    humanzz committed Sep 8, 2021
    Configuration menu
    Copy the full SHA
    babd3ec View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    7379066 View commit details
    Browse the repository at this point in the history
  4. simplify job.test.ts

    humanzz committed Sep 8, 2021
    Configuration menu
    Copy the full SHA
    b62a868 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    80c3f15 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    094929c View commit details
    Browse the repository at this point in the history
  7. update integ.job.ts

    humanzz committed Sep 8, 2021
    Configuration menu
    Copy the full SHA
    f01c0be View commit details
    Browse the repository at this point in the history
  8. fix issues identified trying to run jobs from integ tests

    - fix bug with code bucket permissions
      - Glue role did not have permissions to read from the code bucket which manifested as DeniedAccess exceptions when attempting to run jobs
      - Update Code.bind to take a Job instead of Construct so it can grant it the necessary read permissions
    - Python shell failed saying Glue 2.0 is not supported
      - Console UI for adding a new Python shell jobs shows Glue 1.0 as the only supported version
      - Modified tests to reject using Glue 0.9 but didn't exclude Glue 2.0 or 3.0 for any future support by the new versions
    humanzz committed Sep 8, 2021
    Configuration menu
    Copy the full SHA
    cd2d2ee View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    ea32eab View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    98cc575 View commit details
    Browse the repository at this point in the history
  11. Configuration menu
    Copy the full SHA
    0328615 View commit details
    Browse the repository at this point in the history
  12. Configuration menu
    Copy the full SHA
    6bb18eb View commit details
    Browse the repository at this point in the history