SEAB-6360: download github event from s3 #491

hyunnaye · 2024-05-09T14:19:36Z

Description
This PR creates a module githubdelivery and adds a command download-event to return the github payload from s3 from its key.

Review Instructions

It was the easiest to run on CloudShell
Steps I used was

Install Java sudo yum install java-17-amazon-corretto
clone the repo
Build only the githubdelivery module by running ./mvnw install -pl githubdelivery -am
Configure your github-delivery.config for QA (template in templates folder)
Go to the module and run java -jar target/githubdelivery-1.16.0-SNAPSHOT.jar download-event -k "2024-05-15/004701ac-1307-11ef-94b7-6b9aec991740" and verify the payload is printed on the console.

Issue
https://ucsc-cgl.atlassian.net/browse/SEAB-6360

Security
If there are any concerns that require extra attention from the security team, highlight them here.

Please make sure that you've checked the following before submitting your pull request. Thanks!

Check that you pass the basic style checks and unit tests by running mvn clean install in the project that you have modified (until https://ucsc-cgl.atlassian.net/browse/SEAB-5300 adds multi-module support properly)
Ensure that the PR targets the correct branch. Check the milestone or fix version of the ticket.
If you are changing dependencies, check with dependabot to ensure you are not introducing new high/critical vulnerabilities
If this PR is for a user-facing feature, create and link a documentation ticket for this feature (usually in the same milestone as the linked issue). Style points if you create a documentation PR directly and link that instead.

codecov · 2024-05-09T14:30:47Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 48.42%. Comparing base (98c476c) to head (07d957b).

Additional details and impacted files

@@            Coverage Diff             @@
##             develop     #491   +/-   ##
==========================================
  Coverage      48.42%   48.42%           
  Complexity       305      305           
==========================================
  Files             46       46           
  Lines           2480     2480           
  Branches         200      200           
==========================================
  Hits            1201     1201           
  Misses          1187     1187           
  Partials          92       92

Flag	Coverage Δ
metricsaggregator	`39.63% <ø> (ø)`
toolbackup	`24.87% <ø> (ø)`
tooltester	`18.58% <ø> (ø)`
topicgenerator	`21.08% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

coverbeck · 2024-05-21T23:12:29Z

It was the easiest to run on CloudWatch.

CloudShell :) Too may terms with "Cloud" in them...

githubdelivery/src/main/java/io/dockstore/githubdelivery/package-info.java

githubdelivery/src/main/java/io/dockstore/githubdelivery/GithubDeliveryS3Client.java

githubdelivery/templates/github-delivery.config

githubdelivery/src/main/java/io/dockstore/githubdelivery/GithubDeliveryS3Client.java

kathy-t · 2024-05-22T13:42:43Z

2. clone the repo

3. Build only the `githubdelivery` module by running `./mvnw install -pl githubdelivery -am`

You could perform a release once this is merged to develop so the reviewer can just wget the JAR from artifactory

githubdelivery/src/main/java/io/dockstore/githubdelivery/package-info.java

githubdelivery/src/main/java/io/dockstore/githubdelivery/GithubDeliveryConfig.java

githubdelivery/pom.xml

githubdelivery/src/main/java/io/dockstore/githubdelivery/GithubDeliveryCommandLineArgs.java

githubdelivery/dependency-reduced-pom.xml

kathy-t · 2024-05-22T13:54:01Z

pom.xml

Commenting here but unrelated to this file: should also add this module to the CircleCI file so that it gets built

dockstore-support/.circleci/config.yml

Line 31 in 98c476c

jobs:

kathy-t · 2024-05-22T13:57:48Z

githubdelivery/src/main/java/io/dockstore/githubdelivery/GithubDeliveryCommandLineArgs.java

+    @Parameters(commandNames = { "download-event" }, commandDescription = "Download github event from S3 bucket using key.")
+    public static class DownloadEventCommand {
+
+        @Parameter(names = {"-k", "--key"}, description = "The key of the event in bucket. Format should be YYYY-MM-DD/deliveryid")


We can cross this bridge when we get there, but it could be useful to submit by date

Second, can create follow-up ticket to avoid this PR from dragging too long

denis-yuen

Some clean-up items
Please link created tickets to the comments so we know what was done here and what remains to be done

githubdelivery/pom.xml

denis-yuen · 2024-05-22T14:32:09Z

githubdelivery/src/main/java/io/dockstore/githubdelivery/GithubDeliveryCommandLineArgs.java

+    @Parameters(commandNames = { "download-event" }, commandDescription = "Download github event from S3 bucket using key.")
+    public static class DownloadEventCommand {
+
+        @Parameter(names = {"-k", "--key"}, description = "The key of the event in bucket. Format should be YYYY-MM-DD/deliveryid")


Second, can create follow-up ticket to avoid this PR from dragging too long

denis-yuen · 2024-05-22T14:33:02Z

githubdelivery/src/main/java/io/dockstore/githubdelivery/GithubDeliveryS3Client.java

+            final GithubDeliveryS3Client githubDeliveryS3Client = new GithubDeliveryS3Client(githubDeliveryConfig.getS3Config().bucket());
+
+            if ("download-event".equals(jCommander.getParsedCommand())) {
+                System.out.println(githubDeliveryS3Client.getGitHubDeliveryEventByKey(downloadEventCommand.getBucketKey()));


Should use logging framework, can be follow-up ticket

svonworl · 2024-05-22T18:45:37Z

githubdelivery/src/main/java/io/dockstore/githubdelivery/GithubDeliveryS3Client.java

+    public static void main(String[] args) throws IOException {
+        final GithubDeliveryCommandLineArgs commandLineArgs = new GithubDeliveryCommandLineArgs();
+        final JCommander jCommander = new JCommander(commandLineArgs);
+        final DownloadEventCommand downloadEventCommand = new DownloadEventCommand();
+        jCommander.addCommand(downloadEventCommand);
+
+        try {
+            jCommander.parse(args);
+        } catch (MissingCommandException e) {
+            jCommander.usage();
+            if (e.getUnknownCommand().isEmpty()) {
+                LOG.error("No command entered");
+            } else {
+                LOG.error("Unknown command");
+            }
+            exceptionMessage(e, "The command is missing", GENERIC_ERROR);
+        } catch (ParameterException e) {
+            jCommander.usage();
+            exceptionMessage(e, "Error parsing arguments", GENERIC_ERROR);
+        }
+
+        if (jCommander.getParsedCommand() == null || commandLineArgs.isHelp()) {
+            jCommander.usage();


This code section, which creates the JCommander object and handles errors and prints usage information, appears to repeat in some of our other utilities. If, upon inspection, we find that it's repeated more-or-less verbatim, we might consider moving it to a helper (at some time, doesn't need to happen now). Then, we could do something like:

final GithubDeliveryCommandLineArgs commandLineArgs = new GithubDeliveryCommandLineArgs(); final DownloadEventCommand downloadEventCommand = new DownloadEventCommand(); final JCommander jCommander = JCommanderHelper.createAndParse(commandLineArgs, downloadEventCommand);

I'll add this to the followup :)

githubdelivery/src/main/java/io/dockstore/githubdelivery/GithubDeliveryConfig.java

githubdelivery/src/main/java/io/dockstore/githubdelivery/package-info.java

githubdelivery/src/main/resources/logback.xml

githubdelivery/src/main/java/io/dockstore/githubdelivery/GithubDeliveryS3Client.java

hyunnaye · 2024-05-22T20:05:39Z

I created ticket https://ucsc-cgl.atlassian.net/browse/SEAB-6452 For followup

denis-yuen

minor redundancy

githubdelivery/pom.xml

sonarcloud · 2024-05-23T16:42:23Z

Quality Gate failed

Failed conditions
0.0% Coverage on New Code (required ≥ 80%)
3.8% Duplication on New Code (required ≤ 3%)

See analysis details on SonarCloud

test

c21b8c1

hyunnaye self-assigned this May 9, 2024

hyunnaye added 9 commits May 9, 2024 12:33

test

fecb0bb

build

9c91fb2

progress

78c1704

progress

ecc2c04

checkstyle

ba247f7

oops

b8aefec

progress

44c2eb7

progress

29260f4

pom

9db74e6

hyunnaye changed the title ~~test~~ SEAB-6360: download github event from s3 May 21, 2024

hyunnaye requested review from coverbeck, denis-yuen, kathy-t and svonworl May 21, 2024 20:46

hyunnaye marked this pull request as ready for review May 21, 2024 20:46

add config template

16de5e3

coverbeck approved these changes May 21, 2024

View reviewed changes

kathy-t reviewed May 22, 2024

View reviewed changes

denis-yuen reviewed May 22, 2024

View reviewed changes

svonworl reviewed May 22, 2024

View reviewed changes

hyunnaye added 2 commits May 22, 2024 15:47

pr review feedback

64c8a50

pr review feedback

dc42906

hyunnaye requested review from denis-yuen and svonworl May 22, 2024 20:05

hyunnaye requested a review from kathy-t May 22, 2024 20:05

svonworl approved these changes May 22, 2024

View reviewed changes

denis-yuen approved these changes May 23, 2024

View reviewed changes

githubdelivery/pom.xml Outdated Show resolved Hide resolved

githubdelivery/pom.xml Outdated Show resolved Hide resolved

hyunnaye added 2 commits May 23, 2024 11:37

pr review feedback

1167811

pr review feedback

07d957b

kathy-t approved these changes May 24, 2024

View reviewed changes

hyunnaye merged commit 081311e into develop May 24, 2024
10 of 11 checks passed

hyunnaye deleted the feature/SEAB-6360 branch May 24, 2024 14:04

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SEAB-6360: download github event from s3 #491

SEAB-6360: download github event from s3 #491

hyunnaye commented May 9, 2024 •

edited

Loading

codecov bot commented May 9, 2024 •

edited

Loading

coverbeck commented May 21, 2024

kathy-t commented May 22, 2024

kathy-t May 22, 2024

kathy-t May 22, 2024

denis-yuen May 22, 2024

denis-yuen left a comment •

edited

Loading

denis-yuen May 22, 2024

denis-yuen May 22, 2024

svonworl May 22, 2024

hyunnaye May 22, 2024

hyunnaye commented May 22, 2024

denis-yuen left a comment

sonarcloud bot commented May 23, 2024

SEAB-6360: download github event from s3 #491

SEAB-6360: download github event from s3 #491

Conversation

hyunnaye commented May 9, 2024 • edited Loading

codecov bot commented May 9, 2024 • edited Loading

Codecov Report

coverbeck commented May 21, 2024

kathy-t commented May 22, 2024

kathy-t May 22, 2024

Choose a reason for hiding this comment

kathy-t May 22, 2024

Choose a reason for hiding this comment

denis-yuen May 22, 2024

Choose a reason for hiding this comment

denis-yuen left a comment • edited Loading

Choose a reason for hiding this comment

denis-yuen May 22, 2024

Choose a reason for hiding this comment

denis-yuen May 22, 2024

Choose a reason for hiding this comment

svonworl May 22, 2024

Choose a reason for hiding this comment

hyunnaye May 22, 2024

Choose a reason for hiding this comment

hyunnaye commented May 22, 2024

denis-yuen left a comment

Choose a reason for hiding this comment

sonarcloud bot commented May 23, 2024

Quality Gate failed

hyunnaye commented May 9, 2024 •

edited

Loading

codecov bot commented May 9, 2024 •

edited

Loading

denis-yuen left a comment •

edited

Loading