Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Write initial CSV-W Generation Pipeline #149

Closed
6 tasks done
robons opened this issue Aug 23, 2021 · 1 comment
Closed
6 tasks done

Write initial CSV-W Generation Pipeline #149

robons opened this issue Aug 23, 2021 · 1 comment
Assignees

Comments

@robons
Copy link
Contributor

robons commented Aug 23, 2021

Rough steps:

  • Run the python script to generate the tidy CSV & scraper.json file.
  • Run the csvqb CLI to generate the CSV-W taking the tidy CSV, info.json & scraper.json as inputs.
    • As mentioned in Issue Create CLI for csvqb #108 there will be a command line option for the CSV-qb CLI to force outputting validation errors into a file called validation-errors.json. Ensure that this is kept as an artifact of this pipeline.
  • Run csvlint on the outputted CSV-Ws to ensure they're valid.
  • Ensure CSV-W (and all associated codelist dependencies) are registered as build artifacts.
  • Test/Ensure the build is marked as a failure if validation errors fail.
  • Make sure to write units tests to cover this pipeline (so we can debug locally) - a few rough integration tests to ensure the main branches/paths of the code succeess. Maybe we could check one of these for some of the RDF outputs we expect?

N.B. don't go crazy on the tests since we'd be duplicate a lot of testing we'll already have elsewhere.

@robons robons self-assigned this Oct 8, 2021
@robons
Copy link
Contributor Author

robons commented Oct 11, 2021

Some documentation for this part of the pipeline at https://collaborate2.ons.gov.uk/confluence/display/PHEM/Jenkins+-+Building+a+CSV-W

@robons robons closed this as completed Oct 18, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant