A very basic ETL (extract, transform, load) pipeline for covid stats in Austria

Functions in the extract namespace handle loading & parsing of datasources. Currently implemented are HTML scrapers (loaded into enlive), a simple fetcher & parser for JSONp sources, and a JSON fetcher & parser.

The transform ns contains functions that expect output from extract-scrapers and turn them into a common stats data format: A hashmap with location identifiers (:at for Austria, :wien for Vienna, etc) for keys, and a hashmap of statistics for values. A quick peek into how that map looks like: looks like this:

  {:at {:cases xxx
        :tests xxx
        :tdouble xxx}
   :wien {:cases xxx
          :tdouble xxx}
     
     ...
  }

All calculated stats are combined and, along with an extracted timestamp, later be sent to targets. Currently supported exports:

• Google Sheets
• json file dump

Setup

If you want use the Google sheets export, you'll need a Google Service Account. Here are instructions on how to create one. To grant write access to your service account, you add its "email address" as a collaborator directly within the sheet.

This project uses deps.edn. Run the scraper with clj -m core, or build an uberjar clj -Auberjar.