Bigquery table generator

Creates one or more tables every day as specified by the configuration file.

Note that this is less useful now that Bigquery provides a way to automatically create tables during streaming inserts.

Table names will be of the format [tableNamePrefix]YYYYMMDD for use in Table Wildcard Functions.

Configuration

The configuration file (src/main/resources/config.json) specifies one or more table schemas, each having one or more fields:

{
  "includeCurrentDay": true,
  "numberOfDays": 3,
  "tables": [
    {
      "projectId": "project-id",
      "dataset": "dataset",
      "tableNamePrefix": "prefix",
      "fields": [
        {
          "name": "field1",
          "type": "string",
          "nullable": false
        }
      ]
    },
  ]
}

Usage

• Set up the configuration file as required
• Deploy to an appengine project having service account permissions to the target table project

mvn appengine:update -Dappengine.appId=project-app-id -Ddeploy.version=[version-num] -Ddeploy.module=[default | module-name]

Unit Tests

mvn test

Integration Tests

mvn verify

Note: Requires a json service account file. The file path should be specified by the environment variable GOOGLE_APPLICATION_CREDENTIALS