Skip to content

Instruction file contents

Jump to bottom
Rose Pearson edited this page Sep 25, 2023 · 92 revisions

The processor module classes accept the contents of a JSON instruction file as a dictionary. This page briefly details the expected structure and supported keywords for each stage of the GeoFabrics framework before providing links to different example instruction files. A full example instruction file is shown below.

image

Basic structure

The instruction file has the following basic structure, where the output, data_paths, general and datasets keywords are the only top-level keywords expected by the RawLidarDemGenerator, HydrologicDemGenerator and RoughnessLengthGenerator classes. While the RiverBathymetryGenerator, and WaterwayBedElevationEstimator classes each expect the additional keyword rivers, and drains respectively.

{
    "output": {    
        "crs": { "horizontal": 2193, "vertical": 7839 },
        "grid_params": { "resolution": 10 },
    },
    "data_paths": { "cache_path": "path/to/cache", "subfolder": "run_name", "result_dem": results_dem.nc",
                    "extents": "catchment.geojson" },
    "datasets": { "lidar": { "open_topography": "True" }, 
                  "vector": {"linz": { "land": { "layers": [51153], "type": "GEOMETRY" }, "key": "YOUR_KEY_HERE" },
                  "raster": {"linz": {"coarse_dems": {"layers": [51768]}, "key": "YOUR_KEY_HERE" )
    }
    "general": { ... }
}

Instruction files and runner.py

The runner.py entry point module decides which processor classes to run based on the top-level keyword(s) of an instruction file. The keyword options are default, rivers, drains, dem and roughness. The default values are used to populate any missing entries in the other keywords. The other keywords respectively call through to the RiverBathymetryGenerator, DrainBathymetryGenerator, LidarDemGenerator and HydrologicDemGenerator, and RoughnessLengthGenerator. If multiple keywords are included in the instruction file, each related processor class is called in the order of rivers, drains, dem and roughness.

Below is an example of a single instruction file which will trigger the RiverBathymetryGenerator, DrainBathymetryGenerator and LidarDemGenerator classes to run in turn and produce a DEM with ocean, river and drain bathymetry included.

{ 
"measured": { 
    "output": { ... },
    "data_paths": { ... },
    "general": { ... },
},
"measured": { 
    "output": { ... },
    "data_paths": { ... },
    "measured": { ... },
},
"rivers":  {
    "output": { ... },
    "data_paths": { ... },
    "datasets": { ... },
    "general": { ... },
    "rivers":  { ... }
    },
"waterways":   {
    "output": { ... },
    "data_paths": { ... },
    "datasets": { ... },
    "general": { ... },
    "drains":  { ... }   
    },
"dem": {
    "output": { ... },
    "data_paths": { ... },
    "datasets": { ... },
    "general": { ... }
    },
"roughness": {
    "output": { ... },
    "data_paths": { ... },
    "datasets": { ... },
    "general": { ... }
    }
}

Detailed options

The options for each section are detailed in different sub-pages:

Benchmarking [only used by benchmarking.py]

The benchmarking section is required if running benchmarking.py. It defines the numbers_of_cores and chunk_sizes to iterate over when producing a plot of execution times for various processing settings.

  • numbers_of_cores [Required list] - A list of integers that a cycled through for each number_of_cores and defines the chunk size applied. This number should not exceed the number of cores on the machine or allocated in a distributed system.
  • chunk_sizes [Required list] - A list of integers that a cycled through for each number_of_cores and defines the chunk size applied.
  • title [Required str] - Defines the title of the plot of execution time against processing settings.
  • delete_dems [Required bool] - If True the DEMs generated in each run are deleted.

Examples

Check out unit test instruction files for each test for examples using the instruction keywords. A table detailing the rough contents of each test can be found at Table of all tests. Some select tests are pulled out below.

Example - Create a DEM from remote LiDAR and vector data

Example - Create a DEM from local LiDAR and vector data

Example - Estimate river bathymetry values

Example - Estimate drain bed elevations

Example - Create a DEM from remote LiDAR and vector data and estimated river and drain bed elevations

Home

Clone this wiki locally