Empty output with downloaded data

[luuqh]

Hi,

I have installed ai-models following the instruction, and been able to run GraphCast in online mode with live data from CDS.

However, I am unable to get it done correctly in an offline mode that feeds the downloaded Grib data (using the Bash script below). It can produce graphcast.grib output but size is almost zero.

1. I wonder whether anyone has the same issue and/or know how to fix it?

2. Can anyone share the successful specfile (that includes the exact packages or modules)?

Much appreciated.

672531200
1672552800
Date/hour: 20230101 00
2024-09-01 12:09:44,616 INFO Writing results to graphcast.grib
2024-09-01 12:09:45,069 INFO Writing step 0: 0.3 second.
2024-09-01 12:09:45,142 INFO Writing step 0: 72.1 milliseconds.
2024-09-01 12:09:45,372 INFO Model description: 
GraphCast model at 0.25deg resolution, with 13 pressure levels. This model is
trained on ERA5 data from 1979 to 2017, and fine-tuned on HRES-fc0 data from
2016 to 2021 and can be causally evaluated on 2022 and later years. This model
does not take `total_precipitation_6hr` as inputs and can make predictions in an
operational setting (i.e., initialised from HRES-fc0).

2024-09-01 12:09:45,373 INFO Model license: 
The model weights are licensed under the Creative Commons
Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0). You
may obtain a copy of the License at:
https://creativecommons.org/licenses/by-nc-sa/4.0/.
The weights were trained on ERA5 data, see README for attribution statement.

2024-09-01 12:09:45,373 INFO Loading params/GraphCast_operational - ERA5-HRES 1979-2021 - resolution 0.25 - pressure levels 13 - mesh 2to6 - precipitation output only.npz: 0.2 second.
2024-09-01 12:09:45,373 INFO Building model: 0.2 second.
2024-09-01 12:09:48,594 INFO Creating forcing variables: 3 seconds.
2024-09-01 12:09:49,650 INFO Converting GRIB to xarray: 1 second.
2024-09-01 12:09:49,870 INFO Reindexing: 0.2 second.
2024-09-01 12:09:49,875 INFO Creating training data: 4 seconds.
2024-09-01 12:09:50,214 INFO Extracting input targets: 0.3 second.
2024-09-01 12:09:50,215 INFO Creating input data (total): 4 seconds.
2024-09-01 12:09:50,224 INFO Unable to initialize backend 'cuda': Unable to load cuDNN. Is it installed?
2024-09-01 12:09:50,224 INFO Unable to initialize backend 'rocm': NOT_FOUND: Could not find registered platform with name: "rocm". Available platform names are: CUDA
2024-09-01 12:09:50,225 INFO Unable to initialize backend 'tpu': INTERNAL: Failed to open libtpu.so: libtpu.so: cannot open shared object file: No such file or directory
2024-09-01 12:09:50,225 WARNING CUDA backend failed to initialize: Unable to load cuDNN. Is it installed? (Set TF_CPP_MIN_LOG_LEVEL=0 and rerun for more info.)
/home/corsair/anaconda3/envs/ai-models/lib/python3.10/site-packages/graphcast/autoregressive.py:202: FutureWarning: The return type of `Dataset.dims` will be changed to return a set of dimension names in future, in order to be more consistent with `DataArray.dims`. To access a mapping from dimension names to lengths, please use `Dataset.sizes`.
  scan_length = targets_template.dims['time']
2024-09-01 12:19:59,019 WARNING Skipping gradient checkpointing for sequence length of 1
/home/corsair/anaconda3/envs/ai-models/lib/python3.10/site-packages/graphcast/autoregressive.py:115: FutureWarning: The return type of `Dataset.dims` will be changed to return a set of dimension names in future, in order to be more consistent with `DataArray.dims`. To access a mapping from dimension names to lengths, please use `Dataset.sizes`.
  num_inputs = inputs.dims['time']
2024-09-01 12:29:37,967 INFO Doing full rollout prediction in JAX: 9 minutes 40 seconds.
2024-09-01 12:29:37,989 INFO Converting output xarray to GRIB and saving
2024-09-01 12:29:39,038 INFO Saving output data: 1 second.
2024-09-01 12:29:39,199 INFO Total time: 9 minutes 49 seconds.
2024-09-01 12:29:54,346 INFO Writing results to graphcast.grib
2024-09-01 12:29:55,144 INFO Writing step 0: 0.3 second.
2024-09-01 12:29:55,221 INFO Writing step 0: 76.2 milliseconds.
2024-09-01 12:29:55,666 INFO Model description: 
GraphCast model at 0.25deg resolution, with 13 pressure levels. This model is
trained on ERA5 data from 1979 to 2017, and fine-tuned on HRES-fc0 data from ...

My simple Bash script to run it is

#!/bin/bash

# Start and end dates
start_date="2023-01-01 11:00:00"
end_date="2023-01-01 17:00:00"

# Interval in seconds (6 hours = 21600 seconds)
# interval=21600
interval=3600

# Output folder
input_folder="/home/corsair/work/wait/data/graphcast/input/"
run_folder="/home/corsair/work/wait/data/graphcast/run/"
output_folder="/home/corsair/work/wait/data/graphcast/output/"

# Convert start and end dates to seconds since epoch
start_epoch=$(date -d "$start_date" +%s)
end_epoch=$(date -d "$end_date" +%s)
echo $start_epoch
echo $end_epoch

# Create the output folder if it doesn't exist
mkdir -p $output_folder

# Loop through the date range in 6-hour intervals
current_epoch=$start_epoch
while [ $current_epoch -le $end_epoch ]
do
    # Format the current date and time
    current_date=$(date -u -d "@$current_epoch" +"%Y%m%d")
    current_hour=$(date -u -d "@$current_epoch" +"%H")    
    echo "Date/hour: ${current_date} ${current_hour}"

    # Generate the filename
    filename1="cds_slevel_${current_date}${current_hour}.grib"
    filename2="cds_plevels_${current_date}${current_hour}.grib"
    ln -sf "${input_folder}/${filename1}" "${run_folder}/cds_slevel.grib"
    ln -sf "${input_folder}/${filename2}" "${run_folder}/cds_plevels.grib"

    # Move data from input folder here
    # mv ${input_folder}*${current_date}.grib ./

    for leadtime in {1..6}
    do
        # Execute the command
        ai-models --lead-time ${leadtime} --file ${run_folder}'/cds_*.grib' graphcast
    
        # Pause for 1 minute
        sleep 10
    
        # Move the output file to the specified folder
        cp graphcast.grib $output_folder/graphcast-cds-offline-${current_date}${current_hour}-leadtime${leadtime}.grib
    
        # Increment the current time by the interval
        current_epoch=$(($current_epoch + $interval))
    done
done

My system

OS: Ubuntu 24.04
Card: RTX 3060 (12 GB)
Cuda: cuda_12.0.r12.0/compiler.32267302_0