Planner model testing

This repository contains a bunch of scripts for running experiments with planning domain models in Picat. However it can be configured for use with other planners too.

Prerequisities

Scripts described below has some dependencies:

1. Working planner(s) installed. All planners should be available on PATH.
2. SLURM workload manager. In particular the command sbatch which is used in cplan_all.sh script.

Workflow

Following steps should provide enough guidance to run your own set of experiments.

1. Get domain models and problem descriptions:
2. Prepare all your model files in a directory (refered as models). Each file at the models directory should represent different domain model in given domain.
3. Call ./prepareDomain.sh domainName models to prepare directory structure
4. Edit files mod_list, pla_list and prob_list to configure your experiment. These files has to be simple lists - one record on one line. Number of experiments run will be #models X #planners X #problems.

• mod_list - names of domain models to use (those are stored in models directory for each domain). Models are listed with .pddl extension.
• pla_list - names of planners to use. One planner on the line (e.g.: best_plan). Planners listed here refers to configuration files in the planners directory.
• prob_list - list of problems from problems directory to use in the experiment.

3. Call ./generate_array_file.sh domainName to generate file array.in. The file lists one experiment configuration on line.
4. Call ./launch_array.sh array.in. This will submit an array job to cluster. Results of the experiment will be stored in results/domainName and logs will be in logs directory.

Scripts

1. launch_array.sh - this script takes one parameter - the name of array file with descriptions of experiments. It submits an array job to the cluster. Batch size can be modified by changing BATCH_SIZE

2. cplan_array.sh - this scipt is called by the launch_array.sh script for each experiment configuration that is to be processed Parameters for SLURM (example):

   #SBATCH --chdir /path/to/planner_model_testing
   #SBATCH -p cpu
   #SBATCH --mem 1G
   #SBATCH -t 30:00

• --chdir <DIR> sets the working directory for given SLURM job
• -p <partition> sets the SLURM partition to use
• --mem <memory-limit> using memory over limit cause job termination
• -t MM:SS job will be terminated if it does not finish within given time limit

3. generate_array_file.sh - this script is used to generate all experiment configuration based on the mod_list, pla_list and prob_list files in each domain from the domains directory. By default it generate experiments for all domains in the domains directory. You can select domains for which you want to generate experiments by passing their list as arguments. e.g.:

./generate_array_file.sh depots nomystery

generates only task configurations for depots and nomystery domains 4. prepareDomain.sh - script to initialize directory structure for new domain