PHNX - A SLURM Batch Job Scheduler Utility

PHNX is a utility for managing and scheduling batch jobs on a SLURM-based HPC system. It allows users to manage a queue of tasks to be executed and schedule them according to the user-defined maximum number of concurrent jobs. The utility also includes a status script to monitor the progress of the tasks and provide an estimated time for completion.

Table of Contents

• Requirements
• Installation
• Configuration
• Usage
  • phnx.sh
  • phnx_status.sh
• Example
• License

Requirements

• A Linux-based system with a SLURM-based HPC environment
• Bash shell

Installation

Clone the repository or download the files directly:

git clone [email protected]:zeroknowledgediscovery/phnx.git

Change to the repository directory:

cd phnx

Make sure that the scripts are executable:

chmod +x phnx.sh phnx_status.sh

Configuration

Create a config.sh file in the same directory as the scripts with the following parameters:

#!/bin/bash

# Maximum number of jobs allowed to run concurrently
MAXJOBS=5

# Path to the QUEUE file containing a list of commands to be executed (one command per line)
QUEUE="queue.txt"

# SLURM sbatch settings
TIME="02:00:00" # Target time for jobs (HH:MM:SS)
PARTITION="your_partition" # Partition name
MEMORY="2G" # Memory per node
CORES=1 # Number of cores per task
CPUS=1 # Number of CPUs per task

Make sure to replace your_partition with the appropriate partition name for your HPC system.

Usage

phnx.sh

The phnx.sh script is responsible for managing and scheduling jobs from the queue.txt file. When executed, it checks if a job with the name PXMASTER is running. If so, the script exits. If not, it proceeds to schedule the jobs according to the MAXJOBS parameter defined in the configuration file.

To run the script:

./phnx.sh

phnx_status.sh

The phnx_status.sh script is used to monitor the progress of the tasks being executed. It reads the phnx_info.txt file generated by the phnx.sh script and calculates the number of tasks completed, the percentage of completion, the average time per task, and the estimated time to completion.

To run the script:

./phnx_status.sh

If you want to check the status of tasks in a specific output directory, pass the directory as an argument:

./phnx_status.sh phnxrun_20230422-123456

Example

1. Create a config.sh file with the appropriate configuration parameters.
2. Create a queue.txt file containing a list of commands to be executed, one per line:

echo "Task 1"
echo "Task 2"
sleep 5
echo "Task 3"

3. Run the phnx.sh script to start scheduling jobs:

./phnx.sh

4. Monitor the progress of the tasks using the phnx_status.sh script:

./phnx_status.sh <newly_created_phnx_directory>

License

This project is licensed under the GNU Affero General Public License v3.0