DNN For Complexity Estimation of Scene Images

Setup

1. Install Python 3 (3.8 recommended)
2. Install provided requirements either via pip (pip install -r requirements.txt) or Anaconda environment with required packages (see below)
3. Acquire the model weights and place them somewhere easily accessible (for example, model/complexity_net.pt)
4. Set up your testing data. This should be a directory which contains JPG or PNG images.

The model weights are available here.

Anaconda / Required Packages

This project depends upon core packages:

• Torch
• Torchvision
• Numpy
• Pillow (PIL)
• tqdm

For computing complexity metrics, you will also require:

• Opencv2
• Scikit-image

See requirements.txt for a full list and version information.

Run

1. Either run run.py directly, or edit and execute the provided shell script (sh example_run.sh)
2. On first run, weights for the ResNet backbone will be downloaded.

Quickstart

The general run command is: python3 run.py --model path/to/model/weights.pt --directory input/image/directory --output output/directory/

Additional Details

Example images are provided under samples/ and example outputs under output/. By running the model against the samples/ directory with the weights provided above, predictions matching those provided under the outputs directory should be generated by the neural network.

To do this, (with the model weights under model/) run the following command:

python3 run.py --model model/complexity_net.pt --directory samples/ --output output_dir/

You may also simply import the eval_directory function from ComplexityNet.evaluate and use as required. See run.py for an example of this.

The model will execute on the GPU if possible, otherwise will fall back to CPU.

Required flags for run.py execution:

--model [path/to/model.pt]: The path to the model weights, provided as a .pt file.

--directory [path/to/input/images]: Path to directory for which you wish to predict complexity data. Must contain only .png or .jpg / .jpeg files.

--output [path/to/output/]: Directory to place output (generated) complexity maps and predictions. If it does not exist, it will be created.

Optional flags for run.py

--batch_size [integer]: Set batch size (default 2). As the network is provided pre-trained, it is only strictly necessary to tweak this to minimise IO overhead.

Clutter & Symmetry Metrics

This repository also includes tools for computing the clutter and symmetry scores given in the paper. To do this, run the following command (or see example_metrics.sh):

python3 metrics.py --processes 2 --directory samples/ --output output

This will generate a metrics.csv under the output directory containing clutter and symmetry scores for each image in the input directory. It's recommended to normalise these output values between 0 and 1.0.

A configuration file is provided under the ClutterSym package, configuration.json that contains parameters used for the Normalised Graph Cut and the Symmetry calculator. You may need to tweak these to best fit your data, if you are seeing obviously unexpected results.

Required flags for metrics.py

--directory [path/to/input/images]: Path to directory for which you wish to predict complexity metrics. Must contain only .png or .jpg / .jpeg files. The input images must be square, RGB (three channels), and 8 bit.

--output [path/to/output/]: Directory to place output CSV containing metric data. If it does not exist, it will be created.

Optional flags for metrics.py

--processes [integer, between 1 and n, default: 2]: The number of processes to use. If running the script over a large amount of images, setting this number to (available cores - 1) will reduce runtime considerably.

Reference

If you found this code useful, please cite the paper:

Characterising and dissecting human perception of scene complexity
Cameron Kyle-Davidson, Elizabeth Yue Zhou, Dirk B. Walther, Adrian G. Bors, Karla K. Evans
Journal of Cognition
2023

https://www.sciencedirect.com/science/article/pii/S0010027722003080

@article{kyle2023characterising,
  title={Characterising and dissecting human perception of scene complexity},
  author={Kyle-Davidson, Cameron and Zhou, Elizabeth Yue and Walther, Dirk B and Bors, Adrian G and Evans, Karla K},
  journal={Cognition},
  volume={231},
  pages={105319},
  year={2023},
  publisher={Elsevier}
}

Disclaimer

The material embodied in this software is provided to you "as-is" and without warranty of any kind, express, implied or otherwise, including without limitation, any warranty of fitness for a particular purpose. This material may change at any time, without warning.