Panoptic Segmentation Forecasting

Colin Graber, Grace Tsai, Michael Firman, Gabriel Brostow, Alexander Schwing - CVPR 2021

Animated gif showing visual comparison of our model's results compared against the hybrid baseline

We propose to study the novel task of ‘panoptic segmentation forecasting’: given a set of observed frames, the goal is to forecast the panoptic segmentation for a set of unobserved frames. We also propose a first approach to forecasting future panoptic segmentations. In contrast to typical semantic forecasting, we model the motion of individual object instances and the background separately. This makes instance information persistent during forecasting, and allows us to understand the motion of each moving object.

Image presenting the model diagram

⚙️ Setup

Dependencies

Python 3.7
PyTorch 1.5.1
pyyaml
pandas
h5py
opencv
tensorboard
tqdm
pytorch_scatter 2.0.5
cityscapesscripts (for evaluation)
Google Cloud SDK (for downloading data/models)

Install the code using the following command:
pip install -e ./

Data

To run this code, the gtFine_trainvaltest dataset will need to be downloaded/decompressed from the Cityscapes website into the data/cityscapes/ directory. If you would like to visualize predictions, you will also need to download the leftImg8bit dataset.
Additionally, a few additional Cityscapes ground-truth files will need to be generated. This can be done by running the following commands:
- python -m cityscapesscripts.preparation.createPanopticImgs --dataset-folder data/cityscapes/gtFine/
- CITYSCAPES_DATASET=data/cityscapes/ python -m cityscapesscripts.preparation.createTrainIdLabelImgs
The remainder of the required data can be downloaded using the script download_data.sh. By default, everything is downloaded into the data/ directory.
Training the background model requires generating a version of the semantic segmentation annotations where foreground regions have been removed. This can be done by running the script scripts/preprocessing/remove_fg_from_gt.sh.
Training the foreground model requires additionally downloading a pretrained MaskRCNN model. This can be found at this link. This should be saved as pretrained_models/fg/mask_rcnn_pretrain.pkl.
Training the background model requires additionally downloading a pretrained HarDNet model. This can be found at this link. This should be saved as pretrained_models/bg/hardnet70_cityscapes_model.pkl.

Running our code

The scripts directory contains scripts which can be used to train and evaluate the foreground, background, and egomotion models. Note that these scripts should be run from the root project directory as shown below. Specifically:

scripts/odom/run_odom_train.sh trains the egomotion prediction model.
scripts/odom/export_odom.sh exports the odometry predictions, which can then be used during evaluation by other models
scripts/bg/run_bg_train.sh trains the background prediction model.
scripts/bg/run_export_bg_val.sh exports predictions make by the background using input reprojected point clouds which come from using predicted egomotion.
scripts/fg/run_fg_train.sh trains the foreground prediction model.
scripts/fg/run_fg_eval_panoptic.sh produces final panoptic semgnetation predictions based on the trained foreground model and exported background predictions. This also uses predicted egomotion as input. Note that the background export script must be run before this one so that the full panoptic segmentation outputs can be generated. Also, if you re-run this script, make sure to delete the predictions in the folder experiments/pretrained_fg/exported_panoptics_*_val/ first, as otherwise the generated json file will not contain entries for the sequences where foreground instances are not present.

We provide our pretrained foreground, background, and egomotion prediction models. The data downloading script additionally downloads these models into the directory pretrained_models/

✏️ 📄 Citation

If you found our work relevant to yours, please consider citing our paper:

@inproceedings{graber-2021-panopticforecasting,
 title   = {Panoptic Segmentation Forecasting},
 author  = {Colin Graber and
            Grace Tsai and
            Michael Firman and
            Gabriel Brostow and
            Alexander Schwing},
 booktitle = {Computer Vision and Pattern Recognition ({CVPR})},
 year = {2021}
}