M&Ms Challenge 2020

The CMR images have been segmented by experienced clinicians from the respective institutions, including contours
for the left (LV) and right ventricle (RV) blood pools, as well as for the left ventricular myocardium (MYO).
Labels are: 1 (LV), 2 (MYO) and 3 (RV)

Motivation

In the recent years, many machine/deep learning models have been proposed to accurately segment cardiac structures
in magnetic resonance imaging. However, when these models are tested on unseen datasets acquired from distinct
MRI scanners or clinical centres, the segmentation accuracy can be greatly reduced.

The M&Ms challenge aims to contribute to the effort of building generalisable models that can be applied consistently
across clinical centres. Furthermore, M&Ms will provide a reference dataset for the community to build and assess
future generalisable models in CMR segmentation.

Environment Setup

To use the code, the user needs to set te environment variable to access the data. At your ~/.bashrc add:
```shell script
export MMsCardiac_DATA_PATH=’/path/to/data/M&MsData/‘

Also, the user needs to to pre-install a few packages:
```shell script
$ pip install wheel setuptools
$ pip install -r requirements.txt
$ pip install torch==1.5.0+cu101 torchvision==0.6.0+cu101 -f https://download.pytorch.org/whl/torch_stable.html
$ pip install torchcontrib~=0.0.2

Data preparation

Train csv

You can generate train csv for dataloaders using python3 preprocess/generate_train_df.py.
```shell script
usage: generate_train_df.py [-h] [—meta_graphs]

M&Ms 2020 Challenge - Training info generation

optional arguments:
-h, —help show this help message and exit
—meta_graphs Generate train meta information graphs
```

Data Refactor

Load each volume to extract only 1 slice is time consuming. To solve this, save each slice in numpy arrays:
python3 preprocess/dataloader_refactor.py

Global Training Mean and STD

You can easily get global mean and std from labeled training samples using python3 preprocess/get_mean_std.py.

Data Description

The challenge cohort is composed of 350 patients with hypertrophic and dilated cardiomyopathies
as well as healthy subjects. All subjects were scanned in clinical centres in three different
countries (Spain, Germany and Canada) using four different magnetic resonance
scanner vendors (Siemens, General Electric, Philips and Canon).

Training set (150+25 studies)

The training set will contain 150 annotated images from two different MRI vendors (75 each) and 25 unannotated
images from a third vendor. The CMR images have been segmented by experienced clinicians from the respective
institutions, including contours for the left (LV) and right ventricle (RV) blood pools, as well as for the
left ventricular myocardium (MYO). Labels are: 1 (LV), 2 (MYO) and 3 (RV).

Testing set (200 studies)

The 200 test cases correspond to 50 new studies from each of the vendors provided in the training set and
50 additional studies from a fourth unseen vendor, that will be tested for model generalizability.
20% of these datasets will be used for validation and the rest will be reserved for testing and ranking participants.

Standard Operating Procedure (SOP) for data annotation

In order to build a useful dataset for the community we have decided to build on top of
ACDC MICCAI 2017 challenge SOP and correct our contours accordingly.

In particular, clinical contours have been corrected by two in-house annotators that had to agree on the final result.
These annotators followed these rules:

• LV and RV cavities must be completely covered, with papillary muscles included.
• No interpolation of the LV myocardium must be performed at the base.
• RV must have a larger surface in end-diastole compared to end-systole and avoid the pulmonary artery.

The main difficulty and source of disagreement is the exact RV form in basal slices.

Results

Using ACDC checkpoint:

Average -> 0.7397 -> 0.9933 (background), 0.6931 (LV), 0.5624 (MYO), 0.71(RV)

Calculated using resnet34_unet_imagenet_encoder, Adam and constant learning rate. Fold metrics are calculated
using mean of averaged iou and dice values. Only mnms data.

Principal conclusions: bce_dice_border_ce with 0.5,0.2,0.2,0.2,0.5 - lr 0.001/0.0001 - standardize.

Now, using lr 0.001, standardize and bce_dice_border_ce with 0.5,0.2,0.2,0.2,0.5, explore data augmentation.
Without data augmentation score 0.8360.

Competition Models

Bala 1

Using standardization, data augmentation combination old and bce_dice_border_ce with 0.5,0.2,0.2,0.2,0.5.
Resnet34 Unet with lr 0.001 and adam optimizer.

Bala 2

Using standardization, data augmentation combination old and bce_dice_border_ce with 0.5,0.2,0.2,0.2,0.5

Not Pretrained Model

Folding by patient.

Update: 11/06/2020 Meeting

Changes and ideas:

• Use 2 folds grouping by vendor (A vs. B), instead of n grouping by patient. Then error analysis by vendor
• Since is not permited the use of pre-trained models, try smaller architectures
• Create convolutional network that learns to distinguish if an image comes from vendor A or vendor B. ¿Works?
  • If works then we can create a DCGAN trying to apply a initial transformation to fool the discriminator and
    do something like normalize the input images! Note: Do not add vendor C in CNN classification step since
    we will use it for validate our GAN later.
• Self-Supervised Learning for unseen vendor C

Folding by Vendor Resuts

(Wrong folding, no train subpartition/patients to compare)

Normalization by reescale. Criterion bce_dice_border_ce -> 0.5,0.2,0.2,0.2,0.5.

General conclusions:

• Models can extract more information and thus make better predictions when training with Vendor ‘A’
  and then testing on ‘B’. GAN should approximate images to Vendor A?
• lr 0.001 works better than lower ones.
• Not clear difference using data augmentation and without apply it…
• Intermediate models size, resnet18_pspnet_unet, performs better than bigger ones and smaller ones.

11 random patients to compare

Criterion bce_dice_border_ce -> 0.5,0.2,0.2,0.2,0.5. Using resnet18_pspnet_unet.

General conclusions:

• When using Vendor A as training set, generalizes better to Vendor B cases.

Classification: Vendor ‘A’ - ‘B’ Discriminator

Using resnet18_pspnet_classification model. Adam with bce. 60 epochs and *0.1 steps as 25 and 50.
Img size 224x224. fold_system=”patient” & label_type=”vendor_label”. Normalization standardize. Learning rate 0.001.

Classification: Vendor ‘A’ - ‘B’ - ‘C’ Discriminator

Adam with bce. 80 epochs and *0.1 steps as 25 and 60.
Img size 224x224. fold_system=”patient” & label_type=”vendor_label”. Normalization standardize. Learning rate 0.001.
Data Augmentation combination (old).

Discriminator Entropy backwards ‘A’ - ‘B’ - ‘C’

Using gradient gamma 0.99, max iterations 250, standardize normalization. Segmentator Training with ‘A’.
Baseline: 0.7799 IOU on B.

• Problem with low out thresholds… Waste all iterations and stops.

Discriminator Entropy backwards ‘A’ - ‘B’ - ‘C’ / With blur, unblur and gamma

7 July

Hausdorff loss tests

Mean average values for 5 folds. Data combination old. Lr 0.001 with resnet_unet_scratch.

Other

• Development environment -> CUDA 10.1 and cudnn 7603. Python 3.8.2 - GCC 9.3.0
• Challenge homepage here.
• ACDC nomenclature: 0, 1, 2 and 3 represent voxels located in the background, in the right ventricular cavity,
  in the myocardium, and in the left ventricular cavity, respectively.