Crowdsourcing App - Calista

A Web Application to collect data from pairwise image comparisons via crowdsourcing. Built with MERN stack.

Calista team developed and used this app to build a comparison-based dataset on website aesthetics.

Example from the evaluation process:

Paper: “Calista: A deep learning-based system for understanding and evaluating website aesthetics“

Cite as:

@article{DELITZAS2023,
    title = {Calista: A deep learning-based system for understanding and evaluating website aesthetics},
    journal = {International Journal of Human-Computer Studies},
    volume = {175},
    pages = {103019},
    year = {2023},
    issn = {1071-5819},
    doi = {https://doi.org/10.1016/j.ijhcs.2023.103019},
    url = {https://www.sciencedirect.com/science/article/pii/S1071581923000253},
    author = {Alexandros Delitzas and Kyriakos C. Chatzidimitriou and Andreas L. Symeonidis}
}

Prerequisites

Docker
Docker-compose

Deployment

Step 1 - Images

Define the images that you want to collect data for
Add these images in the folder front-end/public/images/

Step 2 - Environment variables

Add a .env file in the root folder of the project and set the following variables:

Environment variable	Description
MONGO_URI	MongoDB connection string
PORT	Server port
MONGO_PORT	MongoDB port
ME_CONFIG_MONGODB_ADMINUSERNAME	MongoDB username
ME_CONFIG_MONGODB_ADMINPASSWORD	MongoDB password
ME_CONFIG_BASICAUTH_USERNAME	Mongo-express username
ME_CONFIG_BASICAUTH_PASSWORD	Mongo-express password
NODE_ENV	Node environment variable
VOTING_ROUNDS	Number of voting rounds of each vote session
VOTING_TIME	Voting time in seconds of each comparison
NUMBER_OF_IMAGES	Number of images in front-end/public/images/
BASEURL	Base URL that is used for the requests

Step 3 - Setup

This step is required only for the first time that you will run the application. This may take a few minutes as it will download the images required. If you have already completed this step once successfully, then you can jump directy to Step 4.

TL;DR:

Run the following command:

  ./setup.sh --create --sortimages

More detailed options:

The images in front-end/public/images/ must be named with numbers starting from 0 (eg. 0.png, 1.png, 2.png, …). If the images aren’t already in this format, then you can use the option —sortimages to get them automatically renamed during the setup:
```
./setup.sh --sortimages
```
If you want to set a new database, then it is required to have the file db/utils/comparisons_data.json. This file contains all possible pairwise comparisons that can be made between the images with random order. It is used for the database initialization. If you want to use a pre-existing comparisons_data.json file, then make sure that it is in db/utils/ directory. Otherwise, you can use the option —create to get this file automatically created during the setup:
```
./setup.sh --create
```

If you want to restore an existing database, first make sure to move your backup files in db/backup/ directory. Then you can use the option —restore=.

For example, if you had the following structure,

├── db
│   ├── backup
│   │   ├── 14_04_2020
│   │   │   └── votes
│   │   │       ├── comparisons.bson
│   │   │       ├── comparisons.metadata.json
│   │   │       ├── votesessions.bson
│   │   │       └── votesessions.metadata.json
.   .   .
.   .   .
.   .   .

you would run:

  ./setup.sh --restore=14_04_2020/votes

Note: In this case, a comparisons_data.json file isn’t needed.

You can also combine multiple options or use none.

Step 4 - Run

Start:

docker-compose -f docker-compose.yml up --build

Stop:

Ctrl-C

For detached mode:

Start:

docker-compose -f docker-compose.yml up -d --build

Stop:

docker-compose down