Detect video character relations using face recognition

Goal: create visual characters / actors relationship graph by detecting which characters appear in the same video frames.

Popular The Office (U.S.) season 1 series were chosen for processing.

Result:

Workflow

1. Use ageitgey/face_recognition (uses dlib) to detect faces and extract facial features from video into JSON.
   1. Decodes video frames using OpenCV
   2. Uses 1 frame per second from the video input
   3. Streams faces features as JSON output
2. Match extracted faces with known faces using euclidean distance.
   1. Face features are 128-th dimensional vectors of floats.
   2. I had to add manually inspect which actor’s faces were not detected and add face sample and character name mappings for each pair.
3. Record which faces appeared together in the same frames
4. Draw a graph where two frequently together appearing characters are close in the graph too.
   1. Used networkx and matplotlib libraries