File Categorization Utility

Create a categorized image of an unorganized directory (e.g. a disk dump).
Each contained file is classified according to its suffix or signatures (also
known as magic numbers or magic bytes) and moved/copied/linked to a
corresponding directory in the output directory.

Quick Example

If you want to organize a directory which looks like this (or possibly a
thousand times worse):

big_mess
├── definitely_not_porn
│   ├── cool.mkv
│   ├── omg.mkv
│   └── secret
│       ├── nice.jpg
│       └── wow.mkv
├── documents
│   ├── dir1
│   │   ├── not_a_doc.doc
│   │   └── untitled.ods
│   ├── dir2
│   │   ├── document.odt
│   │   └── untitled.ods
│   └── recording.mp3
├── random_stuff
│   ├── document.docx
│   ├── howto.pdf
│   └── paper.docx
└── song.mp3

you can run categorize_files.py -c mime_content -p copy -i flat_name input to
get it done:

big_mess_categorized_by_mime_content
├── application
│   ├── pdf
│   │   └── howto.pdf
│   ├── vnd.oasis.opendocument.spreadsheet
│   │   ├── untitled.ods
│   │   └── untitled.ods.1
│   ├── vnd.oasis.opendocument.text
│   │   ├── document.odt
│   │   └── not_a_doc.doc
│   └── vnd.openxmlformats-officedocument.wordprocessingml.document
│       ├── document.docx
│       └── paper.docx
├── audio
│   └── mpeg
│       ├── recording.mp3
│       └── song.mp3
├── image
│   └── jpeg
│       └── nice.jpg
└── video
    └── x-matroska
        ├── cool.mkv
        ├── omg.mkv
        └── wow.mkv

Parameters

There are three essential parameters: classification criterion, file system
operation used to create the categorized files, and output image structure.
Several other parameters are available, for more information see the built-in
help (-h/--help argument).
To demonstrate how these parameters affect the output, the following directory
tree will be used as input:

input
├── app1
│   ├── python_bytecode.pyc
│   ├── python_script.py
│   └── shell_script
└── app2
    ├── python_bytecode.pyc
    ├── python_script.py
    └── shell_script

Classification Criterion

A criterion by which files are classified.
Set by the -c/--criterion argument with one of the following options:

• suffix — the file suffix (extension)

  input_categorized_by_suffix
  ├── py
  │   ├── python_script.py
  │   └── python_script.py.1
  ├── pyc
  │   ├── python_bytecode.pyc
  │   └── python_bytecode.pyc.1
  └── unknown
    ├── shell_script
    └── shell_script.1

• mime_name — guess the MIME type of the file based on its filename

  input_categorized_by_mime_name
  ├── application
  │   └── x-python-code
  │       ├── python_bytecode.pyc
  │       └── python_bytecode.pyc.1
  ├── text
  │   └── x-python
  │       ├── python_script.py
  │       └── python_script.py.1
  └── unknown
    ├── shell_script
    └── shell_script.1

• mime_content — guess the MIME type of the file based on its content

  input_categorized_by_mime_content
  ├── application
  │   └── octet-stream
  │       ├── python_bytecode.pyc
  │       └── python_bytecode.pyc.1
  └── text
    ├── x-python
    │   ├── python_script.py
    │   └── python_script.py.1
    └── x-shellscript
        ├── shell_script
        └── shell_script.1

File System Operation

A file system operation used to create the categorized file.
Set by the -p/--operation argument with one of the following options:

• move — move (rename) the file
• copy — copy the file
• hard_link — create a hard link pointing to the file
• symbolic_link — create a symbolic link (symlink) pointing to the file

  input_categorized_by_suffix
  ├── py
  │   ├── python_script.py -> /path/to/the/input/app2/python_script.py
  │   └── python_script.py.1 -> /path/to/the/input/app1/python_script.py
  ├── pyc
  │   ├── python_bytecode.pyc -> /path/to/the/input/app2/python_bytecode.pyc
  │   └── python_bytecode.pyc.1 -> /path/to/the/input/app1/python_bytecode.pyc
  └── unknown
    ├── shell_script -> /path/to/the/input/app2/shell_script
    └── shell_script.1 -> /path/to/the/input/app1/shell_script

Output Image Structure

Determines file names and a directory structure of the output image directory.
Set by the -i/--image-structure argument with one of the following options:

• flat_name — input directories are not preserved.
  Collisions are possible.

  input_categorized_by_suffix
  ├── py
  │   ├── python_script.py
  │   └── python_script.py.1
  ├── pyc
  │   ├── python_bytecode.pyc
  │   └── python_bytecode.pyc.1
  └── unknown
    ├── shell_script
    └── shell_script.1

• flat_path — input directories encoded in the file name (path separator
  characters are replaced with underscores).
  Collisions are possible.

  input_categorized_by_suffix
  ├── py
  │   ├── app1_python_script.py
  │   └── app2_python_script.py
  ├── pyc
  │   ├── app1_python_bytecode.pyc
  │   └── app2_python_bytecode.pyc
  └── unknown
    ├── app1_shell_script
    └── app2_shell_script

• nested — input directories are preserved.
  Collisions are not possible.

  input_categorized_by_suffix
  ├── py
  │   ├── app1
  │   │   └── python_script.py
  │   └── app2
  │       └── python_script.py
  ├── pyc
  │   ├── app1
  │   │   └── python_bytecode.pyc
  │   └── app2
  │       └── python_bytecode.pyc
  └── unknown
    ├── app1
    │   └── shell_script
    └── app2
        └── shell_script

Known Bugs and TODOs

• Using the flat_path output image structure may create file names longer than
  the file system can handle.
  • Currently [Errno 36] File name too long: 'filename' is reported and the
    affected file is skipped.
  • In POSIX, the limis is defined by NAME_MAX which is usually set to 255
    chars.
  • To eliminate this, some kind of name ellipsization would be necessary.
• Some information about file format identification can be found
  here or
  here