Hierarchical Neuron Format

The Hierarchical Neuron Format (HNF) is a schema for storing neuron morphologies
and meta data in Hdf5 files.

We provide read/write implementations for R and Python:

• for R see nat.hdf5
• for Python see navis

Preamble

There are a few file formats that can store neuron morphology. To name but a few:

• SWC
  for simple skeletons
• NeuroML is an XML-based format
  primarily used for modelling but can store compartment models (i.e. skeletons)
  of neurons and meta data
• NWB (neurodata without borders) is an HDF5-based format
  focused on physiological data
• NRRD files can be used to
  store dotprops

Why then start a new format?

Because none of the existing formats tick all the boxes! We need a file format
that can hold:

1. thousands of neurons
2. multiple representations (mesh, skeleton, dotprops) of a given neuron
3. annotations (e.g. synapses) associated with each neuron
4. meta data such as names, soma positions, etc.

Enter HDF5: basically a filesystem-in-a-file. The important thing is that we
don’t have to worry about how data is en-/decoded because other libraries
(like h5py for Python or hdf5r for R) take care of that. All we have to
do is come up with a schema.

Schema

HDF5 knows “groups” (=folders), “datasets” and “attributes”. The basic idea for
our schema is this:

• the root contains info about the format as attributes
• each group in root represents a neuron and the group’s name is the neuron’s ID
• a neuron group holds and meta data, and the neuron’s representations (mesh,
  skeleton and/or dotprops) and annotations in separate sub-groups

To illustrate the basic principle:

.
├── attrs: format-related meta data
├── group: neuron1
│   ├── attrs: neuron-related meta data
│   ├── group: skeleton
│   |    ├── attrs: skeleton-related meta data
|   |    └── datasets: node table, etc
│   ├── group: dotprops
│   |    ├── attrs: dotprops-related meta data
|   |    └── datasets: points, tangents, alpha, etc
│   ├── group: mesh
│   |    ├── attrs: mesh-related meta data
|   |    └── datasets: vertices, faces, etc
|   └── group: annotations
|       └── group: e.g. connectors
|           ├── attrs: connector-related meta data
|           └── datasets: connector data
├── group: neuron2
|   ├── ...  
...

Root attributes

The root meta data must contain two attributes:

• format_spec specifies format and version
• format_url points to a library or format specifications

.
├── attr['format_spec']: str = 'hnf_v1'   
├── attr['format_url']: str = 'https://github.com/schlegelp/navis'
...

Neuron base groups

Each neuron group contains properties that apply to all the neuron’s potential
representations - for example a neuron_name. Note that if an attribute is
defined at the neuron level and again at a deeper level (i.e. the skeleton,
mesh or dotprops), the more proximal attribute takes precedence for a given
representation.

.
└── group['123456']  # note that numeric IDs will be "stringified"
    ├── attr["neuron_name"]: str = "some name"
...

Skeletons

Attributes:

• units_nm (float | int | tuple, optional): specifies the units in
  nanometer space - can be a tuple of (x, y, z) if units are
  non-isotropic
• soma (int, optional): the node ID of the soma

Datasets:

• node_id (int): IDs for the nodes
• parent_id (int): for each node, the ID of it’s parent; nodes with
  out parents (i.e. roots) have parent_id of -1
• x, y, z (float | int): node coordinates
• radius (float | int, optional): radius for each node

└── group['123456']
    ├── attr['neuron_name'] = "example neuron with a skeleton"
    ├── attr['units_nm'] = (4, 4, 40)
    └── grp['skeleton']
         ├── attr['soma']: 1
         ├── ds['node_id']: (N, ) array
         ├── ds['parent_id']: (N, ) array
         ├── ds['x']: (N, ) array
         ├── ds['y']: (N, ) array
         ├── ds['z']: (N, ) array
         └── ds['radius']: (N, ) array, optional

Meshes

Meshes are principally represented as vertices + triangular faces (navis
is using trimesh under the hood).

Attributes:

• units_nm (float | int | tuple, optional): specifies the units in
  nanometer space - can be a tuple of (x, y, z) if units are
  non-isotropic
• soma (tuple, optional): tuple of (x, y, z) coordinates of the soma

Datasets:

• vertices (int | float): (N, 3) array of vertex positions
• faces (int): (M, 3) array of vertex indices forming the faces (indices start
  at 0)
• skeleton_map (int, optional): (N, ) array mapping each vertex to a
  node ID in the skeleton

└── group['4353421']
    ├── attr['neuron_name'] = "example neuron with a mesh"
    ├── attr['units_nm'] = (4, 4, 40)
    └── grp['mesh']
         ├── attr['soma']: (1242, 6533, 400)
         ├── ds['vertices']: (N, 3) array
         ├── ds['faces']: (M, 3) array
         └── ds['skeleton_map']: (N, ) array, optional

Dotprops

Attributes:

• k (int): number of k-nearest neighbours used to calculate the tangent
  vectors from the point cloud
• units_nm (float | int | tuple, optional): specifies the units in
  nanometer space - can be a tuple of (x, y, z) if units are
  non-isotropic
• soma (tuple, optional): tuple of (x, y, z) coordinates of the soma

Datasets:

• points (int | float): (N, 3) array of x/y/z positions
• vect (int | float, optional): (N, 3) array of tangent vectors -
  generated if not provided
• alpha (int | float, optional): (N, ) array of alpha values for each
  point in points generated if not provided

└── group['65432']    
    ├── attr['neuron_name'] = "example neuron with dotprops"    
    └── grp['dotprops']
        ├── attr['k'] = 5
        ├── attr['units_nm'] = (4, 4, 40)
        ├── attr['soma']: (1242, 6533, 400)
        ├── ds['points']: (N, 3) array
        ├── ds['vect']: (N, 3) array
        └── ds['alpha']: (N, ) array

Annotations

Annotations are meant to be flexible and are principally parsed into
pandas DataFrames. Because they won’t follow a common format, it is
good practice to leave some (optional) meta data pointing to columns
containing data relevant for e.g. plotting:

Attributes:

• point_col (str | list thereof): pointer to the column(s) containing
  x/y/z positions
• type_col (str): pointer to a column specifying types
• skeleton_map (str): pointer to a column associating the row with
  a node ID in the skeleton

Let’s illustrate this with a mock synapse table:

└── group['32434566']
    ├── attr['neuron_name'] = "example neuron with synapse annotations"
    ├── attr['units_nm'] = 1
    └── grp['annotations']
         └── grp['synapses']
             ├── attr['points']: ['x', 'y', 'z']
             ├── attr['types']: 'prepost'
             ├── attr['skeleton_map']: 'node_id'
             ├── ds['x']: (N, ) array
             ├── ds['x']: (N, ) array
             ├── ds['z']: (N, ) array
             ├── ds['prepost']: (N, ) array of [0, 1, 2, 3, 4]
             └── ds['node_id']: (N, )

“Hidden” attributes & datasets

It can be useful to have attributes and datasets that contain information that’s
only pertinent for the reader/writer but does not directly relate to the neuron.

For this, we prefix the attribute/dataset with a .:

└── group['4353421']
    ├── attr['neuron_name'] = "example neuron with a mesh"
    ├── attr['units_nm'] = (4, 4, 40)
    ├── attr['.hidden_attribute'] = "typically ignored when reading"
    └── grp['mesh']
         ├── attr['soma']: (1242, 6533, 400)

We use hidden attributes to e.g. store a serialized version of a neuron instead/
in addition to the raw data to speed up reading the data.

A final remark

The above schema describes a “minimal” layout - i.e. we expect no less
data than that. However, e.g. the navis implementations for reading/writing
the schema are flexible: you can add more attributes or datasets
and navis will by default try to read and attach them to the neuron.

Is this stable?

Ish? The format is versioned and I will maintain readers/writers for
past versions in navis. In other good news: the HDF5 backend is
stable - so even if navis acts up when parsing your file, you can
always read it manually using h5py.

Changelog

The current version of the format is 1.0.

Changes:

• 2021/02/01: Version 1.0