Hestia

Pseudo-Decentralized Storage Middleware, or in other words, a Multi-Backend Gaia Hub

Feature Overview

• Gaia Compatability
  • To use as your Gaia Hub, simply use {your hestia node url}/gaia!
  • Uses Gaia Authentication token format for simplicity and ease-of-use
• Easily configurable with many whitelisting options
• Uses a Database for storing metadata, indexing the files, and storing user preferences
  • This improves performance when looking up your files instead of just assuming they all
    exist.
• Advance Drivers
  • Multi-Instance: Run more than one driver of a particular type with different
    configuration options
  • Multi-User: Run one driver that supports individual users (for remote cloud storage
    like Dropbox)
  • “Root Only”: Only writing the root folder (profile.json and avatar), to limit storage
    use and encourage users to use their own remote backends
• Plugins
  • From making backups-on-request to providing a dashboard, it’s all possible through the
    Plugin API Interface

Doc Shortcuts:

• About / Goals
• Installation / Setup
• Configuration
• Api
• DB Drivers
• Drivers
• Plugins
• Building & Testing
• License

About / Goals

Background

I am simply trying to finish what Blockstack started in regards to Gaia – or rather,
to try and fulfill the original goal in a different way. I wanted Users to be able to
use their own Dropbox without having to spin up their own node and all of the complexity
that doing so brings – so why not have a pseudo-centralized service handle it all for
them? And beyond that, why not have the ability to hook up multiple backends (as
advertised in the whitepaper) that can replicate or be given to a particular app at
the user’s choosing?

If you want to get into blockstack easily, use this; if you are concerned about
centralization but still want the ease-of-use this brings, run your own node; if you
want to go as deep as you can, run multiple of your own gaia hubs and use a browser
that supports that (if any exist currently).

• Michael Fedora, from here.

Explanation

The goal of Hestia is to serve as a more complex
Gaia Hub. While the original software works well,
it works simply and cannot solve certain problems, such as those posed by using personal
cloud storage providers as backends while also allowing any end-user to do so.

Hestia was made so that any Blockstack user could use their personal cloud storage
(i.e. Dropbox, Google Drive, etc) as their own storage backend, i.e. to have full control
over both ends (writing the files and accessing the backend). While this node, which
serves as middleware, is still controled by a third-party, it can easily be run by
any user as well, whether for themselves, for their family, or their organization.

End User -> Gaia -> Amazon S3 (Node Owned Backend)
End User -> Hestia -> End User's Dropbox (User Owned Backend)

In addition, because of its inherent complexity, more features have been added to allow
extension by third party plugins and other drivers, whether locally created or imported
through npm. There are also more configuration options on the default drivers, such as
the Disk driver being able to limit how much overall storage it is using, as well as
how much each user is allowed to store. Hestia can also whitelist drivers as well as
limit a driver to only being able to be used as an identity folder — i.e. only a
user’s profile.json and avatar will be stored on the driver backend, and all other apps
that attempt to use the driver will fail.

Hestia also provides a singular Gaia interface for all backends — this means the user
manages what drivers handle what buckets. For instance, an end-user could have their Dropbox
handle Stealthy data, while their Google Drive can handle their Travelstack data, and their
One Drive gets everything. All drivers get your identity folder, however, and some drivers
allow you to use them more than once; for instance, you could attatch two dropbox instances to
your account if you so desired — but you can only have one disk driver per driver instance.

User Connections:
- Disk (identity only, 5mb limit)
- Dropbox 1 (personal, 2gb limit, default (store everything))
- Dropbox 2 (team, 2gb limit, stealthy.im only)

Of course, with all of this complexity, there will be some downsides:

• It is required to use an association token for every request, as the Hestia Hub needs to know
  the end-user’s address to be able to read the connection information
• Connection information (such as dropbox tokens) are stored unencrypted within the Hestia Hub
  • These can obviously still be revoked, and this should not seem more unsecure than any other
    app requesting to use your dropbox.
• File metadata and their paths are stored in the local database, unencrypted
  • While it could be concerning because of how easy it is to get it, reading a Gaia Hub’s
    logs (or the server’s HTTP logs) would get you the same information.

Back to top

Installation / Setup

• npm i
• npm run build-prod
• Copy config.sample.json, rename to config.json and configure (see:
  Configuration below)
• npm start

Back to Top

Configuration

A sample config file is provided in config.sample.json, but
here is an annotated configuration below:

interface Config {
  port: number; // The port to serve the application on
  ip: string; // The ip (generally `0.0.0.0`) to serve the application on
  protocol: string; // The protocol (`http` or `https`) to serve the application on
  server_name: string; // The server name (e.x. `localhost:{port}` or `Hestia.otorno.cc`)
  valid_hub_urls?: string[]; // (optional) Other valid hub urls for apps to make requests to
  db_driver_path?: string; // (optional) the path where the db driver is located
                          //     (default: `default-db-drivers/sqlite3`)
  db_driver_config?: any; // (optional) the config for the db driver
  pm2?: boolean; // (optional) whether or not you are using pm2 (for logging issues)
  pm2InstanceVar?: string; // (optional) the instance ID if it is not default
  whitelist?: string[]; // (optional) A list of addresses which are whitelisted to use the node
  max_blob_size?: string | number; // (optional) The maximum blob size for files (i.e. "5mb", 5242880)
                                   // default is 7.5mb
  page_size?: number; // The pagination size for list-files
  root_plugin?: string; // The plugin to use as the `/` plugin, i.e. for a web interface
  // driver configs
  drivers: {
    [id: string]: { // this driver ID
      path: string; // the path where the driver is located
      name?: string; // (optional) a different name to use then the standard one
      icon_url?: string; // (optional) a url to use for the icon instead of the default provided one
      whitelist?: string[]; // (optional) a whitelist of users who can use the driver
      auto_register?: boolean; // (optional) whether or not users should be auto-registered for the
                              // driver when their account is registered
      root_only?: boolean; // (optional) whether or not the driver should only be allowed for
                          // the root (identity) directory
    } 
  };
  // plugin configs
  plugins: {
    [id: string]: { // the plugin ID
      path: string; // the path to the plugin for `import({path})`
    }
  };
}

Back to top

API

Token Types

Most routes need some sort of authentication token in order to work properly, but in
general there are four types of routes:

• Those that do not need any (“None“)
• Those that need normal gaia bucket authentication (“Bucket“)
  • See the Gaia Hub Repo for specs
• Those that only partially check the token (“Partial“)
  • Based off of the Bucket token, but does not check for the user existing, or that
    the claimed Gaia-Hub in the token is the actual Hestia Hub
• Those that need what is called a “user” authentication (“User“)
  • A Bucket token that is either authed for the Hestia Hub itself as if it were an app
    (i.e. properly issued to server_name) or one that is issued for the Id’s root bucket
    (i.e. if the issuer and signer are the same address).

Any token besides ones for which the signer and issuer are equal need to be signed with
an “Association Token,” which is a sub-token inside of the overarching token. Please see
the Gaia Hub Repo for more information on the
structure of both the normal token and the association token.

Meta Routes (/)

Gaia Routes (/gaia)

The gaia route group, of which all (should) align with the spec listed in the
Gaia Hub repo. This route is what is used in
browser configurations which need a link to your gaia hub (i.e. server_name/gaia).

Return type for /gaia/list-files/{address} with state as false or undefined:

{
  entries: string[]; // full-length paths of files
  page?: number; // the next page, if it exists
}

Return type for /gaia/list-files/{address} with state as true:

{
  entries: {
    name: string; // full-length path of the file
    contentLength: number; // the size of the file
    lastModifiedDate: number; // the unix timestamp of when the file was last modified
  }[];
  page?: number; // the next page, if it exists
}

API Routes (/api/v1)

User Routes (/api/v1/user)

Return type of /api/v1/user/list-files

{
  [path: string]: {
    contentType: string;
    size: number;
    hash: string;
    lastModified: string;
    connIds: string[]
  }
}

Return type of /api/v1/user/list-files?global=true

{
  [path: string]: {
    [connId: string] {
      contentType: string;
      size: number;
      hash: string;
      lastModified: string;
    }
  }
}

Connection Routes (/api/v1/connections)

All routes within the /api/v1/connections group require an authed user token, and some of
them are very similar to the routes found under /gaia. One trait they do all share in
common, however, is that the very first parameter is the {id} field, which is the connection
ID itself — you can get this from the current sub-object in the /drivers return object
(when requested with a valid token).

Back to top

DB Drivers

Database drivers can be selected by setting the db_driver_path entry in the config.json
to a path that can be used to import it; default-db-drivers is the prefix for using default
drivers, otherwise you can use a relative import (from the root directory where hestia.js is
run from — i.e when using npm start it will be the root repository directory), or an import
from node_modules as you would normally do from a script (i.e. npm i my-hestia-db-driver
and then, in the config.json, "path": would be "my-hestia-db-driver" and that’s it).

Default database drivers are the Sqlite3 driver (default-drivers/sqlite3), which is the one
Hestia will use if none is specified, and the RethinkDB driver default-drivers/rethinkdb.

DB Driver Configuration

View each database driver’s docs to see how it needs to be configured specifically.

For the default database drivers, see below:

// path: `default-db-drivers/sqlite3`
interface SQLite3Config {
  filename?: string; // (optional) the filename of the database to use
                     // default is `hestia-db.sqlite`
}
// path: `default-db-drivers/sqlite3`
interface RethinkDBConfig {
  host?: string; // (optional) the RethinkDB host (default: `127.0.0.1`)
  port?: number; // (optional) the RethinkDB port (default: `28015`)
}

Back to top

Drivers

Drivers can be included by adding them in the config.json. The path field is used to
import them; default-drivers is the prefix for using default drivers, otherwise you can use
a relative import (from the root directory where hestia.js is run from — i.e when using
npm start it will be the root repository directory), or an import from node_modules as you
would normally do from a script (i.e. npm i my-hestia-driver and then, in the config.json,
"path": would be "my-hestia-driver" and that’s it).

Default drivers are the Disk driver (default-drivers/disk), which allows you to use the local
harddisk and the User-Dropbox driver (default-drivers/user-dropbox), which allows users to use
their own Dropbox account (and also fulfills one of the goals of this project).

Driver Configuration

Every driver can take the following options:

export interface DriverConfig {
  path: string; // the path where the driver is located
  name?: string; // (optional) a different name to use then the standard one
  icon_url?: string; // (optional) a url to use for the icon instead of the default provided one
  whitelist?: string[]; // (optional) a whitelist of users who can use the driver
  auto_register?: boolean; // (optional) whether or not users should be auto-registered for the
                           // driver when their account is registered
  root_only?: boolean; // (optional) whether or not the driver should only be allowed for
                       // the root (identity) directory
}

Tip: root_only is a fantastic option so that users can automatically register for your
Hestia Hub without going through the hoops to register their own storage backend first. For
instance, if you are limited on Disk space, but want to make it easy for users to get started
using your hub, simply use the disk driver (below), limit it to 5mb or less for each user,
and make it root only — Users will automatically be able to register to the Hestia Hub, but
will have to register another driver in order to actually use it for apps.

View each drivers’s docs to see how it needs to be configured.

For the default drivers, see below:

// path: `default-drivers/disk`
interface DiskDriverConfigType {
  storage_root_directory: string; // the directory to put the files (default: `./Hestia-storage`)
  // for storage caps (below), use a number of bytes or a string representation (i.e. "5mb")
  max_user_storage: string | number; // the storage cap for each  user (default: unlimited)
  max_total_storage: string | number; // the overall storage cap for Hestia (default: unlimited)
}
// path: `default-drivers/user-dropbox`
interface UserDropboxDriverConfig {
  client_id: string; // the client Id for the http dropbox API
  secret: string; // the client secret for the http dropbox API
}
// **NOTE**: When using this driver, make sure your Hestia Hub's max_blob_size is LESS THAN OR
// EQUAL TO the Gaia Buckets you are connected to -- otherwise syncing will fail because the
// remote gaia server will fail repeatedly. It WILL NOT separate files into parts for you.
//
// path: `default-drivers/gaia`
interface GaiaDriverConfig {
  token?: string; // (optional - required for hub-backend) the authorization token
                  // if not provided, this will be a per-user backend
}

Back to top

Plugins

Plugins can be included by adding them in the config.json. The path field is used to
import them; default-plugins is the prefix for using default plugins, otherwise you can use
a relative import (from the root directory where hestia.js is run from — i.e when using
npm start it will be the root repository directory), or a import from node_modules as you
would normally do from a script (i.e. npm i my-hestia-plugin and then, in the config.json,
"path": would be "my-hestia-plugin" and that’s it).

Default plugins are the Dashboard plugin (default-plugins/dashboard), which adds the Hestia
frontend to the server (and must be the root_plugin in order to work properly), and the
Backup plugin (default-plugins/backup), which adds backup functionality to the Hestia Hub.
Currently there are a couple of other plugins in the directory but they are still under
construction.

There are no routes added by the Dashboard plugin (besides the root route, which is the
dashboard webpage itself), but the Backup plugin adds three, all of which require an authenticated user
(replace /plugins/backup with /plugins/{your chosen id} for your own instance):

Plugin Configuration

Every plugin config requires a path field to specify where it is located at:

interface PluginConfig {
  path: string; // the path where the plugin is located
}

View each plugin’s docs to see how it needs to be configured specifically.

For the default plugins, see below:

// path: `default-plugins/dashboard`
interface DashboardPluginConfig {
  root_directory?: string; // defaults to `__dirname/../../frontend"`
                           // (i.e. `./build/frontend` or `./build-prod/frontend`)
}
// path: `default-plugins/backup`
interface BackupPluginConfig {
  temp_directory?: string; // default: `__dirname/backups`
                           // (i.e. `./build/backend/default-plugins/backups`)
}

Back to top

Building & Testing

• npm run build - build debug, output to ./build
  • npm run build:backend- just build (debug) the backend
  • npm run build:frontend- just build (debug) the frontend

• npm test - start the debug build (./build)

• npm run build-prod - build prod, output to ./build-prod

• npm start - start the prod build (./build-prod)

Back to top

License

Released under Mozilla Public License 2.0, with graphics under
CC BY-SA 4.0.

What the code license means is that you are free to fork and modify the project,
but any changes to this code should be given back via pull requests. Snippets
are generally ok to take, and your own plugins, drivers, and db drivers do not
need to be added to this repo — those are yours to license as you will. Of course,
I am not a lawyer, so go ask one if you need further help.

The graphics license means that if you take my graphics, you can derive or republish
them at will, as long as you give me attribution and share under the same license.

Back to top