Modulation - Explicit Dependency Management for Ruby

Modulation | mɒdjʊˈleɪʃ(ə)n | Music - a change from one key to another in a
piece of music.

INSTALL |
GUIDE |
API |
EXAMPLES |
RDOC

Modulation provides an alternative way of organizing your Ruby code. Modulation
lets you explicitly import and export declarations in order to better control
dependencies in your codebase. Modulation helps you refrain from littering
the global namespace with a myriad modules, or complex multi-level nested
module hierarchies.

Using Modulation, you will always be able to tell where a class or module comes
from, and you’ll have full control over which parts of a module’s code you wish
to expose to the outside world. Modulation can also help you write Ruby code in
a functional style, minimizing boilerplate code.

Note: Modulation is not a replacement for RubyGems. Rather, Modulation is
intended for managing dependencies between source files inside your Ruby
applications. Though it does support loading gems that were written using
Modulation, it is not intended as a comprehensive solution for using
third-party libraries.

Features

• Complete isolation of each module
  prevents literring of the global namespace.
• Explicit exporting and
  importing of methods and constants lets you
  control the public interface for each module, as well as keep track of all
  dependencies in your code.
• Tagged paths simplify
  the management of dependencies in large applications.
• Lazy Loading improves start up time and memory
  consumption.
• Hot module reloading streamlines your development
  process.
• Module mocking facilitates testing.
• Dependency introspection lets you introspect
  your dependencies at runtime.
• Application packing lets you
  bundle your code in a single, optionally obfuscated file (WIP).

Rationale

You’re probably asking yourself “what the **?” , but when your Ruby app grows
and is split into multiple files loaded using #require, you’ll soon hit some
issues:

• Once a file is #required, any class, module or constant in it is available
  to any other file in your codebase. All “globals” (classes, modules,
  constants) are loaded, well, globally, in a single namespace. Name conflicts
  are easy in Ruby.
• To avoid class name conflicts, classes need to be nested under a single
  hierarchical tree, sometime reaching 4 levels or more. Just look at Rails.
• Since a #required class or module can be loaded in any file and then made
  available to all files, it’s easy to lose track of where it was loaded, and
  where it is used.
• There’s no easy way to hide implementation-specific classes or methods. Yes,
  there’s #private, #private_constant etc, but by default everything is
  #public!
• Extracting functionality is harder when modules are namespaced and
  dependencies are implicit.
• Writing reusable functional code requires wrapping it in modules using
  class << self, def self.foo ..., extend self or include Singleton
  (the pain of implementing singletons in Ruby has been
  discussed
  before.)

There’s a recent discussion on the
Ruby bug tracker regarding possible solutions to the problem of top-level
name collision. Hopefully, the present gem could contribute to an eventual
“official” API.

Personally, I have found that managing dependencies with #require in large
codebases is… not as elegant or painfree as I would expect from a
first-class development environment. I also wanted to have a better solution
for writing in a functional style.

So I came up with Modulation, a small gem that takes a different approach to
organizing Ruby code: any so-called global declarations are hidden unless
explicitly exported, and the global namespace remains clutter-free. All
dependencies between source files are explicit, visible, and easy to understand.

Installing Modulation

You can install the Modulation using gem install, or add it to your Gemfile:

gem 'modulation'

Organizing your code with Modulation

Modulation builds on the idea of a Ruby Module as a
“collection of methods and constants”.
Using modulation, each Ruby source file becomes a module. Modules usually
export method and constant declarations (usually an API for a specific,
well-defined functionality) to be shared with other modules. Modules can also
import declarations from other modules. Anything not exported remains hidden
inside the module and normally cannot be accessed from the outside.

Each source file is evaluated in the context of a newly-created Module
instance, with some additional methods for introspection and miscellaneous
operations such as hot reloading.

Modulation provides alternative APIs for loading modules. Instead of using
require and require_relative, we use import, import_map etc, discussed
in detail in the API reference.

Basic Usage

Exporting declarations

Any class, module or constant be exported using #export:

export :User, :Session
class User
...
end
class Session
...
end

A module may also expose a set of methods without using class << self, for
example when writing in a functional style:

seq.rb

export :fib, :luc
def fib(n)
  (0..1).include?(n) ? n : (fib(n - 1) + fib(n - 2))
end
def luc(n)
  (0..1).include?(n) ? (2 - n) : (luc(n - 1) + luc(n - 2))
end

app.rb

require 'modulation'
Seq = import('./seq')
puts Seq.fib(10)

Another way to export methods and constants is by passing a hash to #export:

module.rb

export(
  foo: :bar,
  baz: -> { 'hello' },
  MY_CONST: 42
)
def bar
  :baz
end

app.rb

m = import('./module')
m.foo #=> :baz
m.baz #=> 'hello'
m::MY_CONST #=> 42

Any capitalized key will be interpreted as a const, otherwise it will be defined
as a method. If the value is a symbol, Modulation will look for the
corresponding method or const definition and will treat the key as an alias.

The export method can be called multiple times. Its behavior is additive:

# this:
export :foo, :bar
# is the same as this:
export :foo
export :bar

Importing declarations

Declarations from another module can be imported using #import:

require 'modulation'
Models = import('./models')
...
user = Models::User.new(...)
...

Alternatively, a module interested in a single declaration from another module
can use the following technique:

require 'modulation'
User = import('./models')::User
...
user = User.new(...)

A word about paths

Paths given to import are always considered relative to the importing file,
unless they are absolute (e.g. /home/dave/repo/my_app), specify a
tag or reference a
gem. This is true for all Modulation APIs
that accept path arguments.

Advanced Usage

Using tags to designate common subdirectories

Normally, module paths are always relative to the file calling the #import
method, just like #require_relative. This can become a problem once you start
moving your source files around. In addition, in applications where your source
files are arranged in multiple directories, it can quickly become tedious to do
stuff like Post = import('../models/post').

Modulation provides an alternative to relative paths in the form of tagged
sources. A tagged source is simply a path associated with a label. For example,
an application may tag lib/models simply as @models. Once tags are defined,
they can be used when importing files, e.g. import('@models/post').

To define tags, use Modulation.add_tags:

Modulation.add_tags(
  models: '../lib/models',
  views:  '../lib/views'
)
...
User = import '@models/user'

Importing all source files in a directory

To load all source files in a directory you can use #import_all:

import_all('./ext')

Groups of modules providing a uniform interface can also be loaded using
#import_map:

API = import_map('./math_api') #=> hash mapping filenames to modules
API.keys #=> ['add', 'mul', 'sub', 'div']
API['add'].(2, 2) #=> 4

The #import_map takes an optional block to transform hash keys:

API = import_map('./math_api') { |name, mod| name.to_sym }
API.keys #=> [:add, :mul, :sub, :div]
API[:add].(2, 2) #=> 4

Importing methods into classes and objects

Modulation provides the #extend_from and #include_from methods to include
imported methods in classes and objects:

module Sequences
  extend_from('./seq.rb')
end
Sequences.fib(5)
# extend integers
require 'modulation'
class Integer
  include_from('./seq.rb')
  def seq(kind)
    send(kind, self)
  end
end
5.seq(:fib)

The #include_from method accepts an optional list of symbols to import:

class Integer
  include_from './seq.rb', :fib
end
5.fib

Default exports

A module may wish to expose just a single class or constant, in which case it
can use #export_default:

user.rb

export_default :User
class User
  ...
end

app.rb

require 'modulation'
User = import('./user')
User.new(...)

The default exported value can also be defined directly thus:

config.rb

export_default(
  host: 'localhost',
  port: 1234,
  ...
)

app.rb

require 'modulation'
config = import('./config')
db.connect(config[:host], config[:port])

Circular dependencies

Circular dependencies, while not the best practice for organizing a code base,
are sometimes useful. Modulation supports circular dependencies, with the
exception of modules with default exports.

Accessing a module’s root namespace from nested modules within itself

The special constant MODULE allows you to access the containing module from
nested modules or classes. This lets you call methods defined in the module’s
root namespace, or otherwise introspect the module:

export :AsyncServer
# Await a promise-like callable
def await
  calling_fiber = Fiber.current
  p = ->(v = nil) {calling_fiber.resume v}
  yield p
  Fiber.yield
end
class AsyncServer < SomeTCPServer
  def async_read
    MODULE.await {|p| on_read {|data| p.(data)}}
  end
end

Accessing the global namespace

If you need to access the global namespace inside a module just prefix the
class name with double colons:

class ::GlobalClass
  ...
end
::ENV = { ... }
what_is = ::THE_MEANING_OF_LIFE

Programmatic module creation

In addition to loading modules from files, modules can be created dynamically at
runtime using Modulation.create. You can create modules by supplying a hash
prototype, a string or a block:

# Using a hash prototype
m = Modulation.create(
  add: -> x, y { x + y },
  mul: -> x, y { x * y }
)
m.add(2, 3)
m.mul(2, 3)
# Using a string
m = Modulation.create <<~RUBY
export :foo
def foo
  :bar
end
RUBY
m.foo
# Using a block
m = Modulation.create do { |mod|
  export :foo
  def foo
    :bar
  end
  class mod::BAZ
    ...
  end
}
m.foo

The creation of a objects using a hash prototype is also available as a separate
gem called eg.

Unit testing modules

Methods and constants that are not exported can be tested using the #__expose!
method. Thus you can keep implementation details hidden, while being able to
easily test them:

parser.rb

export :parse
def parse(inp)
  split(inp).map(&:to_sym)
end
# private method
def split(inp)
  inp.split(',').map(&:strip)
end

test_seq.rb

require 'modulation'
require 'minitest/autorun'
Parser = import('../lib/parser').__expose!
class FibTest < Minitest::Test
  def test_that_split_trims_split_parts
    assert_equal(%w[abc def ghi], Parser.split(' abc ,def , ghi  '))
  end
end

Mocking modules

Modules loaded by Modulation can be easily mocked when running tests or specs,
using Modulation.mock:

require 'minitest/autorun'
require 'modulation'
module MockStorage
  extend self
  def get_user(user_id)
    {
      user_id: user_id,
      name: 'John Doe',
      email: 'johndoe@gmail.com'
    }
  end
end
class UserControllerTest < Minitest::Test
  def test_user_storage
    Modulation.mock('../lib/storage', MockStorage) do
      controller = UserController.new
      ...
    end
  end
end

Modulation.mock accepts a module path and a receiver, and the module stays
mocked within the given block.

Lazy Loading

Modulation allows the use of lazy-loaded modules - loading of modules only once
they’re needed by the application, in similar fashion to Module#auto_load. To
lazy load modules use the #auto_import method, which takes a constant name and
a path:

export :foo
auto_import :BAR, './bar'
def foo
  # the bar module will only be loaded once this method is called
  MODULE::BAR
end

Lazy-loaded constants must always be qualified. When referring to a
lazy-loaded constant from the module’s top namespace, use the MODULE
namespace, as shown above.

The #auto_import method can also take a hash mapping constant names to paths.
This is especially useful when multiple concerns are grouped under a single
namespace:

export_default :SuperNet
module SuperNet
  auto_import(
    HTTP1:      './http1',
    HTTP2:      './http2',
    WebSockets: './websockets'
  )
end
SuperNet::HTTP1 #=> loads the http1 module

Reloading modules

Modules can be reloaded at run-time for easy hot code reloading:

require 'modulation'
SQL = import('./sql')
...
SQL.__reload!

Another way to reload modules is using Modulation.reload, which accepts a
module or a filename:

require 'filewatcher'
FileWatcher.new(['lib']).watch do |fn, event|
  if(event == :changed)
    Modulation.reload(fn)
  end
end

When a module is reloaded, its entire content - constants and methods - will
be replaced. That means that any code using that module could continue to use
it without even being aware it was reloaded, providing its API has not
changed.

Reloading of modules with default exports is also possible. Modulation will
extend the exported value with a #__reload! method. The value will need to be
reassigned:

require 'modulation'
settings = import('settings')
...
settings = settings.__reload!

Please note that Modulation does not include a directory watcher that
automatically reloads changed modules. This is due to multiple considerations
that include the chosen threading model, or the reactor engine in use, or even
the chosen solution for watching files (whether it’s an external gem or an
internal tool).

It is, however, quite trivial to watch files using
directory_watcher:

require 'directory_watcher'
dw = DirectoryWatcher.new 'lib', glob: '**/*.rb', interval: 2, pre_load: true
dw.add_observer do |*events|
  events.each do |e|
    next unless e.type == :modified
    Modulation.reload e.path
  end
end
dw.start

Retaining state between reloads

Before a module is reloaded, all of its methods and constants are removed. In
some cases, a module might need to retain state across reloads. You can do this
by simply using instance variables:

export :value, :inc
@counter ||= 0
def value
  @counter
end
def incr
  @counter += 1
end

Care must be taken not to reassign values outside of methods, as this will
overwrite any value retained in the instance variable. To assign initial values,
use the ||= operator as in the example above. See also the
reload example.

Dependency introspection

Modulation allows runtime introspection of dependencies between modules. You can
interrogate a module’s dependencies (i.e. the modules it imports) by calling
#__depedencies:

m1.rb

import ('./m2')

app.rb

m1 = import('./m1')
m1.__depedencies #=> [<Module m2>]

You can also iterate over a module’s entire dependency tree by using
#__traverse_dependencies:

m1 = import('./m1')
m1.__traverse_dependencies { |mod| ... }

To introspect reverse dependencies (modules using a particular module), use
#__dependent_modules:

m1 = import('./m1')
m1.__depedencies #=> [<Module m2>]
m1.__dependencies.first.__dependent_modules #=> [<Module m1>]

Running Modulation-based applications

Modulation provides a binary script for running Modulation-based applications.
mdl is a wrapper around Ruby that loads your application’s main file as a
module, and then runs your application’s entry point method. Let’s look at a
sample application:

app.rb

def greet(name)
  puts "Hello, #{name}!"
end
def main
  print "Enter your name: "
  name = gets
  greet(name)
end

To run this application, execute mdl app.rb, or mdl run app.rb. mdl will
automatically require the modulation gem and call the application’s entry
point, #main.

Packing applications with Modulation

Note: application packing is at the present time an experimental feature.
There might be security concerns for packaging your app, such as leaking
filenames from the developer’s machine.

Modulation can also be used to package your entire application into a single
portable file that can be copied to another machine and run as is. To package
your app, use mdl pack. This command will perform a dynamic analysis of all
the app’s dependencies and will put them together into a single Ruby file.

For more information have a look at the app example.

Writing gems using Modulation

Modulation can be used to write gems, providing fine-grained control over your
gem’s public APIs and letting you hide any implementation details. In order to
allow loading a gem using either #require or #import, code your gem’s main
file normally, but add require 'modulation/gem' at the top, and export your
gem’s main namespace as a default export, e.g.:

require 'modulation/gem'
export_default :MyGem
module MyGem
  ...
  MyClass = import('my_gem/my_class')
  ...
end

Importing gems using Modulation

Gems written using modulation can also be loaded using #import. If modulation
does not find the module specified by the given relative path, it will attempt
to load a gem by the same name. It is also possible to load specific files
inside modules by specifying a sub-path:

require 'modulation'
MyFeature = import 'my_gem/my_feature'

Note: Since there’s not much of a point in #importing gems that do not
use Modulation to export symbols, Modulation will refuse to import any gem
that does not depend on Modulation.

Writing modules that patch external classes or modules

It is generally recommended you refrain from causing side effects or patching
external code in your modules. When you do have to patch external classes or
modules (i.e. core, stdlib, or some third-party code) in your module, it’s
useful to remember that any module may be eventually reloaded by the application
code. This means that any patching done during the loading of your module must
be idempotent, i.e. have the same
effect when performed multiple times. Take for example the following module
code:

module ::Kernel
  # aliasing #sleep more than once will break your code
  alias_method :orig_sleep, :sleep
  def sleep(duration)
    STDERR.puts "Going to sleep..."
    orig_sleep(duration)
    STDERR.puts "Woke up!"
  end
end

Running the above code more than once would cause an infinite loop when calling
Kernel#sleep. In order to prevent this situation, modulation provides the
Module#alias_method_once method, which prevents aliasing the original method
more than once:

module ::Kernel
  # alias_method_once is idempotent
  alias_method_once :orig_sleep, :sleep
  def sleep(duration)
    STDERR.puts "Going to sleep..."
    orig_sleep(duration)
    STDERR.puts "Woke up!"
  end
end

Coding style recommendations

• Import modules into constants, not variables:

  Settings = import('./settings')

• Place your exports at the top of your module, followed by #requires,
  followed by #imports:

  export :foo, :bar, :baz
  require 'json'
  Core = import('./core')
  ...

API Reference

Kernel

Kernel#auto_import_map(path, options = {})

Returns a hash mapping keys to corresponding module files inside the given
directory path. Modules are loaded automatically upon accessing hash keys.

Kernel#import(path)

Returns a loaded module identified by the given path. The path can contain
tags

Kernel#import_all(path)

Kernel#import_map(path, options = {})

Module

Module#__module_info

Returns a hash containing information about the module. This currently includes
the following entries:

location|Absolute module file path
exported_symbols|Array containing all symbols exported by the module

Module#__reload!

Module#alias_method_once(new_name, old_name)

Module#auto_import(sym, path)

Module#export(*symbols)

Module#export_default(value)

Module#export_from_receiver(receiver)

Module#extend_from(path)

Module#include_from(path, *symbols)

Module::MODULE

Modulation

Modulation.full_backtrace!

Modulation.reload

Why you should not use Modulation

• Modulation is not production-ready.
• Modulation is not thread-safe.
• Modulation doesn’t play well with rdoc/yard.
• Modulation (probably) doesn’t play well with Marshal.
• Modulation (probably) doesn’t play well with code-analysis tools.