Self-Build Tools

A framework for writing small tools in Go which are invoked as regular
commands, using compiled builds.

Think “go run” but with executable tools, and only compiling when something
has changed.

Every tool has a useful --version output, with git identifying information.

This is a demonstration project, suitable for forking and using for your own
tools.

How to use

You should be able to invoke a tool in the bin directory as-is. This section
is really about how to take ownership and start extending with your own tools.

Place the bin/ directory of the checkout of this repo in your $PATH
ahead of ~/go/bin/. Eg:

PATH="$HOME/bin:$HOME/src/self-build-tools/bin:$HOME/go/bin:/usr/bin:/bin:/usr/local/bin"
export PATH

That should be enough for things to work. Read on for extending.

Extending

Pick a namespace of your own to replace example.org/private; things will
work with the defaults, but identification of origin in binaries will make
life better when you’re later trying to identify things.

Edit src/go.mod and src/go_compile_cache to set the module and namespace
entries accordingly, at the top of the files. Adjust the import paths in the
go source files.

Whenever you create a new .go file, be sure to import the golib library and
call golib.FastFlags() at the start of main(), or break that out into
individual calls if wanted. The variables for version information live inside
golib, and that also provides the -version flag handling. Drop a symlink to
the wrapper script into the bin/ dir with a name which matches the go source
file without the .go extension.

So if you create src/foo.go then create a symlink bin/foo pointing to
../src/go_compile_cache.

The // +build exclude_except_for_go_mod build constraint is very much like
// +build ignore except that go mod won’t ignore the files, so you get
dependency tracking in go.mod which works, even while you have several
different files each in package main with their own main() functions in
the one directory.

Rebuilds

The tools are compiled and placed into ~/go/bin/ by default. Whenever the
source file, or the go.sum dependencies file, are newer than the compiled
tool (or the compiled tool doesn’t exist), the wrapper script will
automatically recompile the tool. Various pieces of metadata are taken from
the git repo, including both the current commit of the repo and the latest
commit which modified this tool in question, and both sets are embedded.

Just invoking the wrapper should be enough. Using the --version parameter
is probably a good way to see the results.

Note that you can also use go version -m ~/go/bin/compiled-tool-name to
extract more data about a given tool, including all the module dependencies.
This is where changing the namespace away from example.org/private will pay
off, as the mod lines will give you something more useful for your
environment.

Portability

This is known to work on Linux (glibc and musl) and FreeBSD. The wrapper
script is POSIX sh, and the only tools which are not “very standard POSIX” we
depend upon are git and go.

The binaries are placed in ~/go/bin/ on the assumption that this is safe for
any given architecture. If you have a home-directory shared over NFS for use
on multiple architectures, adjust the exedir definition in
go_compile_cache.

History

This is taken from a personal tools & settings repo, which contains far more
than just these tools. The pattern works well enough to be useful.

TODO

It might be worth having a pre-commit hook which checks that for every .go
file, there exists a symlink in the bin directory.

Demonstration Commands

• dns_ask_auth: issue a DNS query to each of the authoritative nameservers
  for the domains of the parameters, asking for the value from each and
  showing them. This is useful for consistency checks.
  • The default qtype is TXT, and SOA is a useful choice for seeing if
    the SOA serials are consistent.
  • DNSKEY records will have derived DS records and various flags added as
    comments, to aid in glue debugging.
• dns_cache_warm: a way to ask multiple DNS recursors a large number of
  queries in parallel, useful for pre-warming DNS caches.
  • This expects a configuration file, default at
    ~/etc/dns-cache-warm.conf, and an example is in the etc/ directory.
  • This is not the greatest code, but it works for my purposes. The
    configuration file was originally written to drive a shell script which
    invoked host or dig, so there are some unhandled combinations in the
    revamp.
  • Invoke with --progress to reduce the number of lines emitted; during
    resolution, a count of resolutions queries done so far will be
    repeatedly updated on one line, before being replaced with the summary
    when done.
• go_grab_topath: given Go import paths on the command-line, each one is
  downloaded as a VCS repository into the old $GOPATH hierarchy; this
  roughly matches the pre-modules go get behavior, facilitating working on
  the source of some code instead of fetching a static snapshot to use as a
  dependency.
  • This uses the $GOPATH/src layout because it’s a prior existing
    convention, not because it’s right. This is demonstration code, adjust
    to suit local conventions/desires.

I use a wrapper dns_cache_warm_home which puts some .home.arpa hostnames
into the DNS_CACHE_EXTRA_RESOLVE environment variable and invokes
dns_cache_warm with --progress --time-resolvers and the IP addresses of
the DNS resolvers on my home LAN.
The extra hostnames are not in the committed file because that is shared
beyond my home network.