Modernish is a library for writing robust, portable, readable, and powerful programs for POSIX-based shells and utilities.
For code examples, seeEXAMPLES.md
andshare/doc/modernish/examples
[, [[, (( ?find were a built-in shell loop?Modernish is a library for shell script programming which provides features
like safer variable and command expansion, new language constructs for loop
iteration, and much more. Modernish programs are shell programs; the new
constructs are mixed with shell syntax so that the programmer can take
advantage of the best of both.
There is no compiled code to install, as modernish is written entirely in the
shell language. It can be deployed in embedded or multi-user systems in which
new binary executables may not be introduced for security reasons, and is
portable among numerous shell implementations. The installer can also
bundle
a reduced copy of the library with your scripts, so they can run portably with
a known version of modernish without requiring prior installation.
Join us and help breathe some new life into the shell! We
are looking for testers, early adopters, and developers to join us.
Download the latest release
or check out the very latest development code from the master branch.
Read through the documentation below. Play with the example scripts and
write your own. Try to break the library and send reports of breakage.
use safe
use var/loop
use var/local
use var/arith
use var/assignuse var/readfuse var/shellquote
use var/stack
use var/string
use var/unexportuse var/genoptparseruse sys/base
use sys/cmd
use sys/dir
use sys/term
Run install.sh and follow instructions, choosing your preferred shell
and install location. After successful installation you can run modernish
shell scripts and write your own. Run uninstall.sh to remove modernish.
Both the install and uninstall scripts are interactive by default, but
support fully automated (non-interactive) operation as well. Command
line options are as follows:
install.sh [ -n ] [ -s shell ] [ -f ] [ -P pathspec ]
[ -d installroot ] [ -D prefix ] [ -B scriptfile … ]
-n: non-interactive operation-s: specify default shell to execute modernish-f: force unconditional installation on specified shell-P: specify an alternative DEFPATH
for the installation (be careful; usually *not* recommended)
-d: specify root directory for installation-D: extra destination directory prefix (for packagers)-B: bundle modernish with your scripts (-D required, -n implied), see
[Appendix F](#user-content-appendix-f-bundling-modernish-with-your-script)
uninstall.sh [ -n ] [ -f ] [ -d installroot ]
-n: non-interactive operation-f: delete */modernish directories even if files left-d: specify root directory of modernish installation to uninstallIn the simple form, modernish is added to a script written for a specific
shell. In the portable form, your script is shell-agnostic and may run on any
shell that can run modernish.
The simplest way to write a modernish program is to source modernish as a
dot script. For example, if you write for bash:
#! /bin/bash. modernishuse safeuse sys/base...your program starts here...
The modernish use command load modules with optional functionality. Thesafe module initialises the safe mode.
The sys/base module contains modernish versions of certain basic but
non-standardised utilities (e.g. readlink, mktemp, which), guaranteeing
that modernish programs all have a known version at their disposal. There are
many other modules as well. See Modules for more
information.
The above method makes the program dependent on one particular shell (in this
case, bash). So it is okay to mix and match functionality specific to that
particular shell with modernish functionality.
(On zsh, there is a way to integrate modernish with native zsh scripts. See
Appendix E.)
The most portable way to write a modernish program is to use the special
generic hashbang path for modernish programs. For example:
#! /usr/bin/env modernish#! use safe#! use sys/base...your program begins here...
For portability, it is important there is no space after env modernish;
NetBSD and OpenBSD consider trailing spaces part of the name, so env will
fail to find modernish.
A program in this form is executed by whatever shell the user who installed
modernish on the local system chose as the default shell. Since you as the
programmer can’t know what shell this is (other than the fact that it passed
some rigorous POSIX compliance testing executed by modernish), a program in
this form must be strictly POSIX compliant – except, of course, that it
should also make full use of the rich functionality offered by modernish.
Note that modules are loaded in a different way: the use commands are part of
hashbang comment (starting with #! like the initial hashbang path). Only such
lines that immediately follow the initial hashbang path are evaluated; even
an empty line in between causes the rest to be ignored.
This special way of pre-loading modules is needed to make any aliases they
define work reliably on all shells.
Modernish is primarily designed to enhance shell programs/scripts, but also
offers features for use in interactive shells. For instance, the new repeat
loop construct from the var/loop module can be quite practical to repeat
an action x times, and the safe module on interactive shells provides
convenience functions for manipulating, saving and restoring the state of
field splitting and globbing.
To use modernish on your favourite interactive shell, you have to add it to
your .profile, .bashrc or similar init file.
Important: Upon initialising, modernish adapts itself to
other settings, such as the locale. It also removes certain aliases that
may keep modernish from initialising properly. So you have to organise your.profile or similar file in the following order:
PATH, locale, etc.);. modernish and use any modules you want;After installation, the modernish command can be invoked as if it were a
shell, with the standard command line options from other shells (such as-c to specify a command or script directly on the command line), plus some
enhancements. The effect is that the shell chosen at installation time will
be run enhanced with modernish functionality. It is not possible to use
modernish as an interactive shell in this way.
Usage:
modernish [ --use=module | shelloption … ]modernish [ --use=module | shelloption … ]-c [ script [ me-name [ arguments ] ] ]modernish --test [ testoption … ]modernish [ --version | --help ]In the first form, the script in the file scriptfile is
loaded and executed with any arguments assigned to the positional parameters.
In the second form, -c executes the specified modernish
script, optionally with the me-name assigned to $ME and the
arguments assigned to the positional parameters.
The --use option pre-loads any given modernish modules
before executing the script.
The module argument to each specified --use option is split using
standard shell field splitting. The first field is the module name and any
further fields become arguments to that module’s initialisation routine.
Any given short-form or long-form shelloptions are
set or unset before executing the script. Both POSIX
shell options
and shell-specific options are supported, depending on
the shell executing modernish.
Using the shell option -e or -o errexit is an error, because modernish
does not support it and
would break.
The --test option runs the regression test suite and exits. This verifies
that the modernish installation is functioning correctly. See
Appendix B
for more information.
The --version and --help options output the relative information and exit.
modernish --use=var/loop -c 'LOOP for i=1 to 10; DO putln "$i"; DONE'zsh /usr/local/bin/modernish -o xtrace /path/to/program.shModernish includes a battery of shell feature, quirk and bug detection
tests, each of which is given a special capability ID.
See Appendix A for a
list of shell capabilities that modernish currently detects, as well
as further general information on the capability detection framework.
thisshellhas is the central function of the capability detection
framework. It not only tests for the presence of shell features/quirks/bugs,
but can also detect specific shell built-in commands, shell reserved words,
shell options (short or long form), and signals.
Modernish itself extensively uses capability detection to adapt itself to the
shell it’s running on. This is how it works around shell bugs and takes
advantage of efficient features not all shells have. But any script using
the library can do this in the same way, with the help of this function.
Test results are cached in memory, so repeated checks using thisshellhas
are efficient and there is no need to avoid calling it to optimise
performance.
Usage:
thisshellhas item …
_,-- (end-of-options delimiter), disable the recognition of- for subsequent items.--rw= or --kw=, check if the identifier--bi=, similarly check for a shell built-in command.--sig=, check if the shell knows about a signalkill, trap, etc.) by the name or number following the =.REPLY variable, otherwise REPLY is unset. (If multiple --sig= itemsREPLY contains only the last one.)-o followed by a separate word, check if this shell has a-, check if--cache runs all external modernish shell capability tests--show performs a --cache and then outputs all the IDs ofthisshellhas continues to process items until one of them produces a
negative result or is found invalid, at which point any further items are
ignored. So the function only returns successfully if all the items
specified were found on the current shell. (To check if either one item or
another is present, use separate thisshellhas invocations separated by the|| shell operator.)
Exit status: 0 if this shell has all the items in question; 1 if not; 2 if
an item was encountered that is not recognised as a valid identifier.
Note: The tests for the presence of reserved words, built-in commands,
shell options, and signals are different from capability detection tests in an
important way: they only check if an item by that name exists on this shell,
and don’t verify that it does the same thing as on another shell.
All modernish functions require portable variable and shell function names,
that is, ones consisting of ASCII uppercase and lowercase letters, digits,
and the underscore character _, and that don’t begin with digit. For shell
option names, the constraints are the same except a dash - is also
accepted. An invalid identifier is generally treated as a fatal error.
Function-local variables are not supported by the standard POSIX shell; only
global variables are provided for. Modernish needs a way to store its
internal state without interfering with the program using it. So most of the
modernish functionality uses an internal namespace _Msh_* for variables,
functions and aliases. All these names may change at any time without
notice. Any names starting with _Msh_ should be considered sacrosanct and
untouchable; modernish programs should never directly use them in any way.
Of course this is not enforceable, but names starting with _Msh_ should be
uncommon enough that no unintentional conflict is likely to occur.
Modernish provides certain constants (read-only variables) to make life easier.
These include:
$MSH_VERSION: The version of modernish.$MSH_PREFIX: Installation prefix for this modernish installation (e.g.$MSH_MDL: Main modules directory.$MSH_AUX: Main helper scripts directory.$MSH_CONFIG: Path to modernish user configuration directory.$ME: Path to the current program. Replacement for $0. This is#!/usr/bin/env modernish is used, or ifsh /path/to/bin/modernish
/path/to/script.sh, as these set $0 to the path to bin/modernish and$MSH_SHELL: Path to the default shell for this modernish installation,$SIGPIPESTATUS: The exit status of a command killed by SIGPIPE (agrep something somefile.txt |
more and you quit more before grep is finished, grep is killed bySIGPIPE and exits with that particular status.SIGPIPE exitSIGPIPE was set to ignore by the process that invoked the current$SIGPIPESTATUS can’t be detected and is set to the special valueWRN_NOSIGPIPEthisshellhas.$DEFPATH: The default system path guaranteed to find compliant POSIXgetconf PATH.$ERROR: A guaranteed unset variable that can be used to trigger an: "${4+${ERROR:?excess arguments}}" (error on 4 or more arguments)POSIX does not provide for the quoted C-style escape codes commonly used in
bash, ksh and zsh (such as $'\n' to represent a newline character),
leaving the standard shell without a convenient way to refer to control
characters. Modernish provides control character constants (read-only
variables) with hexadecimal suffixes $CC01 .. $CC1F and $CC7F, as well as $CCe,$CCa, $CCb, $CCf, $CCn, $CCr, $CCt, $CCv (corresponding withprintf backslash escape codes). This makes it easy to insert control
characters in double-quoted strings.
More convenience constants, handy for use in bracket glob patterns for use
with case or modernish match:
$CONTROLCHARS: All ASCII control characters.$WHITESPACE: All ASCII whitespace characters.$ASCIIUPPER: The ASCII uppercase letters A to Z.$ASCIILOWER: The ASCII lowercase letters a to z.$ASCIIALNUM: The ASCII alphanumeric characters 0-9, A-Z and a-z.$SHELLSAFECHARS: Safe-list for shell-quoting.$ASCIICHARS: The complete set of ASCII characters (minus NUL).Usage examples:
# Use a glob pattern to check against control characters in a string:if str match "$var" "*[$CONTROLCHARS]*"; thenputln "\$var contains at least one control character"fi# Use '!' (not '^') to check for characters *not* part of a particular set:if str match "$var" "*[!$ASCIICHARS]*"; thenputln "\$var contains at least one non-ASCII character" ;;fi# Safely split fields at any whitespace, comma or slash (requires safe mode):use safeLOOP for --split=$WHITESPACE,/ field in $my_items; DOputln "Item: $field"DONE
The die function reliably halts program execution, even from within
subshells, optionally
printing an error message. Note that die is meant for an emergency program
halt only, i.e. in situations were continuing would mean the program is in an
inconsistent or undefined state. Shell scripts running in an inconsistent or
undefined state may wreak all sorts of havoc. They are also notoriously
difficult to terminate correctly, especially if the fatal error occurs within
a subshell: exit won’t work then. That’s why die is optimised for
killing all the program’s processes (including subshells and external
commands launched by it) as quickly as possible. It should never be used for
exiting the program normally.
On interactive shells, die behaves differently. It does not kill or exit your
shell; instead, it issues SIGINT to the shell to abort the execution of your
running command(s), which is equivalent to pressing Ctrl+C.
In addition, if die is invoked from a subshell such as a background job, it
kills all processes belonging to that job, but leaves other running jobs alone.
Usage: die [ message ]
If the trap stack module
is active, a specialDIE pseudosignal
can be trapped (using plain old trap orpushtrap)
to perform emergency cleanup commands upon invoking die.
If the MSH_HAVE_MERCY variable is set in a script and die is invoked
from a subshell, then die will only terminate the current subshell and its
subprocesses and will not execute DIE traps, allowing the script to resume
execution in the parent process. This is for use in special cases, such as
regression tests, and is strongly discouraged for general use. Modernish
unsets the variable on init so it cannot be inherited from the environment.
The POSIX shell lacks a simple, straightforward and portable way to output
arbitrary strings of text, so modernish adds two commands for this.
put prints each argument separated by a space, without a trailing newline.putln prints each argument, terminating each with a newline character.There is no processing of options or escape codes. (Modernish constants$CCn, etc.
can be used to insert control characters in double-quoted strings. To process escape codes, useprintf
instead.)
The echo command is notoriously unportable and kind of broken, so is
deprecated in favour of put and putln. Modernish does provide its own
version of echo, but it is only activated for
portable-form)
scripts. Otherwise, the shell-specific version of echo is left intact.
The modernish version of echo does not interpret any escape codes
and supports only one option, -n, which, like BSD echo, suppresses the
final newline. However, unlike BSD echo, if -n is the only argument, it is
not interpreted as an option and the string -n is printed instead. This makes
it safe to output arbitrary data using this version of echo as long as it is
given as a single argument (using quoting if needed).
not, so, foreverModernish sets three aliases that can help to make the shell language look
slightly friendlier. Their use is optional.
not is a new synonym for !. They can be used interchangeably.
so is a command that tests if the previous command exited with a status
of zero, so you can test the preceding command’s success with if so orif not so.
forever is a new synonym for while :;. This allows simple infinite loops
of the form: forever do stuff; done.
exitThe exit command can be used as normal, but has gained capabilities.
Extended usage: exit [ -u ] [ status [ message ] ]
exit.$ME minus directories). Note that it is-u option is given, and the shell function showusage is defined,-u option has noshowusage function.showusagechdirchdir is a robust cd replacement for use in scripts.
The standard cd command
is designed for interactive shells and appropriate to use there.
However, for scripts, its features create serious pitfalls:
$CDPATH variable is searched. A script may inherit a user’s$CDPATH, so cd may change to an unintended directory.cd cannot be used with arbitrary directory names (such as untrusted user--. POSIX- changes directory to $OLDPWD. On zsh (even in sh mode+12 or -345 represent./.Thus, robust and portable use of cd in scripts is unreasonably difficult.
The modernish chdir function calls cd in a way that takes care of all
these issues automatically: it disables $CDPATH and special operand
meanings, and resolves symbolic links by default.
Usage: chdir [ -f ] [ -L ] [ -P ] [ -- ] directorypath
Normally, failure to change the present working directory to directorypath
is a fatal error that ends the program. To tolerate failure, add the -f
option; in that case, exit status 0 signifies success and exit status 1
signifies failure, and scripts should always check and handle exceptions.
The options -L (logical: don’t resolve symlinks) and -P (physical:
resolve symlinks) are the same as in cd, except that -P is the default.
Note that on a shell with BUG_CDNOLOGIC (NetBSD sh),
the -L option to chdir does nothing.
To use arbitrary directory names (e.g. directory names input by the user or
other untrusted input) always use the -- separator that signals the end of
options, or paths starting with - may be misinterpreted as options.
insubshellThe insubshell function checks if you’re currently running in a
subshell environment
(usually called simply subshell).
A subshell is a copy of the parent shell that starts out as an exact
duplicate (including non-exported variables, functions, etc.), except for
traps. A new subshell is invoked by constructs like (parentheses),$(command substitutions), pipe|lines, and & (to launch a background
subshell). Upon exiting a subshell, all changes to its state are lost.
This is not to be confused with a newly initialised shell that is
merely a child process of the current shell, which is sometimes
(confusingly and wrongly) called a “subshell” as well.
This documentation avoids such a misleading use of the term.
Usage: insubshell [ -p | -u ]
This function returns success (0) if it was called from within a subshell
and non-success (1) if not. One of two options can be given:
-p: Store the process ID (PID) of the current subshell or main shellREPLY.-u: Store an identifier in REPLY that is useful for determining ifissetisset checks if a variable, shell function or option is set, or has
certain attributes. Usage:
isset varname: Check if a variable is set.isset -v varname: Id.isset -x varname: Check if variable is exported.isset -r varname: Check if variable is read-only.isset -f funcname: Check if a shell function is set.isset -optionletter (e.g. isset -C): Check if shell option is set.isset -o optionname: Check if shell option is set by long name.Exit status: 0 if the item is set; 1 if not; 2 if the argument is not
recognised as a valid identifier.
Unlike most other modernish commands, isset does not treat an invalid
identifier as a fatal error.
When checking a shell option, a nonexistent shell option is not an error,
but returns the same result as an unset shell option. (To check if a shell
option exists, use thisshellhas.
Note: just isset -f checks if shell option -f (a.k.a. -o noglob) is
set, but with an extra argument, it checks if a shell function is set.
Similarly, isset -x checks if shell option -x (a.k.a -o xtrace)
is set, but isset -x varname checks if a variable is exported. If you
use unquoted variable expansions here, make sure they’re not empty, or
the shell’s empty removal mechanism will cause the wrong thing to be checked
(even in the safe mode).
setstatussetstatus manually sets the exit status $? to the desired value. The
function exits with the status indicated. This is useful in conditional
constructs if you want to prepare a particular exit status for a subsequentexit or return command to inherit under certain circumstances.
The status argument is a parsed as a shell arithmetic expression. A negative
value is treated as a fatal error. The behaviour of values greater than 255
is not standardised and depends on your particular shell.
The test/[ command is the bane of casual shell scripters. Even advanced
shell programmers are frequently caught unaware by one of the many pitfalls
of its arcane, hackish syntax. It attempts to look like shell grammar without
being shell grammar, causing myriad problems
(1,
2).
Its -a, -o, ( and ) operators are inherently and fatally broken as
there is no way to reliably distinguish operators from operands, so POSIX
deprecates their use;
however, most manual pages do not include this essential information, and
even the few that do will not tell you what to do instead.
Ksh, zsh and bash offer a [[ alternative that fixes many of these problems,
as it is integrated into the shell grammar. Nevertheless, it increases
confusion, as entirely different grammar and quoting rules apply
within [[…]] than outside it, yet many scripts end up using them
interchangeably. It is also not available on all POSIX shells. (To make
matters worse, Busybox ash has a false-friend [[ that is just an alias
of [, with none of the shell grammar integration!)
Finally, the POSIX test/[ command is incompatible with the modernish
“safe mode” which aims to eliminate most of the need to quote variables.
See use safe for more information.
Modernish deprecates test/[ and [[ completely. Instead, it offers a
comprehensive alternative command design that works with the usual shell
grammar in a safer way while offering various feature enhancements. The
following replacements are available:
To test if a string is a valid number in shell syntax, str isint is
available. See String tests.
letAn implementation of let as in ksh, bash and zsh is now available to all
POSIX shells. This makes C-style signed integer arithmetic evaluation
available to every
supported shell,
with the exception of the unary ++ and -- operators
(which are a nonstandard shell capability detected by modernish under the ID ofARITHPP).
This means let should be used for operations and tests, e.g. bothlet "x=5" and if let "x==5"; then… are supported (note: single = for
assignment, double == for comparison). See POSIX
2.6.4 Arithmetic Expansion
for more information on the supported operators.
Multiple expressions are supported, one per argument. The exit status of let
is zero (the shell’s idea of success/true) if the last expression argument
evaluates to non-zero (the arithmetic idea of true), and 1 otherwise.
It is recommended to adopt the habit to quote each let expression with"double quotes", as this consistently makes everything work as expected:
double quotes protect operators that would otherwise be misinterpreted as
shell grammar, while shell expansions starting with $ continue to work.
Various handy functions that make common arithmetic operations
and comparisons easier to program are available from thevar/arith module.
The following notes apply to all commands described in the subsections of
this section:
test/[, paranoid quoting to avoid empty removal is not needed.)The str function offers various operators for tests on strings. For
example, str in $foo "bar" tests if the variable foo contains “bar”.
The str function takes unary (one-argument) operators that check a property
of a single word, binary (two-argument) operators that check a word against a
pattern, as well as an option that makes binary operators check multiple words
against a pattern.
Usage: str operator [ word ]
The word is checked for the property indicated by operator; if the result
is true, str returns status 0, otherwise it returns status 1.
The available unary string test operators are:
empty: The word is empty.isint: The word is a decimal, octal or hexadecimal integer number inlet, $((…)) and otherisvarname: The word is a valid portable shell variable or function name.If word is omitted, it is treated as empty, on the assumption that it is
an unquoted empty variable. Passing more than one argument after the
operator is a fatal error.
Usage: str operator [ [ word ] pattern ]
The word is compared to the pattern according to the operator; if it
matches, str returns status 0, otherwise it returns status 1.
The available binary matching operators are:
eq: word is equal to pattern.ne: word is not equal to pattern.in: word includes pattern.begin: word begins with pattern.end: word ends with pattern.match: word matches pattern as a shell glob patterncase construct).str to return status 2.ematch: word matches pattern as a POSIXthisshellhas WRN_EREMBYTElt: word lexically sorts before (is ‘less than’) pattern.le: word is lexically ‘less than or equal to’ pattern.gt: word lexically sorts after (is ‘greater than’) pattern.ge: word is lexically ‘greater than or equal to’ pattern.If word is omitted, it is treated as empty on the assumption that it is an
unquoted empty variable, and the single remaining argument is assumed to be
the pattern. Similarly, if both word and pattern are omitted, an empty
word is matched against an empty pattern. Passing more than two
arguments after the operator is a fatal error.
Usage: str -M operator [ [ word … ] pattern ]
The -M option causes str to compare any number of words to the
pattern. The available operators are the same as the binary string
matching operators listed above.
All matching words are stored in the REPLY variable, separated
by newline characters ($CCn) if there is more than one match.
If no words match, REPLY is unset.
The exit status returned by str -M is as follows:
Usage example: the following matches a given GNU-style long-form command
line option $1 against a series of available options. To make it possible
for the options to be abbreviated, we check if any of the options begin with
the given argument $1.
if str -M begin --fee --fi --fo --fum --foo --bar --baz --quux "$1"; thenputln "OK. The given option $1 matched $REPLY"elsecase $? in( 1 ) putln "No such option: $1" >&2 ;;( * ) putln "Ambiguous option: $1" "Did you mean:" "$REPLY" >&2 ;;esacfi
These avoid the snags with symlinks you get with [ and [[.
By default, symlinks are not followed. Add -L to operate on files
pointed to by symlinks instead of symlinks themselves (the -L makes
no difference if the operands are not symlinks).
These commands all take one argument. If the argument is absent, they return
false. More than one argument is a fatal error. See notes 1-3 in the
parent section.
is present file: Returns true if the file is present in the file
system (even if it is a broken symlink).
is -L present file: Returns true if the file is present in the file
system and is not a broken symlink.
is sym file: Returns true if the file is a symbolic link (symlink).
is -L sym file: Returns true if the file is a non-broken symlink, i.e.
a symlink that points (either directly or indirectly via other symlinks)
to a non-symlink file that is present in the file system.
is reg file: Returns true if file is a regular data file.
is -L reg file: Returns true if file is either a regular data file
or a symlink pointing (either directly or indirectly via other symlinks)
to a regular data file.
Other commands are available that work exactly like is reg and is -L reg
but test for other file types. To test for them, replace reg with one of:
dir for a directoryfifo for a named pipe (FIFO)socket for a socketblockspecial for a block special filecharspecial for a character special fileThe following notes apply to these commands:
-L before the operator argument, e.g. is -L newer.is newer file1 file2: Compares file timestamps, returning true if file1
is newer than file2. Also returns true if file1 exists, but file2 does
not; this is consistent for all shells (unlike test file1 -nt file2).
is older file1 file2: Compares file timestamps, returning true if file1
is older than file2. Also returns true if file1 does not exist, but file2
does; this is consistent for all shells (unlike test file1 -ot file2).
is samefile file1 file2: Returns true if file1 and file2 are the same
file (hardlinks).
is onsamefs file1 file2: Returns true if file1 and file2 are on the
same file system. If any non-regular, non-directory files are specified, their
parent directory is tested instead of the file itself.
These always follow symlinks.
is nonempty file: Returns true if the file exists, is not a broken
symlink, and is not empty. Unlike [ -s file ], this also works
for directories, as long as you have read permission in them.
is setuid file: Returns true if the file has its set-user-ID flag set.
is setgid file: Returns true if the file has its set-group-ID flag set.
is onterminal FD: Returns true if file descriptor FD is associated
with a terminal. The FD may be a non-negative integer number or one of the
special identifiers stdin, stdout and stderr which are equivalent to
0, 1, and 2. For instance, is onterminal stdout returns true if commands
that write to standard output (FD 1), such as putln, would write to the
terminal, and false if the output is redirected to a file or pipeline.
Any symlinks given are resolved, as these tests would be meaningless
for a symlink itself.
can read file: True if the file’s permission bits indicate that you can read
the file - i.e., if an r bit is set and applies to your user.
can write file: True if the file’s permission bits indicate that you can
write to the file: for non-directories, if a w bit is set and applies to your
user; for directories, both w and x.
can exec file: True if the file’s type and permission bits indicate that
you can execute the file: for regular files, if an x bit is set and applies
to your user; for other file types, never.
can traverse file: True if the file is a directory and its permission bits
indicate that a path can traverse through it to reach its subdirectories: for
directories, if an x bit is set and applies to your user; for other file
types, never.
In modernish, every variable and shell option gets its own stack. Arbitrary
values/states can be pushed onto the stack and popped off it in reverse
order. For variables, both the value and the set/unset state is (re)stored.
Usage:
push [ --key=value ] item [ item … ]pop [ --keepstatus ] [ --key=value ] item [ item … ]where item is a valid portable variable name, a short-form shell option
(dash plus letter), or a long-form shell option (-o followed by an option
name, as two arguments).
Before pushing or popping anything, both functions check if all the given
arguments are valid and pop checks all items have a non-empty stack. This
allows pushing and popping groups of items with a check for the integrity of
the entire group. pop exits with status 0 if all items were popped
successfully, and with status 1 if one or more of the given items could not
be popped (and no action was taken at all).
The --key= option is an advanced feature that can help different modules
or functions to use the same variable stack safely. If a key is given topush, then for each item, the given key value is stored along with the
variable’s value for that position in the stack. Subsequently, restoring
that value with pop will only succeed if the key option with the same key
value is given to the pop invocation. Similarly, popping a keyless value
only succeeds if no key is given to pop. If there is any key mismatch, no
changes are made and pop returns status 2. Note that this is
a robustness/convenience feature, not a security feature; the keys are not
hidden in any way.
If the --keepstatus option is given, pop will exit with the
exit status of the command executed immediately prior to calling pop. This
can avoid the need for awkward workarounds when restoring variables or shell
options at the end of a function. However, note that this makes failure to pop
(stack empty or key mismatch) a fatal error that kills the program, as pop
no longer has a way to communicate this through its exit status.
push and pop allow saving and restoring the state of any shell option
available to the set builtin. The precise shell options supported
(other than the ones guaranteed by POSIX) depend on
the shell modernish is running on.
To facilitate portability, nonexistent shell options are treated as unset.
Long-form shell options are matched to their equivalent short-form shell
options, if they exist. For instance, on all POSIX shells, -f is
equivalent to -o noglob, and push -o noglob followed by pop -f works
correctly. This also works for shell-specific short & long option
equivalents.
On shells with a dynamic no option name prefix, that is on ksh, zsh and
yash (where, for example, noglob is the opposite of glob), the no
prefix is ignored, so something like push -o glob followed by pop -o
noglob does the right thing. But this depends on the shell and should never
be used in portable scripts.
Modernish can also make traps stack-based, so that each
program component or library module can set its own trap commands
without interfering with others. This functionality is provided
by the var/stack/trap module.
As modularity is one of modernish’s
design principles,
much of its essential functionality is provided in the form of loadable
modules, so the core library is kept lean. Modules are organised
hierarchically, with names such as safe, var/loop and sys/cmd/harden. Theuse command loads and initialises a module or a combined directory of modules.
Internally, modules exist in files with the name extension .mm in
subdirectories of lib/modernish/mdl – for example, the modulevar/stack/trap corresponds to the file lib/modernish/mdl/var/stack/trap.mm.
Usage:
use modulename [ argument … ]use [ -q | -e ] modulenameuse -lThe first form loads and initialises a module. All arguments, including the
module name, are passed on to the dot script unmodified, so modules know
their own name and can implement option parsing to influence their
initialisation. See also
Two basic forms of a modernish program
for information on how to use modules in portable-form scripts.
In the second form, the -q option queries if a module is loaded, and the -e
option queries if a module exists. use returns status 0 for yes, 1 for no,
and 2 if the module name is invalid.
The -l option lists all currently loaded modules in the order in which
they were originally loaded. Just add | sort for alphabetical order.
If a directory of modules, such as sys/cmd or even just sys, is given as the
modulename, then all the modules in that directory and any subdirectories are
loaded recursively. In this case, passing extra arguments is a fatal error.
If a module file X.mm exists along with a directory X, resolving to the
same modulename, then use will load the X.mm module file without
automatically loading any modules in the X directory, because it is expected
that X.mm handles the submodules in X manually. (This is currently the case
for var/loop which auto-loads submodules containing loop types on first use).
The complete lib/modernish/mdl directory path, which depends on where
modernish is installed, is stored in the system constant $MSH_MDL.
The following subchapters document the modules that come with modernish.
use safeThe safe module sets the ‘safe mode’ for the shell. It removes most of the
need to quote variables, parameter expansions, command substitutions, or glob
patterns. It uses shell settings and modernish library functionality to secure
and demystify split and glob mechanisms. This creates a new and safer way of
shell script programming, essentially building a new shell language dialect
while still running on all POSIX-compliant shells.
One of the most common headaches with shell scripting is caused by a
fundamental flaw in the shell as a scripting language: constantly
active field splitting (a.k.a. word splitting) and pathname expansion
(a.k.a. globbing). To cope with this situation, it is hammered into
programmers of shell scripts to be absolutely paranoid about properly
quoting nearly everything, including
variable and parameter expansions, command substitutions, and patterns passed
to commands like find.
These mechanisms were designed for interactive command line usage, where they
do come in very handy. But when the shell language is used as a programming
language, splitting and globbing often ends up being applied unexpectedly to
unquoted expansions and command substitutions, helping cause thousands of
buggy, brittle, or outright dangerous shell scripts.
One could blame the programmer for forgetting to quote an expansion properly,
or one could blame a pitfall-ridden scripting language design where hammering
punctilious and counterintuitive habits into casual shell script programmers is
necessary. Modernish does the latter, then fixes it.
Every POSIX shell comes with a little-used ability to disable global field
splitting and pathname expansion: IFS=''; set -f. An empty IFS variable
disables split; the -f (or -o noglob) shell option disables pathname
expansion. The safe mode sets these, and two others (see below).
The reason these safer settings are hardly ever used is that they are not
practical to use with the standard shell language. For instance, for
textfile in *.txt, or for item in $(some command) which both (!)
field-splits and pathname-expands the output of a command, all break.
However, that is where modernish comes in. It introduces several powerful
new loop constructs, as well as arbitrary code
blocks with local settings, each of which
has straightforward, intuitive operators for safely applying field splitting
or pathname expansion – to specific command arguments only. By default,
they are not both applied to the arguments, which is much safer. And your
script code as a whole is kept safe from them at all times.
With global field splitting and pathname expansion removed, a third issue
still affects the safe mode: the shell’s empty removal mechanism. If the
value of an unquoted expansion like $var is empty, it will not expand to
an empty argument, but will be removed altogether, as if it were never
there. This behaviour cannot be disabled.
Thankfully, the vast majority of shell and Un*x commands order their arguments
in a way that is actually designed with empty removal in mind, making it a
good thing. For instance, when doing ls $option some_dir, if $option is-l the listing will be long-format and if is empty it will be removed, which
is the desired behaviour. (An empty argument there would cause an error.)
However, one command that is used in almost all shell scripts, test/[,
is completely unable to cope with empty removal due to its idiosyncratic
and counterintuitive syntax. Potentially empty operands come before options,
so operands removed as empty expansions cause errors or, worse, false
positives. Thus, the safe mode does not remove the need for paranoid
quoting of expansions used with test/[ commands. Modernish fixes
this issue by deprecating test/[ completely and offering
a safe command design
to use instead, which correctly deals with empty removal.
With the ‘safe mode’ shell settings, plus the safe, explicit and readable
split and glob operators and test/[ replacements, the only quoting
requirements left are:
"$@" and "$*" until shell bugs are fixed (see notes below).In addition to the above, the safe mode also sets these shell options:
set -C (set -o noclobber) to prevent accidentally overwriting files using>| instead of >.set -u (set -o nounset) to make it an error to use unset (that is,${var-} instead of $var (be careful).safe module. It is highlyuse safe.case constructs. The safe mode changes nothing here.BUG_PP_*, the positional$@ and $* should still always be quoted. As ofBUG_PP_* shell bugs, quoting "$@""$*" remains mandatory even in safe mode (unless you know with"$*" changes in safe mode. It uses the first character$IFS as the separator for combining all positional parameters intoIFS is emptied in safe mode, there is no separator,push IFS; IFS=' '; var="$*"; pop IFSLOCAL IFS=' '; BEGIN var="$*"; ENDput"$@" instead.)Usage: use safe [ -k | -K ] [ -i ]
The -k and -K module options install an extra handler that
reliably kills the program
if it tries to execute a command that is not found, on shells that have the
ability to catch and handle ‘command not found’ errors (currently bash, yash,
and zsh). This helps catch typos, forgetting to load a module, etc., and stops
your program from continuing in an inconsistent state and potentially causing
damage. The MSH_NOT_FOUND_OK variable may be set to temporarily disable this
check. The uppercase -K module option aborts the program on shells that
cannot handle ‘command not found’ errors (so should not be used for portable
scripts), whereas the lowercase -k variant is ignored on such shells.
If the -i option is given, or the shell is interactive, two extra one-letter
functions are loaded, s and g. These are pre-command modifiers for use when
split and glob are globally disabled; they allow running a single command with
local split and glob applied to that command’s arguments only. They also have
some options designed to manipulate, examine, save, restore, and generally
experiment with the global split and glob state on interactive shells. Types --help and g --help for more information. In general, the safe mode is
designed for scripts and is not recommended for interactive shells.
use var/loopThe var/loop module provides an innovative, robust and extensible
shell loop construct. Several powerful loop types are provided, while
advanced shell programmers may find it easy and fun to
create their own.
This construct is also ideal for the
safe mode:
the for, select and find loop types allow you to selectively
apply field splitting and/or pathname expansion to specific arguments
without subjecting a single line of your code to them.
The basic form is a bit different from native shell loops. Note the caps:LOOP looptype arguments; DO
your commands hereDONE
The familiar do…done block syntax cannot be used because the shell
will not allow modernish to add its own functionality to it. TheDO…DONE block does behave in the same way as do…done: you can
append redirections at the end, pipe commands into a loop, etc. as usual.
The break and continue shell builtin commands also work as normal.
Remember: using lowercase do…done with modernish LOOP will
cause the shell to throw a misleading syntax error. So will using uppercaseDO…DONE with the shell’s native loops. To help you remember to use the
uppercase variants for modernish loops, the LOOP keyword itself is also in
capitals.
Loops exist in submodules of var/loop named after the loop type; for
instance, the find loop lives in the var/loop/find module. However, the
core var/loop module will automatically load a loop type’s module when
that loop is first used, so use-ing individual loop submodules at your
script’s startup time is optional.
The LOOP block internally uses file descriptor 8 to do
its thing.
If your script happens to use FD 8 for other purposes, you should
know that FD 8 is made local to each loop block, and always appears
initially closed within DO…DONE.
This simply iterates the loop the number of times indicated. Before the first
iteration, the argument is evaluated as a shell integer arithmetic expression
as in let
and its value used as the number of iterations.
LOOP repeat 3; DOputln "This line is repeated 3 times."DONE
for loopThis is a slightly enhanced version of theFOR loop in BASIC.
It is more versatile than the repeat loop but still very easy to use.
LOOP for varname=initial to limit [ step increment ]; DO
some commandsDONE
To count from 1 to 20 in steps of 2:
LOOP for i=1 to 20 step 2; DOputln "$i"DONE
Note the varname=initial needs to be one argument as in a shell
assignment (so no spaces around the =).
If “step increment“ is omitted, increment defaults to 1 if limit is
equal to or greater than initial, or to -1 if limit is less than
initial (so counting backwards ‘just works’).
Technically precise description: On entry, the initial, limit and
increment values are evaluated once as shell arithmetic expressions as inlet,
the value of initial is assigned to varname, and the loop iterates.
Before every subsequent iteration, the value of increment (as determined
on the first iteration) is added to the value of varname, then the limit
expression is re-evaluated; as long as the current value of varname is
less (if increment is non-negative) or greater (if increment is
negative) than or equal to the current value of limit, the loop reiterates.
for loopA C-style for loop akin to for (( )) in ksh93, bash and zsh is now
available on all POSIX-compliant shells, with a slightly different syntax.
The one loop argument contains three arithmetic expressions (as inlet),
separated by semicolons within that argument. The first is only evaluated
before the first iteration, so is typically used to assign an initial value.
The second is evaluated before each iteration to check whether to continue
the loop, so it typically contains some comparison operator. The third is
evaluated before the second and further iterations, and typically increases
or decreases a value. For example, to count from 1 to 10:
LOOP for "i=1; i<=10; i+=1"; DOputln "$i"DONE
However, using complex expressions allows doing much more powerful things.
Any or all of the three expressions may also be left empty (with their
separating ; character remaining). If the second expression is empty, it
defaults to 1, creating an infinite loop.
(Note that ++i and i++ can only be used on shells withARITHPP,
but i+=1 or i=i+1 can be used on all POSIX-compliant shells.)
for/select loop with safe split/globThe enumarative for and select loop types mirror those already present in
native shell implementations. However, the modernish versions provide safe
field splitting and globbing (pathname expansion) functionality that can be
used without globally enabling split or glob for any of your code – ideal
for the safe mode. They also add a unique operator
for processing text in fixed-size slices. The select loop type bringsselect functionality to all POSIX shells and not just ksh, zsh and bash.
Usage:
LOOP [ for | select ] [ operators ] varname in argument … ;DO commands ; DONE
Simple usage example:
LOOP select --glob textfile in *.txt; DOputln "You chose text file $textfile."DONE
If the loop type is for, the loop iterates once for each argument, storing
it in the variable named varname.
If the loop type is select, the loop presents before each iteration a
numbered menu that allows the user to select one of the arguments. The prompt
from the PS3 variable is displayed and a reply read from standard input. The
literal reply is stored in the REPLY variable. If the reply was a number
corresponding to an argument in the menu, that argument is stored in the
variable named varname. Then the loop iterates. If the user enters ^D (end of
file), REPLY is cleared and the loop breaks with an exit status of 1. (To
break the menu loop under other conditions, use the break command.)
The following operators are supported. Note that the split and glob
operators are only for use in the safe mode.
--split or --split=characters. This operator safely applies--split operator applies the shell’s default field splitting by space,QRK_IFSFINAL treat both whitespace and--glob or --fglob. These operators safely apply shell pathname- or + or is identical to ! or(, ./ is prefixed to keep various commands from misparsing it as an--glob: Any non-matching patterns are quietly removed. If none match,--fglob: All patterns must match. Any nonexistent path terminates the--base=string. This operator prefixes the given string to each of the--glob or --fglob are given, then the string is used as a base--glob was given, the loop--fglob was given, the program terminates.--slice or --slice=number. This operator divides theWRN_MULTIBYTE willIf multiple operators are given, their mechanisms are applied in the
following order: split, glob, base, slice.
find loopThis powerful loop type turns your local POSIX-compliantfind utility
into a shell loop, safely integrating both find
and xargs functionality into the POSIX shell. The infamous
pitfalls and limitations
of using find and xargs as external commands are gone, as all
the results from find are readily available to your main shell
script. Any “dangerous” characters in file names (including
whitespace and even newlines) “just work”, especially if the
safe mode
is also active. This gives you the flexibility to use either the find
expression syntax, or shell commands (including your own shell functions), or
some combination of both, to decide whether and how to handle each file found.
Usage:
LOOP find [ options ] varname [ in path … ]
[ find-expression ] ; DO commands ; DONE
LOOP find [ options ] --xargs[=arrayname] [ in path … ]
[ find-expression ] ; DO commands ; DONE
The loop recursively walks down the directory tree for each path given.
For each file encountered, it uses the find-expression to decide
whether to iterate the loop with the path to the file stored in the
variable referenced by varname. The find-expression is a standardfind
utility expression except as described below.
Any number of paths to search may be specified after the in keyword.
By default, a nonexistent path is a fatal error.
The entire in clause may be omitted, in which case it defaults to in .
so the current working directory will be searched. Any argument that starts
with a -, or is identical to ! or (, indicates the end of the paths
and the beginning of the find-expression; if you need to explicitly
specify a path with such a name, prefix ./ to it.
Except for syntax errors, any errors or warnings issued by find are
considered non-fatal and will cause the exit status of the loop to be
non-zero, so your script has the opportunity to handle the exception.
find utility. Note that-H and -L only, so portable scripts should only use these.-f on BSD find) are not supported.--xargs. This operator is specified instead of the varname; it is a"$@" and--xargs therefore overwrites the current PPs (however,LOCAL block will givebreak), the last chunk survives.KSHARRAY--xargs=arrayname which uses the named--try. If this option is specified, then if one of the primaries used infind utility used byLOOP find will not throw aREPLY variable.--split or --split=characters. This operator, which is onlyfind--split operator applies the shell’s default- or + or is identical to ! or (, ./ is prefixed.--glob or --fglob. These operators are only accepted in thefind utility- or + or is identical to ! or (, ./ is--glob: Any pattern not matching an existing path will output a--fglob: Any pattern not matching an existing path is a fatal error.--base=basedirectory. This operator prefixes the given basedirectoryfind), after--glob or --fglob are given, then wildcard characters are only--fglob was given, the program terminates.LOOP find can use all expression operands supported by your local find
utility; see its manual page. However, portable scripts should use only
operands specified by POSIX
along with the modernish additions described below.
The modernish -iterate expression primary evaluates as true and causes the
loop to iterate, executing your commands for each matching file. It may be
used any number of times in the find-expression to start a corresponding
series of loop iterations. If it is not given, the loop acts as if the entire
find-expression is enclosed in parentheses with -iterate appended. If the
entire find-expression is omitted, it defaults to -iterate.
The modernish -ask primary asks confirmation of the user. The text of the
prompt may be specified in one optional argument (which cannot start with -
or be equal to ! or (). Any occurrences of the characters {} within the
prompt text are replaced with the current pathname. If not specified, the
default prompt is: "{}"? If the answer is affirmative (y or Y in the
POSIX locale), -ask yields true, otherwise false. This can be used to make
any part of the expression conditional upon user input, and (unlike commands in
the shell loop body) is capable of influencing directory traversal mid-run.
The standard -exec and -ok primaries are integrated into the main shell
environment. When used with LOOP find, they can call a shell builtin command
or your own shell function directly in the main shell (no subshell). Its exit
status is used in the find expression as a true/false value capable of
influencing directory traversal (for example, when combined with -prune),
just as if it were an external command -exec’ed with the standard utility.
Some familiar, easy-to-use but non-standard find operands from GNU and/or
BSD may be used with LOOP find on all systems. Before invoking the find
utility, modernish translates them internally to portable equivalents.
The following expression operands are made portable:
-or, -and and -not operators: same as -o, -a, !.-true and -false primaries, which always yield true/false.-depth n primary, e.g. -depth +4 yields true on depth-depth -4 yields true on depth less than 4-depth 4 yields true on a depth of exactly 4.-mindepth and -maxdepth global options.-depth, these GNU-isms are pseudo-primaries thatLOOP find operation.Expression primaries that write output (-print and friends) may be used for
debugging or logging the loop. Their output is redirected to standard error.
find utilityUpon initialisation, the var/loop/find module searches for a POSIX-compliantfind utility under various names in $DEFPATH and then in $PATH. To see a
trace of the full command lines of utility invocations when the loop runs, set
the _loop_DEBUG variable to any value.
For debugging or system-specific usage, it is possible to use a certain find
utility in preference to any others on the system. To do this, add an argument
to a use var/loop/find command before the first use of the loop. For example:
use var/loop/find bsdfind (prefer utility by this name)use var/loop/find /opt/local/bin (look for a utility here first)use var/loop/find /opt/local/bin/gfind (try this one first)find utilitiesSome systems come with obsolete or broken find utilities that don’t fully
support -exec ... {} + aggregating functionality as specified by POSIX.
Normally, this is a fatal error, but passing the -b/-B option to theuse command, e.g. use var/loop/find -b, enables a compatibility mode
that tolerates this defect. If no compliant find is found, then an obsolete
or broken find is used as a last resort, a warning is printed to standard
error, and the variable _loop_find_broken is set. The -B option is
equivalent to -b but does not print a warning. Loop performance may suffer as
modernish adapts to using older exec ... {} \; which is very inefficient.
Scripts using this compatibility mode should handle their logic using shell
code in the loop body as much as possible (after DO) and use only simplefind expressions (before DO), as obsolete utilities are often buggy and
breakage is likely if complex expressions or advanced features are used.
find loop usage examplesSimple example script: without the safe mode, the *.txt pattern
must be quoted to prevent it from being expanded by the shell.
. modernishuse var/loopLOOP find TextFile in ~/Documents -name '*.txt'DOputln "Found my text file: $TextFile"DONE
Example script with safe mode: the --glob option
expands the patterns of the in clause, but not the expression – so it
is not necessary to quote any pattern.
. modernishuse safeuse var/loopLOOP find --glob lsProg in /*bin /*/*bin -type f -name ls*DOputln "This command may list something: $lsProg"DONE
Example use of the modernish -ask primary: ask the user if they want to
descend into each directory found. The shell loop body could skip unwanted
results, but cannot physically influence directory traversal, so skipping large
directories would take long. A find expression can prevent directory
traversal using the standard -prune primary, which can be combined with-ask, so that unwanted directories never iterate the loop in the first place.
. modernishuse safeuse var/loopLOOP find file in ~/Documents \-type d \( -ask 'Descend into "{}" directory?' -or -prune \) \-or -iterateDOput "File found: "ls -li $fileDONE
The modernish loop construct is extensible. To define a new loop type, you
only need to define a shell function called _loopgen_type where type
is the loop type. This function, called the loop iteration generator, is
expected to output lines of text to file descriptor 8, containing properly
shell-quoted
iteration commands for the shell to run, one line per iteration.
The internal commands expanded from LOOP, DO and DONE (which are
defined as aliases) launch that loop iteration generator function in the
background with safe mode enabled, while causing
the main shell to read lines from that background process through a pipe,evaling each line as a command before iterating the loop. As long as that
iteration command finishes with an exit status of zero, the loop keeps
iterating. If it has a nonzero exit status or if there are no more commands
to read, iteration terminates and execution continues beyond the loop.
Instead of the normal internal namespace
which is considered off-limits for modernish scripts, var/loop and its
submodules use a _loop_* internal namespace for variables, which is also
for use by user-implemented loop iteration generator functions.
The above is just the general principle. For the details, study the comments
and the code in lib/modernish/mdl/var/loop.mm and the loop generators inlib/modernish/mdl/var/loop/*.mm.
use var/localThis module defines a new LOCAL…BEGIN…END shell code block
construct with local variables, local positional parameters and local shell
options. The local positional parameters can be filled using safe field
splitting and pathname expansion operators similar to those in the LOOP
construct described above.
Usage: LOCAL [ localitem | operator … ] [ -- [ word … ] ] ;BEGIN commands ; END
The commands are executed once, with the specified localitems applied.
Each localitem can be:
= immediately followed by a value.- or + sign. Thisset. Long-form shell options like -o+o optionname are also supported. It depends on thethisshellhas to checkModernish implements LOCAL blocks as one-time shell functions that use
the stack
to save and restore variables and settings. So the return command exits the
block, causing the global variables and settings to be restored and resuming
execution at the point immediately following END. Like any shell function, aLOCAL block exits with the exit status of the last command executed within
it, or with the status passed on by or given as an argument to return.
The positional parameters ($@, $1, etc.) are always local to the block, but
a copy is inherited from outside the block by default. Any changes to the
positional parameters made within the block will be discarded upon exiting it.
However, if a double-dash -- argument is given in the LOCAL command line,
the positional parameters outside the block are ignored and the set of words
after -- (which may be empty) becomes the positional parameters instead.
These words can be modified prior to entering the LOCAL block using the
following operators. The safe glob and split operators are only accepted in
the safe mode. The operators are:
--split or --split=characters. This operator safely applies--split operator applies the shell’s default field splitting by space,QRK_IFSFINAL treat both whitespace and--glob or --fglob. These operators safely apply shell pathname- or + or is identical to ! or (, ./--glob: Any non-matching patterns are quietly removed.--fglob: All patterns must match. Any nonexistent path terminates the--base=string. This operator prefixes the given string to each of the--glob or --fglob are given, then the string is used as a base--glob was given, all--fglob was given, the program terminates.--slice or --slice=number. This operator divides theWRN_MULTIBYTE willIf multiple operators are given, their mechanisms are applied in the
following order: split, glob, base, slice.
var/local usage notesLOCAL hasBEGIN…END block instead of the shell’s do…done.LOCAL blocks do not mix well with use of the shell capabilityLOCALVARSQRK_LOCALUNS or QRK_LOCALUNS2. Using both with the same variablesbreak or continue within a LOCAL block toQRK_BCDANGER allow this, preventing END fromreturn 1, then append something like || break or || continue toEND. Note that this caveat only applies when crossing BEGIN…ENDcontinue and break to continue or break loopsuse var/arithThese shortcut functions are alternatives for usinglet.
inc, dec, mult, div, mod: simple integer arithmetic shortcuts. The first
argument is a variable name. The optional second argument is an
arithmetic expression, but a sane default value is assumed (1 for inc
and dec, 2 for mult and div, 256 for mod). For instance, inc X is
equivalent to X=$((X+1)) and mult X Y-2 is equivalent to X=$((X*(Y-2))).
ndiv is like div but with correct rounding down for negative numbers.
Standard shell integer division simply chops off any digits after the
decimal point, which has the effect of rounding down for positive numbers
and rounding up for negative numbers. ndiv consistently rounds down.
These have the same name as their test/[ option equivalents. Unlike
with test, the arguments are shell integer arith expressions, which can be
anything from simple numbers to complex expressions. As with $(( )),
variable names are expanded to their values even without the $.
Function: Returns successfully if:eq <expr> <expr> the two expressions evaluate to the same numberne <expr> <expr> the two expressions evaluate to different numberslt <expr> <expr> the 1st expr evaluates to a smaller number than the 2ndle <expr> <expr> the 1st expr eval's to smaller than or equal to the 2ndgt <expr> <expr> the 1st expr evaluates to a greater number than the 2ndge <expr> <expr> the 1st expr eval's to greater than or equal to the 2nd
use var/assignThis module is provided to solve a common POSIX shell language annoyance: in a
normal shell variable assignment, only literal variable names are accepted, so
it is impossible to use a variable whose name is stored in another variable.
The only way around this is to use eval which is too difficult to use safely.
Instead, you can now use the assign command.
Usage: assign [ [ +r ] variable=value … ] | [ -r variable=variable2 … ] …
assign safely processes assignment-arguments in the same form as customarily
given to the readonly and export commands, but it only assigns values to
variables without setting any attributes. Each argument is grammatically an
ordinary shell word, so any part or all of it may result from an expansion. The
absence of a = character in any argument is a fatal error. The text preceding
the first = is taken as the variable name in which to store the value; an
invalid variable name is a fatal error. No whitespace is accepted before the= and any whitespace after the = is part of the value to be assigned.
The -r (reference) option causes the part to the right of the = to be
taken as a second variable name variable2, and its value is assigned to
variable instead. +r turns this option back off.
Examples: Each of the lines below assigns the value ‘hello world’ to the
variable greeting.
var=greeting; assign $var='hello world'var=greeting; assign "$var=hello world"tag='greeting=hello world'; assign "$tag"var=greeting; gvar=myinput; myinput='hello world'; assign -r $var=$gvar
use var/readfreadf reads arbitrary data from standard input into a variable until end
of file, converting it into a format suitable for passing to theprintf
utility. For example, readf var <foo; printf "$var" >bar will copy foo to
bar. Thus, readf allows storing both text and binary files into shell
variables in a textual format suitable for manipulation with standard shell
facilities.
All non-printable, non-ASCII characters are converted to printf octal or
one-letter escape codes, except newlines. Not encoding newline characters
allows for better processing by line-based utilities such as grep, sed,awk, etc. However, if the file ends in a newline, that final newline is
encoded to \n to protect it from being stripped by command substitutions.
Usage: readf [ -h ] varname
The -h option disables conversion of high-byte characters (accented letters,
non-Latin scripts). Do not use for binary files; this is only guaranteed to
work for text files in an encoding compatible with the current locale.
Caveats:
printf format typicallyprintf as a builtinprintf command will fail if the encoded filegetconf ARG_MAX will obtain this limit for your system). Shell builtinprintf builtin usingthisshellhas if you need to be sure,hardenprintf!use var/shellquoteThis module provides an efficient, fast, safe and portable shell-quoting
algorithm for quoting arbitrary data in such a way that the quoted values are
safe to pass to the shell for parsing as string literals. This is essential
for any context where the shell must grammatically parse untrusted input,
such as when supplying arbitrary values to trap or eval.
The shell-quoting algorithm is optimised to minimise exponential growth when
quoting repeatedly. By default, it also ensures that quoted strings are
always one single printable line, making them safe for terminal output and
processing by line-oriented utilities.
shellquoteUsage: shellquote [ -f|+f|-P|+P ] varname[=value] …
The values of the variables specified by name are shell-quoted and stored
back into those variables.
Repeating a variable name will add another level of shell-quoting.
If a = plus a value (which may be empty) is appended to the varname,
that value is shell-quoted and assigned to the variable.
Options modify the algorithm for variable names following them, as follows:
By default, newlines and any control characters are converted into${CC*}
expansions and quoted with double quotes, ensuring that the quoted string
consists of a single line of printable text. The -P option forces pure
POSIX quoted strings that may span multiple lines; +P turns this back off.
By default, a value is only quoted if it contains characters not present
in $SHELLSAFECHARS. The -f option forces unconditional quoting,
disabling optimisations that may leave shell-safe characters unquoted;+f turns this back off.
shellquote will die if you
attempt to quote an unset variable (because there is no value to quote).
shellquoteparamsThe shellquoteparams command shell-quotes the current positional
parameters in place using the default quoting method of shellquote. No
options are supported and any attempt to add arguments results in a syntax
error.
use var/stackModules that extend the stack.
use var/stack/extraThis module contains stack query and maintenance functions.
If you only need one or two of these functions, they can also be loaded as
individual submodules of var/stack/extra.
For the four functions below, item can be:
-o followed by an option name (two arguments)--trap=SIGNAME to refer to the trap stack for the indicated signalpushtrap from var/stack/trap)stackempty [ --key=value ] [ --force ] item: Tests if the stack
for an item is empty. Returns status 0 if it is, 1 if it is not. The key
feature works as in pop: by default, a key
mismatch is considered equivalent to an empty stack. If --force is given,
this function ignores keys altogether.
clearstack [ --key=value ] [ --force ] item [ item … ]:
Clears one or more stacks, discarding all items on it.
If (part of) the stack is keyed or a --key is given, only clears until a
key mismatch is encountered. The --force option overrides this and always
clears the entire stack (be careful, e.g. don’t use withinLOCAL … BEGIN … END).
Returns status 0 on success, 1 if that stack was already empty, 2 if
there was nothing to clear due to a key mismatch.
stacksize [ --silent | --quiet ] item: Leaves the size of a stack in
the REPLY variable and, if option --silent or --quiet is not given,
writes it to standard output.
The size of the complete stack is returned, even if some values are keyed.
printstack [ --quote ] item: Outputs a stack’s content.
Option --quote shell-quotes each stack value before printing it, allowing
for parsing multi-line or otherwise complicated values.
Column 1 to 7 of the output contain the number of the item (down to 0).
If the item is set, column 8 and 9 contain a colon and a space, and
if the value is non-empty or quoted, column 10 and up contain the value.
Sets of values that were pushed with a key are started with a special
line containing --- key:value. A subsequent set pushed with no key is
started with a line containing --- (key off).
Returns status 0 on success, 1 if that stack is empty.
use var/stack/trapThis module provides pushtrap and poptrap. These functions integrate
with the main modernish stack
to make traps stack-based, so that each
program component or library module can set its own trap commands without
interfering with others.
This module also provides a newDIE pseudosignal
that allows pushing traps to execute whendie
is called.
Note an important difference between the trap stack and stacks for variables
and shell options: pushing traps does not save them for restoring later, but
adds them alongside other traps on the same signal. All pushed traps are
active at the same time and are executed from last-pushed to first-pushed
when the respective signal is triggered. Traps cannot be pushed and popped
using push and pop but use dedicated commands as follows.
Usage:
pushtrap [ --key=value ] [ --nosubshell ] [ -- ] command sigspec [ sigspec … ]poptrap [ --key=value ] [ -R ] [ -- ] sigspec [ sigspec … ]pushtrap works like regular trap, with the following exceptions:
thisshellhas --sig=yoursignalpushtrap does not accept an empty command as it would be pointless.exit will only exit the trap and not the program.--nosubshell option overrides this behaviour, causing that particularDIE traps.$? initially set to the exit statuspushtrap stores current $IFS (field splitting) and $- (shell options)IFS and the shell options f (noglob), unounset) and C (noclobber) to the values in effect during thepushtrap. This is to avoid unexpected effects in case a trap--nosubshell option disables this functionality for the trap pushed.--key option applies the keying functionality inherited frompush to the trap stack.poptrap takes just signal names or numbers as arguments. It takes the
last-pushed trap for each signal off the stack. By default, it discards
the trap commands. If the -R option is given, it stores commands to
restore those traps into the REPLY variable, in a format suitable for
re-entry into the shell. Again, the --key option works as in
plain pop.
With the sole exception ofDIE traps,
all stack-based traps, like native shell traps, are reset upon entering a
subshell.
However, commands for printing traps will print the traps for
the parent shell, until another trap, pushtrap or poptrap command is
invoked, at which point all memory of the parent shell’s traps is erased.
Modernish tries hard to avoid incompatibilities with existing trap practice.
To that end, it intercepts the regular POSIX trap command using an alias,
reimplementing and interfacing it with the shell’s builtin trap facility
so that plain old regular traps play nicely with the trap stack. You should
not notice any changes in the POSIX trap command’s behaviour, except for
the following:
trap command does not overwrite stack traps (but stillSIG prefix;trap command with no arguments, which prints the traps that are setpushtrap commands. (bash users might notice the SIG-p option, including its yash-style --printvar=$(trap)) now works on everytrap command accepts bothtrap INT to unset a SIGINTtrap command is given exactly onetrap alias to set a trap using the shell builtin commandtrapPOSIX traps for each signal are always executed after that signal’s stack-based
traps; this means they should not rely on modernish modules that use the trap
stack to clean up after themselves on exit, as those cleanups would already
have been done.
DIE pseudosignalThe var/stack/trap module adds new DIE pseudosignal whose traps are
executed upon invoking die.
This allows for emergency cleanup operations upon fatal program failure,
as EXIT traps cannot be executed after die is invoked.
DIE is its own pseudosignal with its own trapdieDIE trap simultaneously as a background job, then gathersSIGKILLDIE trap processes. Unlike other traps, DIEpushtrap maydie willDIE trap actions before being SIGKILLed itself. (Note that anyDIE traps pushed or set within a subshell will still be forgotten uponDIE is simply an alias for INT, and INT trapsdie uses SIGINT for command interruption on interactive shells, andINT traps on interactive shells douse var/stringString comparison and manipulation functions.
use var/string/touplowtoupper and tolower: convert case in variables.
Usage:
toupper varname [ varname … ]tolower varname [ varname … ]Arguments are taken as variable names (note: they should be given without
the $) and case is converted in the contents of the specified variables,
without reading input or writing output.
toupper and tolower try hard to use the fastest available method on the
particular shell your program is running on. They use built-in shell
functionality where available and working correctly, otherwise they fall back
on running an external utility.
Which external utility is chosen depends on whether the current locale uses
the Unicode UTF-8 character set or not. For non-UTF-8 locales, modernish
assumes the POSIX/C locale and tr is always used. For UTF-8 locales,
modernish tries hard to find a way to correctly convert case even for
non-Latin alphabets. A few shells have this functionality built in withtypeset. The rest need an external utility. Modernish initialisation
tries tr, awk, GNU awk and GNU sed before giving up and setting
the variable MSH_2UP2LOW_NOUTF8. If isset MSH_2UP2LOW_NOUTF8, it
means modernish is in a UTF-8 locale but has not found a way to convert
case for non-ASCII characters, so toupper and tolower will convert
only ASCII characters and leave any other characters in the string alone.
use var/string/trimtrim: strip whitespace from the beginning and end of a variable’s value.
Whitespace is defined by the [ character class. In the POSIX
]
locale, this is tab, newline, vertical tab, form feed, carriage return, and
space, but in other locales it may be different.
(On shells with BUG_NOCHCLASS,$WHITESPACE
is used to define whitespace instead.) Optionally, a string of literal
characters can be provided in the second argument. Any characters appearing
in that string will then be trimmed instead of whitespace.
Usage: trim varname [ characters ]
use var/string/replaceinreplacein: Replace leading, -trailing or -all occurrences of a string by
another string in a variable.
Usage: replacein [ -t | -a ] varname oldstring newstring
use var/string/appendappend and prepend: Append or prepend zero or more strings to a
variable, separated by a string of zero or more characters, avoiding the
hairy problem of dangling separators.
Usage: append|prepend [ --sep=separator ] [ -Q ] varname [ string … ]
If the separator is not specified, it defaults to a space character.
If the -Q option is given, each string is
shell-quoted
before appending or prepending.
use var/unexportThe unexport function clears the “export” bit of a variable, conserving
its value, and/or assigns values to variables without setting the export
bit. This works even if set -a (allexport) is active, allowing an “export
all variables, except these” way of working.
Usage is like export, with the caveat that variable assignment arguments
containing non-shell-safe characters or expansions must be quoted as
appropriate, unlike in some specific shell implementations of export.
(To get rid of that headache, use safe.)
Unlike export, unexport does not work for read-only variables.
use var/genoptparserAs the getopts builtin is not portable when used in functions, this module
provides a command that generates modernish code to parse options for your
shell function in a standards-compliant manner. The generated parser
supports short-form (one-character) options which can be stacked/combined.
Usage:generateoptionparser [ -o ] [ -f func ] [ -v varprefix ]
[ -n options ] [ -a options ] [ varname ]
-o: Write parser to standard output.-f: Function name to prefix to error messages. Default: none.-v: Variable name prefix for options. Default: opt_.-n: String of options that do not take arguments.-a: String of options that require arguments.REPLY.At least one of -n and -a is required. All other arguments are optional.
Option characters must be valid components of portable variable names, so
they must be ASCII upper- or lowercase letters, digits, or the underscore.
generateoptionparser stores the generated parser code in a variable: eitherREPLY or the varname specified as the first non-option argument. This makes
it possible to generate and use the parser on the fly with a command likeeval "$REPLY" immediately following the generateoptionparser invocation.
For better efficiency and readability, it will often be preferable to insert
the option parser code directly into your shell function instead. The -o
option writes the parser code to standard output, so it can be redirected to
a file, inserted into your editor, etc.
Parsed options are shifted out of the positional parameters while setting or
unsetting corresponding variables, until a non-option argument, a --
end-of-options delimiter argument, or the end of arguments is encountered.
Unlike with getopts, no additional shift command is required.
Each specified option gets a corresponding variable with a name consisting
of the varprefix (default: opt_) plus the option character. If an option
is not passed to your function, the parser unsets its variable; otherwise it
sets it to either the empty value or its option-argument if it requires one.
Thus, your function can check if any option x was given usingisset,
for example, if isset opt_x; then…
use sys/baseSome very common and essential utilities are not specified by POSIX, differ
widely among systems, and are not always available. For instance, thewhich and readlink commands have incompatible options on various GNU and
BSD variants and may be absent on other Unix-like systems. The sys/base
module provides a complete re-implementation of such non-standard but basic
utilities, written as modernish shell functions. Using the modernish version
of these utilities can help a script to be fully portable. These versions
also have various enhancements over the GNU and BSD originals, some of which
are made possible by their integration into the modernish shell environment.
use sys/base/mktempA cross-platform shell implementation of mktemp that aims to be just as
safe as native mktemp(1) implementations, while avoiding the problem of
having various mutually incompatible versions and adding several unique
features of its own.
Creates one or more unique temporary files, directories or named pipes,
atomically (i.e. avoiding race conditions) and with safe permissions.
The path name(s) are stored in REPLY and optionally written to stdout.
Usage: mktemp [ -dFsQCt ] [ template … ]
-d: Create a directory instead of a regular file.-F: Create a FIFO (named pipe) instead of a regular file.-s: Silent. Store output in $REPLY, don’t write any output or message.-Q: Shell-quote each unit of output. Separate by spaces, not newlines.-C: Automated cleanup.
[Pushes a trap](#user-content-the-trap-stack)to remove the fileson exit. On an interactive shell, that's all this option does. On anon-interactive shell, the following applies: Clean up on receiving`SIGPIPE` and `SIGTERM` as well. On receiving `SIGINT`, clean up if theoption was given at least twice, otherwise notify the user of filesleft. On the invocation of[`die`](#user-content-reliable-emergency-halt),clean up if the option was given at least three times, otherwise notifythe user of files left.
-t: Prefix one temporary files directory to all the templates:
`$XDG_RUNTIME_DIR` or `$TMPDIR` if set, or `/tmp`. The *template*smay not contain any slashes. If the template has neither any trailing`X`es nor a trailing dot, a dot is added before the random suffix.
The template defaults to “/tmp/temp.”. An suffix of random shell-safe ASCII
characters is added to the template to create the file. For compatibility with
other mktemp implementations, any optional trailing X characters in the
template are removed. The length of the suffix will be equal to the amount ofXes removed, or 10, whichever is more. The longer the random suffix, the
higher the security of using mktemp in a shared directory such as tmp.
Since /tmp is a world-writable directory shared by other users, for best
security it is recommended to create a private subdirectory using mktemp -d
and work within that.
Option -C cannot be used without option -s when in a
subshell.
Modernish will detect this and treat it as a
fatal error. The reason is that a typical command substitution liketmpfile=$(mktemp -C)
is incompatible with auto-cleanup, as the cleanup EXIT trap would be
triggered not upon exiting the program but upon exiting the command
substitution subshell that just ran mktemp, thereby immediately undoing
the creation of the file. Instead, do something like:mktemp -sC; tmpfile=$REPLY
This module depends on the trap stack to do auto-cleanup (the -C option),
so it will automatically use var/stack/trap on initialisation.
use sys/base/readlinkreadlink reads the target of a symbolic link, robustly handling strange
filenames such as those containing newline characters. It stores the result
in the REPLY variable and optionally writes it on standard output.
Usage: readlink [ -nsefmQ ] path [ path … ]
-n: If writing output, don’t add a trailing newline.-s: Silent operation: don’t write output, only store it in REPLY.-e, -f, -m: Canonicalise. Convert each path found into a canonical. and .. are resolved. All symlinks encountered are-e: All pathname components must exist to produce a result.-f: All but the last pathname component must exist to produce a result.-m: No pathname component needs to exist; this always produces a result.-Q: Shell-quote each unit of output. Separate by spaces insteadThe exit status of readlink is 0 on success and 1 if the path either is
not a symlink, or could not be canonicalised according to the option given.
use sys/base/revrev copies the specified files to the standard output, reversing the order
of characters in every line. If no files are specified, the standard input
is read.
Usage: like rev on Linux and BSD, which is like cat except that - is
a filename and does not denote standard input. No options are supported.
use sys/base/seqA cross-platform implementation of seq that is more powerful and versatile
than native GNU and BSD seq(1) implementations. The core is written inbc, the POSIX arbitrary-precision calculator language. That means thisseq inherits the capacity to handle numbers with a precision and size only
limited by computer memory, as well as the ability to handle input numbers
in any base from 1 to 16 and produce output in any base 1 and up.
Usage: seq [ -w ] [ -L ] [ -f format ] [ -s string ] [ -S scale ]
[ -B base ] [ -b base ] [ first [ incr ] ] last
seq prints a sequence of arbitrary-precision floating point numbers, one
per line, from first (default 1), to as near last as possible, in increments of
incr (default 1). If first is larger than last, the default incr is -1.
An incr of zero is treated as a fatal error.
-w: Equalise width by padding with leading zeros. The longest of the-L: Use the current locale’s radix point in the output instead of the
full stop (`.`).
-f: printf-style floating-point format. The format string is passed on
(with an added `\n`) to `awk`'s builtin `printf` function. Becauseof that, the `-f` option can only be used if the output base is 10.Note that `awk`'s floating point precision is limited, so verylarge or long numbers will be rounded.
-s: Instead of writing one number per line, write all numbers on one
line separated by *string* and terminated by a newline character.
-S: Explicitly set the scale (number of digits after theDefaults to the largest number of digits after the radix point
[radix point](https://en.wikipedia.org/wiki/Radix_point)).
-B: Set input and output base from 1 to 16. Defaults to 10.-b: Set arbitrary output base from 1. Defaults to input base.
See the `bc`(1) manual for more information on the output formatfor bases greater than 16.
The -S, -B and -b options take shell integer numbers as operands. This
means a leading 0X or 0x denotes a hexadecimal number and a leading 0
denotes an octal number.
For portability reasons, modernish seq uses a full stop (.) for the
radix point, regardless of the
system locale. This applies both to command arguments and to output.
The -L option causes seq to use the current locale’s radix point
character for output only.
seqThe -S, -B and -b options are modernish innovations.
The -w, -f and -s options are inspired by GNU and BSD seq.
The following differences apply:
-s option-t option to-s option-argument is taken as literal\n.seq 5 1 counts backwards from 5 to 1, andseq 5 -1 1 as with GNU is not needed.yes instead.-f option accepts any format specifiersawk‘s printf() function.The sys/base/seq module depends on, and automatically loads,var/string/touplow.
use sys/base/shufShuffle lines of text.
A portable reimplementation of a commonly used GNU utility.
Usage:
shuf [ -n max ] [ -r rfile ] fileshuf [ -n max ] [ -r rfile ] -i low-highshuf [ -n max ] [ -r rfile ] -e argument …By default, shuf reads lines of text from standard input, or from file
(the file - signifies standard input).
It writes the input lines to standard output in random order.
-i: Use sequence of non-negative integers low through high as input.-e: Instead of reading input, use the arguments as lines of input.-n: Output a maximum of max lines.-r: Use rfile as the source of random bytes. Defaults to /dev/urandom.Differences with GNU shuf:
-o/--output-file option is not supported; use output redirection.--random-source=file is changed to -r file.-z/--zero-terminated option is not supported.use sys/base/tactac (the reverse of cat) is a cross-platform reimplementation of the GNUtac utility, with some extra features.
Usage: tac [ -rbBP ] [ -S separator ] file [ file … ]
tac outputs the files in reverse order of lines/records.
If file is - or is not given, tac reads from standard input.
-s: Specify the record (line) separator. Default: linefeed.-r: Interpret the record separator as an-b: Assume the separator comes before each record in the input, and also-B.-B: Assume the separator comes after each record in the input, but output-b.-P: Paragraph mode: output text last paragraph first. Input paragraphsDifferences between GNU tac and modernish tac:
-B and -P options were added.-r option interprets the record separator as an extended regulartac unless expressionsuse sys/base/whichThe modernish which utility finds external programs and reports their
absolute paths, offering several unique options for reporting, formatting
and robust processing. The default operation is similar to GNU which.
Usage: which [ -apqsnQ1f ] [ -P number ] program [ program … ]
By default, which finds the first available path to each given program.
If program is itself a path name (contains a slash), only that path’s base
directory is searched; if it is a simple command name, the current $PATH
is searched. Any relative paths found are converted to absolute paths.
Symbolic links are not followed. The first path found for each program is
written to standard output (one per line), and a warning is written to
standard error for every program not found. The exit status is 0 (success)
if all programs were found, 1 otherwise.
which also leaves its output in the REPLY variable. This may be useful
if you run which in the main shell environment. The REPLY value will
not survive a command substitution
subshell
as in ls_path=$(which ls).
The following options modify the default behaviour described above:
-a: List all programs that can be found in the directories searched,$PATH.-p: Search in $DEFPATHPATH provided by the operating system)$PATH, which is vulnerable to manipulation.-q: Be quiet: suppress all warnings.-s: Silent operation: don’t write output, only store it in the REPLYwhich -s in a subshell,REPLY variable will not survive the subshell.-n: When writing to standard output, do not write a final newline.-Q: Shell-quote each unit of output. Separate by spaces instead-1 (one): Output the results for at most one of the arguments in-s.which -f -1 gnutar gtar tarwhich -1-f: Throw a fatal errorwhich would otherwise return status 1 (non-success).-P: Strip the indicated number of pathname elements from the output,-P1: strip /program;-P2: strip /*/program,--help: Show brief usage information.use sys/base/yesyes very quickly outputs infinite lines of text, each consisting of its
space-separated arguments, until terminated by a signal or by a failure to
write output. If no argument is given, the default line is y. No options
are supported.
This infinite-output command is useful for piping into commands that need an
indefinite input data stream, or to automate a command requiring interactive
confirmation.
Modernish yes is like GNU yes in that it outputs all its arguments,
whereas BSD yes only outputs the first. It can output multiple gigabytes
per second on modern systems.
use sys/cmdModules in this category contain functions for enhancing the invocation of
commands.
use sys/cmd/externextern is like command but always runs an external command, without
having to know or determine its location. This provides an easy way to
bypass a builtin, alias or function. It does the same $PATH search
the shell normally does when running an external command. For instance, to
guarantee running external printf just do: extern printf ...
Usage: extern [ -p ] [ -v ] [ -u varname … ]
[ varname=value … ] command [ argument … ]
-p: The command, as well as any commands it further invokes, are searched in$DEFPATHPATH provided by the operating system)$PATH, which is vulnerable to manipulation.extern -p is much more reliable than the shell’s builtincommand -pcommand -p; (b) command -p does not export the default PATH, socommand -p sudo cp foo /bin/bar searches only sudo incp.-v: don’t execute command but show the full path name of the command thatextern exits with status 0 if all the commands-p.-u: Temporary export override. Unset the given variable in the=value assignment-arguments: These variables/values areextern cause unwanted,extern is a shell function. Beextern instead.-- end-of-options delimiter are not parsed;= sign to be executed.use sys/cmd/hardenThe harden function allows implementing emergency halt on error
for any external commands and shell builtin utilities. It is
modernish’s replacement for set -e a.k.a. set -o errexit (which is
fundamentally
flawed,
not supported and will break the library).
It depends on, and auto-loads, the sys/cmd/extern module.
harden sets a shell function with the same name as the command hardened,
so it can be used transparently. This function hardens the given command by
checking its exit status against values indicating error or system failure.
Exactly what exit statuses signify an error or failure depends on the
command in question; this should be looked up in the
POSIX specification
(under “Utilities”) or in the command’s man page or other documentation.
If the command fails, the function installed by harden calls die, so it
will reliably halt program execution, even if the failure occurred within a
subshell.
Usage:
harden [ -f funcname ] [ -[cSpXtPE] ] [ -e testexpr ]
[ var=value … ] [ -u var … ] command_name_or_path
[ command_argument … ]
The -f option hardens the command as the shell function funcname instead
of defaulting to command_name_or_path as the function name. (If the latter
is a path, that’s always an invalid function name, so the use of -f is
mandatory.) If command_name_or_path is itself a shell function, that
function is bypassed and the builtin or external command by that name is
hardened instead. If no such command is found, harden dies with the message
that hardening shell functions is not supported. (Instead, you should invokedie directly from your shell function upon detecting a fatal error.)
The -c option causes command_name_or_path to be hardened and run
immediately instead of setting a shell function for later use. This option
is meant for commands that run once; it is not efficient for repeated use.
It cannot be used together with the -f option.
The -S option allows specifying several possible names/paths for a
command. It causes the command_name_or_path to be split by comma and
interpreted as multiple names or paths to search. The first name or path
found is used. Requires -f.
The -e option, which defaults to >0, indicates the exit statuses
corresponding to a fatal error. It depends on the command what these are;
consult the POSIX spec and the manual pages.
The status test expression testexpr, argument
to the -e option, is like a shell arithmetic
expression, with the binary operators == != <= >= < > turned
into unary operators referring to the exit status of the command in
question. Assignment operators are disallowed. Everything else is the same,
including && (logical and) and || (logical or) and parentheses.
Note that the expression needs to be quoted as the characters used in it
clash with shell grammar tokens.
The -X option causes harden to always search for and harden an external
command, even if a built-in command by that name exists.
The -E option causes the hardening function to consider it a fatal error
if the hardened command writes anything to the standard error stream. This
option allows hardening commands (such asbc)
where you can’t rely on the exit status to detect an error. The text written
to standard error is passed on as part of the error message printed bydie. Note that:
-E cannotharden -E cd renders cd ineffective).-E does not disable exit status checks; by default, any exit status greater-e
'>125', limiting the exit status check to reserved values indicating errorsThe -p option causes harden to search for commands using the
system default path (as obtained with getconf PATH) as opposed to the
current $PATH. This ensures that you’re using a known-good external
command that came with your operating system. By default, the system-default
PATH search only applies to the command itself, and not to any commands that
the command may search for in turn. But if the -p option is specified at
least twice, the command is run in a subshell with PATH exported as the
default path, which is equivalent to adding a PATH=$DEFPATH assignment
argument (see below).
Examples:
harden make # simple check for status > 0harden -f tar '/usr/local/bin/gnutar' # id.; be sure to use this 'tar' versionharden -e '> 1' grep # for grep, status > 1 means errorharden -e '==1 || >2' gzip # 1 and >2 are errors, but 2 isn't (see manual)
As far as the shell is concerned, hardened commands are shell functions and
not external or builtin commands. This essentially changes one behaviour of
the shell: variable assignments preceding the command will not be local to
the command as usual, but will persist after the command completes.
(POSIX technically makes that behaviour
optional
but all current shells behave the same in POSIX mode.)
For example, this means that something like
harden -e '>1' grep# [...]LC_ALL=C grep regex some_ascii_file.txt
should never be done, because the meant-to-be-temporary LC_ALL locale
assignment will persist and is likely to cause problems further on.
To solve this problem, harden supports adding these assignments as
part of the hardening command, so instead of the above you do:
harden -e '>1' LC_ALL=C grep# [...]grep regex some_ascii_file.txt
With the -u option, harden also supports unsetting variables for the
duration of a command, e.g.:
harden -e '>1' -u LC_ALL grep
The -u option may be specified multiple times.
It causes the hardened command to be invoked from a
subshell
with the specified variables unset.
If you’re piping a command’s output into another command that may close
the pipe before the first command is finished, you can use the -P option
to allow for this:
harden -e '==1 || >2' -P gzip # also tolerate gzip being killed by SIGPIPEgzip -dc file.txt.gz | head -n 10 # show first 10 lines of decompressed file
head will close the pipe of gzip input after ten lines; the operating
system kernel then kills gzip with the PIPE signal before it’s finished,
causing a particular exit status that is greater than 128. This exit status
would normally make harden kill your entire program, which in the example
above is clearly not the desired behaviour. If the exit status caused by a
broken pipe were known, you could specifically allow for that exit status in
the status expression. The trouble is that this exit status varies depending
on the shell and the operating system. The -p option was made to solve
this problem: it automatically detects and whitelists the correct exit
status corresponding to SIGPIPE termination on the current system.
Tolerating SIGPIPE is an option and not the default, because in many
contexts it may be entirely unexpected and a symptom of a severe error if a
command is killed by a broken pipe. It is up to the programmer to decide
which commands should expect SIGPIPE and which shouldn’t.
Tip: It could happen that the same command should expect SIGPIPE in one
context but not another. You can create two hardened versions of the same
command, one that tolerates SIGPIPE and one that doesn’t. For example:
harden -f hardGrep -e '>1' grep # hardGrep does not tolerate being abortedharden -f pipeGrep -e '>1' -P grep # pipeGrep for use in pipes that may break
Note: If SIGPIPE was set to ignore by the process invoking the current
shell, the -p option has no effect, because no process or subprocess of
the current shell can ever be killed by SIGPIPE. However, this may cause
various other problems and you may want to refuse to let your program run
under that condition.thisshellhas WRN_NOSIGPIPE can help
you easily detect that condition so your program can make a decision. See
the WRN_NOSIGPIPE description for more information.
The -t option will trace command output. Each execution of a command
hardened with -t causes the command line to be output to standard
error, in the following format:
[functionname]> commandline
where functionname is the name of the shell function used to harden the
command and commandline is the actual command executed. Thecommandline is properly shell-quoted in a format suitable for re-entry
into the shell; however, command lines longer than 512 bytes will be
truncated and the unquoted string (TRUNCATED) will be appended to the
trace. If standard error is on a terminal that supports ANSI colours,
the tracing output will be colourised.
The -t option was added to harden because the commands that you harden
are often the same ones you would be particularly interested in tracing. The
advantage of using harden -t over the shell’s builtin tracing facility
(set -x or set -o xtrace) is that the output is a lot less noisy,
especially when using a shell library such as modernish.
Note: Internally, -t uses the shell file descriptor 9, redirecting it to
standard error (using exec 9>&2). This allows tracing to continue to work
normally even for commands that redirect standard error to a file (which is
another enhancement over set -x on most shells). However, this does meanharden -t conflicts with any other use of the file descriptor 9 in your
shell program.
If file descriptor 9 is already open before harden is called, harden
does not attempt to override this. This means tracing may be redirected
elsewhere by doing something like exec 9>trace.out before callingharden. (Note that redirecting FD 9 on the harden command itself will
not work as it won’t survive the run of the command.)
Sometimes you just want to trace the execution of some specific commands as
in harden -t (see above) without actually hardening them against command
errors; you might prefer to do your own error handling. trace makes this
easy. It is modernish’s replacement or complement for set -x a.k.a. set
-o xtrace.
Unlike harden -t, it can also trace shell functions.
Usage 1: trace [ -f funcname ] [ -[cSpXE] ]
[ var=value … ] [ -u var … ] command_name_or_path
[ command_argument … ]
For non-function commands, trace acts as a shortcut forharden -t -P -e '>125 && !=255' command_name_or_path.
Any further options and arguments are passed on to harden as given. The
result is that the indicated command is automatically traced upon execution.
A bonus is that you still get minimal hardening against fatal system errors.
Errors in the traced command itself are ignored, but your program is
immediately halted with an informative error message if the traced command:
SIGPIPE (exit status > 128, exceptSIGPIPE, and except 255 which isssh and rsync, to return an error).Note: The caveat for command-local variable assignments for harden also
applies to trace. See
Important note on variable assignments
above.
Usage 2: [ #! ] trace -f funcname
If no further arguments are given, trace -f will trace the shell
function funcname without applying further hardening (except against
nonexistence). trace -f can be used to trace the execution of modernish
library functions as well as your own script’s functions. The trace output
for shell functions shows an extra () following the function name.
Internally, this involves setting an alias under the function’s name, so
the limitations of the shell’s alias expansion mechanism apply: only
function calls that the shell had not yet parsed before calling trace -f
will be traced. So you should use trace -f at the beginning of your
script, before defining your own functions. To facilitate this, trace -f
does not check that the function funcname exists while setting up
tracing, but only when attempting to execute the traced function.
In portable-form
modernish scripts, trace -f should be used as a hashbang command to be
compatible with alias expansion on all shells. Only the trace -f form
may be used that way. For example:
#! /usr/bin/env modernish#! use safe -k#! use sys/cmd/harden#! trace -f push#! trace -f pop...your program begins here...
use sys/cmd/maprmapr (map records) is an alternative to xargs that shares features with themapfile command in bash 4.x. It is fully integrated into your script’s main
shell environment, so it can call your shell functions as well as builtin and
external utilities.
It depends on, and auto-loads, the sys/cmd/procsubst module.
Usage: mapr [ -d delimiter | -P ] [ -s count ] [ -n number ]
[ -m length ] [ -c quantum ] callback
mapr reads delimited records from the standard input, invoking the specified
callback command once or repeatedly as needed, with batches of input records
as arguments. The callback may consist of multiple arguments. By default, an
input record is one line of text.
Options:
-d delimiter: Use the single character delimiter to delimit input records,NUL (0) character and multi-byte-P: Paragraph mode. Input records are delimited by sequences consisting of-d.-s count: Skip and discard the first count records read.-n number: Stop processing after passing a total of number records to-n is not supplied or number is 0, all-s.-m length: Set the maximum argument length in bytes of each callback-c quantum: Pass at most quantum arguments at a time to each call to-c is not supplied or if quantum is 0, the number of-m; whichever limit isArguments:
mapr. If the callback command exits or returns with statusSIGPIPE signal, mapr will not process anymapr exits with the status of the last-executedmapfilemapr was inspired by the bash 4.x builtin command mapfile a.k.a.readarray, and uses similar options, but there are important differences.
mapr passes all the records as arguments to the callback command.mapr does not support assigning records directly to an array. Instead,-C option,xargs).-t option to remove it).mapr supports paragraph mode.xargsmapr shares important characteristics withxargs
while avoiding its myriad pitfalls.
mapr is fully integrated into themapr is line-oriented by default, so it is safe to use for inputmapr does not parse or modify the input arguments in any way, e.g. itxargs does.mapr supports paragraph mode.use sys/cmd/procsubstThis module provides a portable
process substitution
construct, the advantage being that this is not limited to bash, ksh or zsh
but works on all POSIX shells capable of running modernish. It is not
possible for modernish to introduce the original ksh syntax into other
shells. Instead, this module provides a % command for use within a$(command substitution).
The % command takes one simple command as its arguments, executes it in
the background, and writes a file name from which to read its output. So
if % is used within a command substitution as intended, that file name
is passed on to the invoking command as an argument.
The % command supports one option, -o. If that option is given, then it is
expected that, instead of reading input, the invoking command writes output to
the file name passed on to it, so that the command invoked by % -o can read
that data from its standard input.
| ksh/bash/zsh | modernish |
|---|---|
diff -u <(ls) <(ls -a) | diff -u $(% ls) $(% ls -a) |
IFS=’ ‘ read -r user vsz args < <(ps -o ‘user= vsz= args=’ -p $$) | IFS=’ ‘ read -r user vsz args < $(% ps -o ‘user= vsz= args=’ -p $$) |
{ some commands; } > >(tee stdout.log) 2> >(tee stderr.log)(both tee commands write terminal output to standard output) | { some commands; } > $(% -o tee stdout.log) 2> $(% -o tee stderr.log)(both tee commands write terminal output to standard error) |
Unlike the bash/ksh/zsh version, modernish process substitution only works
with simple commands. This includes shell function calls, but not aliases or
anything involving shell grammar or reserved words (such as redirections,
pipelines, loops, etc.). To use such complex commands, enclose them in a shell
function and call that function from the process substitution.
Also note that anything that a command invoked by the % -o writes to its
standard output is redirected to standard error. The main shell environment’s
standard output is not available because the command substitution subsumes it.
use sys/cmd/sourceThe source command sources a dot script like the . command, but
additionally supports passing arguments to sourced scripts like you would
pass them to a function. It mostly mimics the behaviour of the source
command built in to bash and zsh.
If a filename without a directory path is given, then, unlike the .
command, source looks for the dot script in the current directory by
default, as well as searching $PATH.
It is a fatal error to attempt to source a directory, a file with no read
permission, or a nonexistent file.
use sys/dirFunctions for working with directories.
use sys/dir/countfilescountfiles: Count the files in a directory using nothing but shell
functionality, so without external commands. (It’s amazing how many pitfalls
this has, so a library function is needed to do it robustly.)
Usage: countfiles [ -s ] directory [ globpattern … ]
Count the number of files in a directory, storing the number in REPLY
and (unless -s is given) printing it to standard output.
If any globpatterns are given, only count the files matching them.
use sys/dir/mkcdThe mkcd function makes one or more directories, then, upon success,
change into the last-mentioned one. mkcd inherits mkdir‘s usage, so
options depend on your system’s mkdir; only the
POSIX options
are guaranteed.
When mkcd is run from a script, it uses cd -P to change the working
directory, resolving any symlinks in the present working directory path.
use sys/termUtilities for working with the terminal.
use sys/term/putrThis module provides commands to efficiently output a string repeatedly.
Usage:
putr [ number | - ] stringputrln [ number | - ] stringOutput the string number times. When using putrln, add a newline at
the end.
If a - is given instead of a number, then the total length of the output
is the line length of the terminal divided by the length of the string,
rounded down.
Note that, unlike with put and putln, only a single string
argument is accepted.
Example: putrln - '=' prints a full terminal line of equals signs.
use sys/term/readkeyreadkey: read a single character from the keyboard without echoing back to
the terminal. Buffering is done so that multiple waiting characters are read
one at a time.
Usage: readkey [ -E ERE ] [ -t timeout ] [ -r ] [ varname ]
-E: Only accept characters that match the extended regular expression
ERE (the type of RE used by grep -E/egrep). readkey will silently
ignore input not matching the ERE and wait for input matching it.
-t: Specify a timeout in seconds (one significant digit after the
decimal point). After the timeout expires, no character is read andreadkey returns status 1.
-r: Raw mode. Disables INTR (Ctrl+C), QUIT, and SUSP (Ctrl+Z) processing
as well as translation of carriage return (13) to linefeed (10).
The character read is stored into the variable referenced by varname,
which defaults to REPLY if not specified.
This module depends on the trap stack to save and restore the terminal state
if the program is stopped while reading a key, so it will automaticallyuse var/stack/trap on initialisation.
This appendix lists all the shell
capabilities,
quirks, and
bugs
that modernish can detect in the current shell, so that modernish scripts
can easily query the results of these tests and decide what to do. Certain
problematic system conditions
are also detected this way and listed here.
The all-caps IDs below are all usable with thethisshellhas
function. This makes it easy for a cross-platform modernish script to
be aware of relevant conditions and decide what to do.
Each detection test has its own little test script in thelib/modernish/cap directory. These tests are executed on demand, the
first time the capability or bug in question is queried usingthisshellhas. See README.md in that directory for further information.
The test scripts also document themselves in the comments.
Modernish currently identifies and supports the following non-standard
shell capabilities:
ADDASSIGN: Add a string to a variable using additive assignment,+=stringANONFUNC: zsh anonymous functions (basically the native zsh equivalentARITHCMD: standalone arithmetic evaluation using a command like((expression)).ARITHFOR: ksh93/C-style arithmetic for loops of the formfor ((exp1;exp2;exp3)) docommands; done.ARITHPP: support for the ++ and -- unary operators in shell arithmetic.CESCQUOT: Quoting with C-style escapes, like $'\n' for newline.DBLBRACKET: The ksh88-style [[ double-bracket command ]],DBLBRACKETERE: DBLBRACKET plus the =~ binary operator to match aDBLBRACKETV: DBLBRACKET plus the -v unary operator to test if a[[ -v 1 ]]) does not work on bash or ksh93; check $# instead.)DOTARG: Dot scripts support arguments.HERESTR: Here-strings, an abbreviated kind of here-document.KSH88FUNC: define ksh88-style shell functions with the function keyword,typeset builtin.KSH93FUNC: the same, but with static scoping for local variables. (ksh93 only)KSHARRAY: ksh93-style arrays. Supported on bash, zsh (under emulate sh),LEPIPEMAIN: execute last element of a pipe in the main shell, so that| read somevariable work. (zsh, AT&T ksh,LINENO: the $LINENO variable contains the current shell script lineLOCALVARS: the local command creates dynamically scoped local variablesNONFORKSUBSH: as a performance optimisation,ulimit -t unlimited forcesBUG_FNSUBSH.)PRINTFV: The shell’s printf builtin has the -v option to print to a variable,PROCREDIR: the shell natively supports <(process redirection),< <(command).PROCSUBST where this <(syntax)PROCSUBST: the shell natively supports <(process substitution),< <(command).PROCREDIR where the same <(syntax)PSREPLACE: Search and replace strings in variables using special parameterRANDOM: the $RANDOM pseudorandom generator.RANDOM either working properly or being unset.RANDOM is setting a known seed to getawk‘s srand(yourseed) and int(rand()*32768).)ROFUNC: Set functions to read-only with readonly -f. (bash, yash)TESTERE: The regular test/[ builtin command supports the =~ binaryTESTO: The test/[ builtin supports the -o unary operator to check ifTRAPPRSUBSH: The ability to obtain a list of the current shell’s nativevar=$(trap),var/stack/trap module transparently reimplements thisTRAPZERR: This feature ID is detected if the ERR trap is an alias forZERR trap. According to the zsh manual, this is the case for zsh onSIGERR signal. (TheVARPREFIX: Expansions of type ${!prefix@} and ${!prefix*} yieldprefix in the same way and with$@ and $*, respectively.BUG_VARPREFIX.Modernish currently identifies and supports the following shell quirks:
QRK_32BIT: mksh: the shell only has 32-bit arithmetic. Since every modernQRK_ANDORBG: On zsh, the & operator takes the last simple commanda && b || c & is interpreted asa && b || { c & } and not { a && b || c; } &.QRK_ARITHEMPT: In yash, with POSIX mode turned off, a set but emptyfoo=''; echo $((foo)) outputs an empty line.QRK_ARITHWHSP: In yashisint function (to determine if a string is a validQRK_ARITHWHSP compatible, tolerating onlyQRK_BCDANGER: break and continue can affect non-enclosing loops,QRK_EMPTPPFLD: Unquoted $@ and $* do not discard empty fields.$@ and unquoted $* that empty positional parametersQRK_EMPTPPWRD: POSIX says"$@" generates zero fields but empty '' or "" or"$emptyvariable" generates one empty field. But it leaves unspecified"$@$emptyvariable" generates zero fields or oneQRK_EVALNOOPT: eval does not parse options, not even --, which makes iteval -- "$command" whereas on other shells this is necessary if the command-, or the command would be interpreted as an option to eval.QRK_EXECFNBI: In pdksh and zsh, exec looks up shell functions andexit. Thisexec is supposed to launch aexec is always external to the shell. However, since theQRK_FNRDREXIT: On FreeBSD sh and NetBSD sh, an error in a redirectionharden.QRK_GLOBDOTS: Pathname expansion of .* matches the pseudonames . and.. so that, e.g., cp -pr .* backup/ cannot be used to copy all yourQRK_HDPARQUOT: Double quotes within certain parameter substitutions invar is set, ${var+"x"} in a here-document yields "x", not x.${var#"$x"} and ${var%"$x"}; those are defined by POSIXQRK_IFSFINAL: in field splitting, a final non-whitespace IFS delimiterQRK_LOCALINH: On a shell with LOCALVARS, local variables, when declaredQRK_LOCALSET: On a shell with LOCALVARS, local variables are immediately setQRK_LOCALSET2: Like QRK_LOCALSET, but only if the variable by theQRK_LOCALUNS: On a shell with LOCALVARS, local variables lose their localunset will not necessarily unset the variable! (yash, pdksh/mksh. Note:typeset, to which modernish aliases localQRK_LOCALUNS2: This is a more treacherous version of QRK_LOCALUNS thatunset command works as expected when used on a localQRK_LOCALUNS2 is a special case of QRK_LOCALUNS, modernishshopt -s localvar_unset.QRK_OPTABBR: Long-form shell option names can be abbreviated down to aQRK_OPTCASE: Long-form shell option names are case-insensitive. (yash, zsh)QRK_OPTDASH: Long-form shell option names ignore the -. (ksh93, yash)QRK_OPTNOPRFX: Long-form shell option names use a dynamic no prefix forglob is the oppositenoglob, and nonotify is the opposite of notify. (ksh93, yash, zsh)QRK_OPTULINE: Long-form shell option names ignore the _. (yash, zsh)QRK_PPIPEMAIN: On zsh \<= 5.5.1, in all elements of a pipeline, parameterv, in the pipeline cmd1 ${v:=foo} | cmd2, the assignment tov survives, though cmd1 itself is executed in a subshell.QRK_SPCBIXP: Variable assignments directly precedingQRK_UNSETF: If ‘unset’ is invoked without any option flag (-v or -f), andModernish currently identifies and supports the following shell bugs:
BUG_ALIASCSHD: A spurious syntax error occurs if a here-document} occurs because the aliasvar/local andvar/loopBUG_ALIASPOSX: Running any command “foo” in POSIX mode likePOSIXLY_CORRECT=y foo will globally disable alias expansion on aPROCSUBST) isn’t available.BUG_ARITHINIT: Using unset or empty variables (dash <= 0.5.9.1 on macOS)BUG_ARITHLNNO: The shell supports $LINENO, but the variable is$(( LINENO > 0 )).set -u and default to zero otherwise.$(( $LINENO > 0 )). (FreeBSD sh)BUG_ARITHNAN: The case-insensitive special floating point constantsInf and NaN are recognised in arithmetic evaluation, overriding anyInf, NaN, INF, nan, etc. (AT&T ksh93;BUG_ARITHSPLT: Unquoted $((arithmetic expressions)) are notBUG_ASGNCC01: if IFS contains a $CC01 (^A) character, unquoted expansions inBUG_ASGNLOCAL: If you have a function-local variable (see LOCALVARS)BUG_BRACQUOT: shell quoting within bracket patterns has no effect (zsh < 5.3;- retains it special meaning of ‘character! (and, on some shells, ^) retains the meaning ofBUG_CASEEMPT: An empty case list on a single line, as in case x in esac,BUG_CASELIT: If a case pattern doesn’t match as a pattern, it’s triedmatch works around this.BUG_CASEPAREN: case patterns without an opening parenthesis$( ).BUG_CASESTAT: The case conditional construct prematurely clobbers the$?. (found in zsh \< 5.3, Busybox ash \<= 1.25.0, dash \<BUG_CDNOLOGIC: The cd built-in command lacks the POSIX-specified -L-P-L optionchdir ineffective. (NetBSD sh)BUG_CDPCANON: cd -P (and hence also modernishchdir) does not correctly canonicalise/normalise a$PWD. (zsh \<= 5.7.1)BUG_CMDEXEC: using command exec (to open a file descriptor, usingcommand to avoid exiting the shell on failure) within a function causesBUG_CMDEXPAN: if the command command results from an expansion, it actscommand -v, showing the path of the command instead of executing it.v=command; "$v" ls or set -- command ls; "$@" don’t work.BUG_CMDOPTEXP: the command builtin does not recognise options if they-pdefaultpath and then do command $defaultpath
someCommand. (found in zsh \< 5.3)BUG_CMDPV: command -pv does not find builtins ({pd,m}ksh), does not-pBUG_CMDSETPP: using command set -- has no effect; it does not set theset without command. (mksh \<= R57)BUG_CMDSPASGN: preceding acommand does not stop preceding invocation-local variableBUG_CMDSPEXIT: preceding aeval, exec, return or exit)command does not always stopBUG_CSNHDBKSL: Backslashes within non-expanding here-documents withinBUG_CSUBBTQUOT: A spurious syntax erorr is thrown when using doubleBUG_CSUBLNCONT: Backslash line continuation is not processed correctly$(command substitutions).BUG_CSUBRMLF: A bug affecting the stripping of final linefeeds fromBUG_CSUBSTDO: If standard output (file descriptor 1) is closed beforeecho orputln will not work within the command substitution, acting as if standardcap/BUG_CSUBSTDO.t.BUG_DEVTTY: the shell can’t redirect output to /dev/tty ifset -C/set -o noclobber (part of safe mode)>| /dev/tty instead of > /dev/tty.BUG_DOLRCSUB: parsing problem where, inside a command substitution of$(...), the sequence $$'...' is treated as $'...' (i.e. as$$"..." as $"..." (bash-specific translatableBUG_DQGLOB: globbing is not properly deactivated within* or ? immediatelycase. Found in: dash.match.)BUG_EXPORTUNS: Setting the export flag on an otherwise unset variableBUG_FNSUBSH: Function definitions within subshells (including commandunset -f is also silentlyNONFORKSUBSH.BUG_FORLOCAL: a for loop in a function makes the iteration variableBUG_GETOPTSMA: The getopts builtin leaves a : instead of a ? in:. (zsh)BUG_HDOCBKSL: Line continuation using backslashes in expandingBUG_HDOCMASK: Here-documents (and here-strings, see HERESTRING) useumask setting disallows theumask when using here-documents.BUG_IFSCC01PP: If IFS contains a $CC01 (^A) control character, the"$@" (even quoted) is gravely corrupted. Since many modernishBUG_IFSGLOBC: In glob pattern matching (such as in case and [[), if aIFS, it is matched literally instead of as a*, ?, [ and ].case for argument validation andIFS contains any of these three characters!BUG_IFSGLOBP: In pathname expansion (filename globbing), if aIFS, it is matched literally instead of as a*, ?, [ and ].BUG_IFSGLOBS: in glob pattern matching (as in case or parameter# and %), if IFS starts with ? or * and the"$*" parameter expansion inserts any IFS separator characters, thoseBUG_IFSISSET: AT&T ksh93 (2011/2012 versions): ${IFS+s} always yields sIFS is unset. This applies to IFS only.BUG_ISSETLOOP: AT&T ksh93: Expansions like ${var+set}for, while oruntil loop; the expansions don’t change along with the state of theBUG_KBGPID: AT&T ksh93: If a single command ending in & (i.e. a background{ braces; } block with an I/O redirection, the $!BUG_KUNSETIFS: AT&T ksh93: Unsetting IFS fails to activate default fieldIFS is set and emptyNONFORKSUBSH).BUG_LNNONEG: $LINENO becomes wildly inaccurate, even negative, whenBUG_LOOPRET1: If a return command is given with a status argument withinwhile or until loop (i.e., betweenwhile/until and do), the status argument is ignored and the functionBUG_LOOPRET2: If a return command is given without a status argumentwhile or until loop (i.e.,while/until and do), the exit status passed down from theBUG_LOOPRET3: If a return command is given within the set of conditionalwhile or until loop (i.e., between while/until anddo), and the return status (either the status argument to return or thereturn without awhile) or true (for until), and the whole construct isBUG_MULTIBIFS: We’re on a UTF-8 locale and the shell supports UTF-8WRN_MULTIBYTE) – however, usingIFS field delimiters still doesn’t work. For"$*" joins positional parameters on the first byte of IFSBUG_NOCHCLASS: POSIX-mandated character [:classes:] within bracket[expressions] are not supported in glob patterns. (mksh)BUG_NOEXPRO: Cannot export read-only variables. (zsh <= 5.7.1 in sh mode)BUG_OPTNOLOG: on dash, setting -o nolog causes $- to wreak havoc:$- silently aborts parsing of an entire argument,"one,$-,two" yields "one,". (Same applies to -o debug.)BUG_PP_01: POSIX says"$@" generates zero fields but empty '' or "" or"$emptyvariable" generates one empty field. This means concatenating"$@" with one or more other, separately quoted, empty strings (like"$@""$emptyvariable") should still produce one empty field. But onBUG_PP_02: Like BUG_PP_01, but with unquoted $@ and only"$emptyvariable"$@, not $@"$emptyvariable".BUG_PP_03: When IFS is unset or empty (zsh 5.3.x) or empty (mksh \<= R50),var=$* only assigns the first field, failing to join andvar="$*"var=$@, etc. undefined, so we don’t test for those.)BUG_PP_03A: When IFS is unset, assignments like var=$*BUG_PP_03B: When IFS is unset, assignments like var=${var+$*},BUG_PP_03C: When IFS is unset, assigning var=${var-$*} only assignsvar=${var-"$*"}BUG_PP_04A: Like BUG_PP_03A, but for conditional assignments within: ${var=$*} or : ${var:=$*}.$* within the expansion or the expansionBUG_PP_04E: When assigning the positional parameters ($) to a variable, the fields are always joined and separated by spaces,
except ifIFS` is set and empty. Workaround as in BUG_PP_04A.BUG_PP_04_S: When IFS is null (empty), the result of a substitution${var=$*} is incorrectly field-split on spaces.BUG_PP_05: POSIX says$@ and $* generate zero fields, but with null IFS, empty$@ and $* yield one empty field. Found on: dash 0.5.9BUG_PP_06A: POSIX says$@ and $* initially generate as many fields as there are$@ or $* is unquoted) each field isIFS. With this bug, the latter step is notIFS is unset (i.e. default split). Found on: zsh \< 5.4BUG_PP_07: unquoted $* and $@ (including in substitutions like${1+$@} or ${var-$*}) do not perform default field splitting ifIFS is unset. Found on: zsh (up to 5.3.1) in sh modeBUG_PP_07A: When IFS is unset, unquoted $* undergoes word splittingIFS=' ', and not the expected IFS=" ${CCt}${CCn}".BUG_PP_08: When IFS is empty, unquoted $@ and $* do not generateBUG_PP_08B: When IFS is empty, unquoted $* within a substitution (e.g.${1+$*} or ${var-$*}) does not generate one field for each positionalBUG_PP_09: When IFS is non-empty but does not contain a space,$* within a substitution (e.g. ${1+$*} or ${var-$*}) doesIFS does not contain a space).BUG_PP_10: When IFS is null (empty), assigning var=$* removes any$CC01 (^A) and $CC7F (DEL) characters. (bash 3, 4)BUG_PP_10A: When IFS is non-empty, assigning var=$* prefixes each$CC01 (^A) and $CC7F (DEL) character with a $CC01 character. (bash 4.4)BUG_PP_1ARG: When IFS is empty on bash <= 4.3 (i.e. field${1+"$@"} or "${1+$@}" is counted as a single"${1+foobar $@}".BUG_PP_MDIGIT: Multiple-digit positional parameters don’t require expansion$10 = ${10} (dash; Busybox ash). This is classed as a bugBUG_PP_MDLEN: For ${#x} expansions where x >= 10, only the first digit of${#10}, ${#12},${#123} are all parsed as if they are ${#1}. Then, string parsing isBUG_PSUBASNCC: in an assignment parameter substitution of the form${foo=value}, if the characters $CC01 (^A) or $CC7F (DEL) are in the$CC01 is stripped. This bug is independent of the state of IFS, except ifIFS is null, the assignment in ${foo=$*} (unquoted) is buggy too: it$CC01 from the assigned value. (Found on bash 4.2, 4.3, 4.4)BUG_PSUBBKSL1: A backslash-escaped } character within a quoted parameterBUG_PSUBEMIFS: if IFS is empty (no split, as in safe mode), then if a${foo-$*}, ${foo+$*}, ${foo:-$*} or${foo:+$*} occurs in a command argument, the characters $CC01 (^A) or$CC7F (DEL) are stripped from the expanded argument. (Found on: bash 4.4)BUG_PSUBEMPT: Expansions of the form ${V-} and ${V:-} are not${V+$V} and${V:+$V} work as expected. (Found on FreeBSD 10.3 sh)BUG_PSUBIFSNW: When field-splitting unquoted parameter substitutions like${var#foo}, ${var##foo}, ${var%foo} or ${var%%foo} on non-whitespaceIFS, if there is an initial empty field, a spurious extra initial emptyBUG_PSUBNEWLN: Due to a bug in the parser, parameter substitutions$CCn.BUG_PSUBSQUOT: in pattern matching parameter substitutions${param#pattern}, ${param%pattern}, ${param##pattern} and${param%%pattern}), if the whole parameter substitution is quoted withx=foobar; echo "${x#'foo'}" should yield barfoobar. (dash \<= 0.5.9.1; Busybox 1.27 ash)BUG_PSUBSQHD: Like BUG_PSUBSQUOT, but included a here-document instead ofBUG_PUTIOERR: Shell builtins that output strings (echo, printf, ksh/zshprint), and thus also modernish put and putln, do not check for I/OSIGPIPE is ignored.BUG_READWHSP: If there is more than one field to read, read does notIFS whitespace. (dash 0.5.7, 0.5.8)BUG_REDIRIO: the I/O redirection operator <> (open a file descriptor1<>) instead of defaulting to opening standard input (0<>) asBUG_REDIRPOS: Buggy behaviour occurs if a redirection is positionedBUG_SCLOSEDFD: bash \< 5.0 and dash fail to establish a block-local scope} 8<&- or done 7>&-). If that FDexec it. So with this bug, it is not straightforward to make adone 7>/dev/null 7>&- will establishdo…done block while stillBUG_SETOUTVAR: The set builtin (with no arguments) only prints nativeBUG_SHIFTERR0: The shift builtin silently returns a successful exitBUG_SPCBILOC: Variable assignments precedingBUG_TESTERR1A: test/[ exits with a non-error false statusBUG_TESTILNUM: On dash (up to 0.5.8), giving an illegal number to test -t[ -t causes some kind of corruption so the next test/[ invocationBUG_TESTONEG: The test/[ builtin supports a -o unary operator tono prefix on shell[ -o noclobber ] gives a false positive.TESTO feature test implicitly checksBUG_TRAPEMPT: The trap builtin does not quote empty traps in itstrap '' INT; trap outputs “trap -- INT“ instead of “trap -- '' INT“.BUG_TRAPEXIT: the shell’s trap builtin does not know the EXIT trap byBUG_TRAPFNEXI: When a function issues a signal whose trap exits theBUG_TRAPRETIR: Using return within eval triggers infinite recursion iffunctrace shell option are active. This bug infunctrace shell option if aRETURN trap is set or pushed and this bug is detected. (bash 4.3, 4.4)BUG_TRAPSUB0: Subshells in traps fail to pass down a nonzero exit status ofBUG_TRAPUNSRE: When a trap unsets itself and then resends its own signal,BUG_UNSETUNXP: If an unset variable is given the export flag using theexport command, a subsequent unset command does not remove that exportBUG_VARPREFIX: On a shell with the VARPREFIX feature, expansions of type${!prefix@} and ${!prefix*} do not find the variable nameBUG_ZSHNAMES: A series of lowerase names, normally okay for script useemulate sh post invocationaliases argvbuiltins cdpath commands dirstack dis_aliases dis_builtinsdis_functions dis_functions_source dis_galiases dis_patcharsdis_reswords dis_saliases fignore fpath funcfiletracefuncsourcetrace funcstack functions functions_source functracegaliases histchars history historywords jobdirs jobstatesjobtexts keymaps mailpath manpath module_path modules nameddirsoptions parameters patchars path pipestatus prompt psvarreswords saliases signals status termcap terminfo userdirsusergroups watch widgets zsh_eval_context zsh_scheduled_eventsBUG_ZSHNAMES2: Two lowercase variable names histchars and signals,sh--emulate sh option at startup).BUG_ZSHNAMES isWarning IDs do not identify any characteristic of the shell, but instead
warn about a potentially problematic system condition that was detected at
initialisation time.
WRN_EREMBYTE: The current system locale setting supports Unicode UTF-8str ematch[:classes:]WRN_MULTIBYTE: The current system locale setting supports Unicode UTF-8if thisshellhas WRN_MULTIBYTE and resortWRN_NOSIGPIPE: Modernish has detected that the process that launchedSIGPIPE to ignore, an irreversible condition$SIGPIPESTATUSSIGPIPE. The-P option to hardenfoo is implemented, it is now possiblefoo | head -n 10 never ends; if foo doesn’tSIGPIPE as head terminates.if thisshellhas
WRN_NOSIGPIPE and either employ workarounds or refuse to run if so.Modernish comes with a suite of regression tests to detect bugs in modernish
itself, which can be run using modernish --test after installation. By
default, it will run all the tests verbosely but without tracing the command
execution. The install.sh installer will run modernish --test -eqq on the
selected shell before installation.
A few options are available to specify after --test:
-h: show help.-e: disable or reduce expensive (i.e. slow or memory-hogging) tests.-q: quieter operation; report expected fails [known shell bugs]-q again for-s: entirely silent operation.-t: run only specific test sets or tests. Test sets are those listedmodernish --test. This option requires:num1,num2,…/testset2:num1,num2,…/…modernish --test -t match:2,4,7/arith/shellquote:1 runsmatch set, the entire arith set, and onlyshellquote set.-x: trace each test using the shell’s xtrace facility. Each trace is-x again to keep expected fails as well, and again to-E: don’t run any tests, but output a command to open the tests that wouldVISUAL or EDITORvi as a default. This option should be-t to specify tests. All other options are ignored.-F: takes an argument with the name or path to a find utility toLOOP find.These short options can be combined so, for example,--test -qxx is the same as --test -q -x -x.
Note the difference between these regression tests and the cap tests listed in
Appendix A. The latter are
tests for whatever shell is executing modernish: they detect capabilities
(features, quirks, bugs) of the current shell. They are meant to be run viathisshellhas and are designed to
be taken advantage of in scripts. On the other hand, these tests run bymodernish --test are regression tests for modernish itself. It does not
make sense to use these in a script.
New/unknown shell bugs can still cause modernish regression tests to fail,
of course. That’s why some of the regression tests also check for
consistency with the results of the capability detection tests: if there is a
shell bug in a widespread release version that modernish doesn’t know about
yet, this in turn is considered to be a bug in modernish, because one of its
goals is to know about all the shell bugs in all released shell versions
currently seeing significant use.
The testshells.sh program in share/doc/modernish/examples can be used to
run the regression test suite on all the shells installed on your system.
You could put it as testshells in some convenient location in your$PATH, and then simply run:
testshells modernish --test
(adding any further options you like – for instance, you might like to add-q to avoid very long terminal output). On first run, testshells will
generate a list of shells it can find on your system and it will give you a
chance to edit it before proceeding.
modernish, like most shells, fully supports two system locales: POSIX
(a.k.a. C, a.k.a. ASCII) and Unicode’s UTF-8. It will work in other locales,
but things like converting to upper/lower case, and matching single
characters in patterns, are not guaranteed.
Caveat: some shells or operating systems have bugs that prevent (or lack
features required for) full locale support. If portability is a concern,
check for thisshellhas WRN_MULTIBYTE or thisshellhas BUG_NOCHCLASS
where needed. See Appendix A.
Scripts/programs should not change the locale (LC_* or LANG) after
initialising modernish. Doing this might break various functions, as
modernish sets specific versions depending on your OS, shell and locale.
(Temporarily changing the locale is fine as long as you don’t use
modernish features that depend on it – for example, setting a specific
locale just for an external command. However, if you use harden, see
the important note
in its documentation!)
Modernish builds on the
POSIX 2018 Edition
standard, so it should run on any sufficiently POSIX-compliant shell and
operating system. It uses both
bug/feature detection
and
regression testing
to determine whether it can run on any particular shell, so it does not
block or support particular shell versions as such. However, modernish has
been confirmed to run correctly on the following shells:
Currently known not to run modernish due to excessive bugs:
This appendix is specific to zsh.
While modernish duplicates some functionality already available natively
on zsh, it still has plenty to add. However, writing a normal
simple-form modernish script turnsemulate sh on for the entire script, so you lose important aspects
of the zsh language.
But there is another way – modernish functionality may be integrated
with native zsh scripts using ‘sticky emulation’, as follows:
emulate -R sh -c '. modernish'
This causes modernish functions to run in sh mode while your script will still
run in native zsh mode with all its advantages. The following notes apply:
${~var} and ${=var} flags that apply--split and --glob operators to constructs such asLOOP findfind remains necessary.LOCAL is not recommended.LOOP find orSee man zshbuiltins under emulate, option -c, for more information.
The modernish installer install.sh can bundle one or more scripts with a
stripped-down version of the modernish library. This allows the bundled scripts
to run with a known version of modernish, whether or not modernish is installed
on the user’s system. Like modernish itself, bundling is cross-platform and
portable (or as portable as your script is).
Bundled scripts are not modified. Instead, for each script, a wrapper script is
installed under the same name in the installation root directory. This wrapper
automatically looks for a suitable
POSIX-compliant shell that passes the modernish battery of fatal bug tests,
then sets up the environment to run the real script with modernish on that
shell. Your modernish script can be run through the supplied wrapper script
from any directory location on any POSIX-compliant operating system, as long as
all files remain in the same location relative to each other.
Bundling is always a non-interactive installer operation, with options
specified on the command line. The installer usage for bundling is as follows:
install.sh -B -D rootdir [ -d subdir ]
[ -s shell ] scriptfile [ scriptfile … ]
The -B option enables bundling mode. The option does not itself take an
option-argument. Instead, any number of scriptfiles to bundle can be given
as arguments following all other options. All scripts are bundled with a
single copy of modernish. The bundling operation does not deal with any
auxiliary files the scripts may require (other than modernish modules); any
such need to be added manually after bundling is complete.
The -D option specifies the path to the bundled installation’s root
directory, where wrapper scripts are installed. This option is mandatory.
If the directory doesn’t exist, it is created.
The -d option specifies the subdirectory of the -D root directory where the
bundled scripts and modernish are installed. It can contain slashes to install
the bundle at a deeper directory level. The default subdirectory is bndl.
The option-argument can be empty or /, in which case the bundle is installed
directly into the installation root directory.
The -s option specifies a preferred shell for the bundled scripts. A shell
name or a full path to a shell can be given. Wrapper scripts try the full path
first (if any), then try to find a shell with its basename, and then try to
find a shell with that basename minus any version number (e.g. bash instead
of bash-5.0 or ksh instead of ksh93). If all that doesn’t produce a shell
that passes fatal bugs tests, it continues with the normal shell search.
This means the script won’t fail to launch if the preferred shell can’t be
found. Instead, it is up to the script itself to refuse to run if required
shell-specific conditions are not met. Script should use thethisshellhas
function to check for any nonstandard
capabilities
required, or any
bugs
or
quirks
that the script is incompatible with (or indeed requires!).
Bundling is supported for both
portable-form
and
simple-form
modernish scripts. The installer automatically adapts the wrapper scripts to
the form used. For simple-form scripts, the directory containing the bundled
modernish core library (by default, .../bndl/bin/modernish) is prefixed to$PATH so that . modernish works. Since simple-form scripts are often more
shell-specific, you may want to specify a preferred shell with the -s option.
To save space, the bundled copy of the modernish library is reduced such that
all comments are stripped from the code,
interactive use
is not supported,
the regression test suite
is not included,thisshellhas
does not have the --cache and --show operators,
and thecap/*.t capability detection scripts
are “statically linked” (directly included) into bin/modernish instead of
shipped as separate files.
A README.modernish file is added with a short explanation, the licence,
and a link for people to get the complete version of modernish. Please do
not remove this when distributing bundled scripts.
EOF