Horizon chart tool for terminal.
Horizon charts provide a nice balance between information density and clarity in the case of multiple time series.
Interactive examples of horizon charts can be found on @d3/horizon-chart-ii">observable and, specifically for system monitoring, in Square’s cubism.
hcl (Horizon CLi) is a command-line tool for visualizing arbitrary data in horizon format.
You can think of it as a chart version of a less +F
for numeric data.
$ git clone https://github.com/okuvshynov/hcl.git
$ cd hcl
$ cargo build --release
$ cat tests/sine.csv | ./target/release/hcl # test on static data
Coming soon; Once CI is set up, they’ll be here;
hcl reads data in two formats:
1) [default] CSV-like, with comma-separated values. Each column in the file represents individual series, and each row in the file becomes a ‘column’ in the chart: example
2) title:value pairs; Empty line represents a separator between different ‘columns’ in the chart: example
Data is provided to hcl either by piping it to stdin, or by providing file name as a last argument.
$ cat tests/sine.csv | hcl
Equivalent to
$ hcl tests/sine.csv
If empty line is encountered, new set of titles is expected. hcl will attempt to ‘merge’ the data after that, filling missing values with NaN.
This can be useful in the cases like ‘show top N processes reading from HDD, with 1 second granularity’, when set of ‘top N’ might be different every time.
Check this simple example which simulates that.
There’s an option to use one of the columns as an ‘x’ axis. Most commonly that would be some form of
time/date, but it’s not required - it can be an arbitrary string. X is configured using -x
All other columns need to have floating-point numbers as their values. Missing values or the ones failed to parse will be represented as ‘.’ (NaN).
There’s no filtering/aggregation functionality. If there’s a need to do so, external tool (awk, xsv, sed, …) should be used before piping the input to hcl.
The scale is a piecewise linear map from an input domain to [-1;0] and [0;1] intervals. [-1;1] values will be mapped to Unicode block characters, full dark green representing ‘1’ and full dark red representing ‘-1’.
Scales are defined in a comma-separated list, for example
0..10,cpu:100,ram:32G,cycles:auto
There’re four scales here, one global [0..10], one with ‘cpu’ pattern, another with ‘ram’ pattern, and an autoscale for ‘cycles’ pattern.
Input domain for the scale is repesented by two intervals: [a;b] and [b;c].
[a;b] is mapped to [-1;0] and [b;c] is mapped to [0;1].
The format for the scale is ‘[a..[b..]]c’ OR ‘auto’.
Each number can have K/M/G/T suffix to allow 10^3, 10^6, 10^9, 10^12 scaling.
Examples:
Picking a scale for the series follow such procedure:
Complete example:
‘cpu:100,ram:16G,wow_change:95..100..105’ will break down into 3 scales.
Autoscaling is an attempt to provide reasonable default behavior. Autoscaling works on individual series level by default; ‘-s auto’ would force global autoscale.
After iterating over all values in a series, following logic is applied:
In cases when ‘auto’ is used explicitly, for example: ‘cpu:auto,ram:12G’, all series matching ‘cpu’ would use same scale, computed based on combined data.
For example, we have following input data (load.csv):
cpu#1,cpu#2,ram_free_gb
50,20,1
100,20,2,
50,20,3
If we run cat load.csv | hcl
, each series will compute autoscale individually, so:
If, instead, we do cat load.csv | hcl -s cpu:auto
, both cpu series will be used together, and [0;100] will be used for both.
In case of cpu load, which often is represented as percentage, it might be easier to just specify ‘cpu:100’.
For counters which might have no obviously good upper bound, auto could be benefitial.
For example:
Autoscale is recomputed on every data update, as new values might change the interval boundaries;
Autoscaling has its drawbacks, specifically old data might change its appearance as new data arrives, which could look confusing;
Thus, it’s often a better option to provide scale as an input. ‘-s core:100’ will be sufficient for the example above.
‘-s auto’ is a way to treat all series together, and pick a single scale for all of them.
In progress;
USAGE:
FLAGS:
OPTIONS:
ARGS:
Interactive commands resemble those of applications like ‘less’.
Vertical navigation:
Horizontal navigation:
Mouse:
Other:
cat tests/sine.csv | hcl
vmstat is a convenient tool to monitor current CPU/Memory/IO on Linux/BSD.
$ vmstat -n 1 | awk -W interactive -v OFS=',' '{if (NR>1) { $1=$1; print; }}' | hcl
atop can be very useful to look into historical data on a single host.
cpu usage per core:
$ atopsar -c -S -b 21:25 | awk '$1 ~ /[0-9][0-9]:[0-9][0-9]/ && $2 != "cpu" { if ($2 == "all") { print ""} else {print "cpu"$2":"$3+$5}}' | hcl -p -s cpu:100
top 3 processes by RAM:
$ atopsar -G -S -b 21:35 | awk '! /_top3_/ && $1 ~ /[0-9][0-9]:[0-9][0-9]/ { print "t," $3 "-" $2 "," $7 "-" $6 "," $11 "-" $10 "\n" $1","$4+0","$8+0","$12+0"\n"}' | hcl -x t
cpu usage by executable name since 11am:
atop -PPRC -r -b 11:00 | awk '{if ($8 != "") { if ($11+$12>0)print $8":"$11+$12; } else {print "";}}' | hcl -p -s auto
hcl can work with dtrace and display dynamic tracing information in realtime. The following example is from running dtrace on MacOS
syscalls count by syscall
$ dtrace -q -n 'syscall:::entry { @n[probefunc] = count(); } profile:::tick-1sec { printa("%S:%@d\n", @n); printf("\n"); clear(@n)}' | hcl -p -s auto
io count by executable name
$ dtrace -q -n 'io:::start { @n[execname] = count(); } profile:::tick-1sec { printa("%S:%@d\n", @n); printf("\n"); clear(@n)}' | hcl -p -s auto
io size by executable name
$ dtrace -q -n 'io:::start { @n[execname] = sum(args[0]->b_bcount); } profile:::tick-1sec { printa("%S:%@d\n", @n); printf("\n"); clear(@n)}' | hcl -p -s auto
Collect io size distribution for 10 seconds and show the result.
$ dtrace -q -n 'io:::start { @ = quantize(args[0]->b_bcount); } profile:::tick-1sec { printa("%@d", @); clear(@)} profile:::tick-10s {exit(0);}' | awk '{if (NF==3) print (0+$1)":"(0+$3); if ($0=="") print "";}' | hcl -s auto -p
Visualize page faults by process, update every second:
$ bpftrace -e 'software:faults:1 { @[comm] = count(); } interval:s:1 { print(@); clear(@)}' | hcl -p -s auto
For CPU counters, linux perf can be used to print out PMU events.
Show IPC by CPU:
$ perf stat -a -A -x ',' -e cycles,instructions --log-fd 1 -I 1000 | awk -v cores=`getconf _NPROCESSORS_ONLN` -F',' -W interactive '$0 ~ /instructions/ {print $2":"$8; if (NR % cores == 0) { print ""; } }' | hcl -p -s 2
In theory, HCL can handle many use-cases, like
However, for most such cases there’re usually better tools available; hcl is most useful for visualizing the data which is ‘right here on the machine’.