Xedge>> alp>> 返回
项目作者： tkuchiki

项目描述 ：
Access Log Profiler
高级语言： Go
项目地址: git://github.com/tkuchiki/alp.git
创建时间： 2015-09-24T10:10:02Z
项目社区：https://github.com/tkuchiki/alp
开源协议：MIT License
下载


alp

Test

alp is Access Log Profiler

日本語

Installation

Binary distribution

You can pick your download here, and install it as follows:

  1. sudo install <downloaded file> /usr/local/bin/alp

Using your distribution’s package system

macOS (Homebrew)

Install alp with Homebrew

  • brew install alp

asdf

Install alp with asdf and asdf-alp

  1. asdf plugin-add alp https://github.com/asdf-community/asdf-alp.git
  2. asdf install alp <version>
  3. asdf global alp <version>

The difference between v0.4.0 and v1.0.0

See: The difference between v0.4.0 and v1.0.0

Usage

  1. $ alp --help
  2. alp is the access log profiler for LTSV, JSON, Pcap, and others.
  4. Usage:
  5.   alp [flags]
  6.   alp [command]
  8. Available Commands:
  9.   completion  Generate the autocompletion script for the specified shell
  10.   count       Count by log entries
  11.   diff        Show the difference between the two profile results
  12.   help        Help about any command
  13.   json        Profile the logs for JSON
  14.   ltsv        Profile the logs for LTSV
  15.   pcap        Profile the HTTP requests for captured packets
  16.   regexp      Profile the logs that match a regular expression
  18. Flags:
  19.   -h, --help      help for alp
  20.   -v, --version   version for alp
  22. Use "alp [command] --help" for more information about a command.
  24. $ alp ltsv --help
  25. Profile the logs for LTSV
  27. Usage:
  28.   alp ltsv [flags]
  30. Flags:
  31.       --apptime-label string     Change the apptime label (default "apptime")
  32.       --config string            The configuration file
  33.       --decode-uri               Decode the URI
  34.       --dump string              Dump profiled data as YAML
  35.       --file string              The slowlog file
  36.   -f, --filters string           Only the logs are profiled that match the conditions
  37.       --format string            The output format (table, markdown, tsv, csv and html) (default "table")
  38.   -h, --help                     help for ltsv
  39.       --limit int                The maximum number of results to display (default 5000)
  40.       --load string              Load the profiled YAML data
  41.       --location string          Location name for the timezone (default "Local")
  42.   -m, --matching-groups string   Specifies Query matching groups separated by commas
  43.       --method-label string      Change the method label (default "method")
  44.       --noheaders                Output no header line at all (only --format=tsv, csv)
  45.       --nosave-pos               Do not save position file
  46.   -o, --output string            Specifies the results to display, separated by commas (default "all")
  47.       --page int                 Number of pages of pagination (default 100)
  48.       --percentiles string       Specifies the percentiles separated by commas
  49.       --pos string               The position file
  50.       --qs-ignore-values         Ignore the value of the query string. Replace all values with xxx (only use with -q)
  51.   -q, --query-string             Include the URI query string
  52.       --reqtime-label string     Change the reqtime label (default "reqtime")
  53.   -r, --reverse                  Sort results in reverse order
  54.       --show-footers             Output footer line at all (only --format=table, markdown)
  55.       --size-label string        Change the size label (default "size")
  56.       --sort string              Output the results in sorted order (default "count")
  57.       --status-label string      Change the status label (default "status")
  58.       --time-label string        Change the time label (default "time")
  59.       --uri-label string         Change the uri label (default "uri")
  61. $ alp json --help
  62. Profile the logs for JSON
  64. Usage:
  65.   alp json [flags]
  67. Flags:
  68.       --body-bytes-key string    Change the body_bytes key (default "body_bytes")
  69.       --config string            The configuration file
  70.       --decode-uri               Decode the URI
  71.       --dump string              Dump profiled data as YAML
  72.       --file string              The slowlog file
  73.   -f, --filters string           Only the logs are profiled that match the conditions
  74.       --format string            The output format (table, markdown, tsv, csv and html) (default "table")
  75.   -h, --help                     help for json
  76.       --limit int                The maximum number of results to display (default 5000)
  77.       --load string              Load the profiled YAML data
  78.       --location string          Location name for the timezone (default "Local")
  79.   -m, --matching-groups string   Specifies Query matching groups separated by commas
  80.       --method-key string        Change the method key (default "method")
  81.       --noheaders                Output no header line at all (only --format=tsv, csv)
  82.       --nosave-pos               Do not save position file
  83.   -o, --output string            Specifies the results to display, separated by commas (default "all")
  84.       --page int                 Number of pages of pagination (default 100)
  85.       --percentiles string       Specifies the percentiles separated by commas
  86.       --pos string               The position file
  87.       --qs-ignore-values         Ignore the value of the query string. Replace all values with xxx (only use with -q)
  88.   -q, --query-string             Include the URI query string
  89.       --reqtime-key string       Change the request_time key (default "request_time")
  90.       --restime-key string       Change the response_time key (default "response_time")
  91.   -r, --reverse                  Sort results in reverse order
  92.       --show-footers             Output footer line at all (only --format=table, markdown)
  93.       --sort string              Output the results in sorted order (default "count")
  94.       --status-key string        Change the status key (default "status")
  95.       --time-key string          Change the time key (default "time")
  96.       --uri-key string           Change the uri key (default "uri")
  98. $ alp regexp --help
  99. Profile the logs that match a regular expression
  101. Usage:
  102.   alp regexp [flags]
  104. Flags:
  105.       --body-bytes-subexp string   Change the body_bytes sub expression (default "body_bytes")
  106.       --config string              The configuration file
  107.       --decode-uri                 Decode the URI
  108.       --dump string                Dump profiled data as YAML
  109.       --file string                The slowlog file
  110.   -f, --filters string             Only the logs are profiled that match the conditions
  111.       --format string              The output format (table, markdown, tsv, csv and html) (default "table")
  112.   -h, --help                       help for regexp
  113.       --limit int                  The maximum number of results to display (default 5000)
  114.       --load string                Load the profiled YAML data
  115.       --location string            Location name for the timezone (default "Local")
  116.   -m, --matching-groups string     Specifies Query matching groups separated by commas
  117.       --method-subexp string       Change the method sub expression (default "method")
  118.       --noheaders                  Output no header line at all (only --format=tsv, csv)
  119.       --nosave-pos                 Do not save position file
  120.   -o, --output string              Specifies the results to display, separated by commas (default "all")
  121.       --page int                   Number of pages of pagination (default 100)
  122.       --pattern string             Regular expressions pattern matching the log (default "^(\\S+)\\s\\S+\\s+(\\S+\\s+)+\\[(?P<time>[^]]+)\\]\\s\"(?P<method>\\S*)\\s?(?P<uri>(?:[^\"]*(?:\\\\\")?)*)\\s([^\"]*)\"\\s(?P<status>\\S+)\\s(?P<body_bytes>\\S+)\\s\"((?:[^\"]*(?:\\\\\")?)*)\"\\s\"(?:.+)\"\\s(?P<response_time>\\S+)(?:\\s(?P<request_time>\\S+))?$")
  123.       --percentiles string         Specifies the percentiles separated by commas
  124.       --pos string                 The position file
  125.       --qs-ignore-values           Ignore the value of the query string. Replace all values with xxx (only use with -q)
  126.   -q, --query-string               Include the URI query string
  127.       --reqtime-subexp string      Change the request_time sub expression (default "request_time")
  128.       --restime-subexp string      Change the response_time sub expression (default "response_time")
  129.   -r, --reverse                    Sort results in reverse order
  130.       --show-footers               Output footer line at all (only --format=table, markdown)
  131.       --sort string                Output the results in sorted order (default "count")
  132.       --status-subexp string       Change the status sub expression (default "status")
  133.       --time-subexp string         Change the time sub expression (default "time")
  134.       --uri-subexp string          Change the uri sub expression (default "uri")
  136. $ alp pcap --help
  137. Profile the HTTP requests for captured packets
  139. Usage:
  140.   alp pcap [flags]
  142. Flags:
  143.       --config string             The configuration file
  144.       --decode-uri                Decode the URI
  145.       --dump string               Dump profiled data as YAML
  146.       --file string               The slowlog file
  147.   -f, --filters string            Only the logs are profiled that match the conditions
  148.       --format string             The output format (table, markdown, tsv, csv and html) (default "table")
  149.   -h, --help                      help for pcap
  150.       --limit int                 The maximum number of results to display (default 5000)
  151.       --load string               Load the profiled YAML data
  152.       --location string           Location name for the timezone (default "Local")
  153.   -m, --matching-groups string    Specifies Query matching groups separated by commas
  154.       --noheaders                 Output no header line at all (only --format=tsv, csv)
  155.       --nosave-pos                Do not save position file
  156.   -o, --output string             Specifies the results to display, separated by commas (default "all")
  157.       --page int                  Number of pages of pagination (default 100)
  158.       --pcap-server-ip strings    HTTP server IP address of the captured packets (default [127.0.0.1])
  159.       --pcap-server-port uint16   HTTP server TCP port of the captured packets (default 80)
  160.       --percentiles string        Specifies the percentiles separated by commas
  161.       --pos string                The position file
  162.       --qs-ignore-values          Ignore the value of the query string. Replace all values with xxx (only use with -q)
  163.   -q, --query-string              Include the URI query string
  164.   -r, --reverse                   Sort results in reverse order
  165.       --show-footers              Output footer line at all (only --format=table, markdown)
  166.       --sort string               Output the results in sorted order (default "count")
  168. $ alp diff --help
  169. Show the difference between the two profile results
  171. Usage:
  172.   alp diff <from> <to> [flags]
  174. Flags:
  175.       --from string   The comparison source file
  176.   -h, --help          help for diff
  177.       --to string     The comparison target file
  179. $ alp count --help
  180. Count by log entries
  182. Usage:
  183.   alp count [flags]
  185. Flags:
  186.       --file string      The access log file
  187.       --format string    Log format (json,ltsv,regexp) (default "json")
  188.   -h, --help             help for count
  189.       --keys string      Log key names (comma separated)
  190.       --pattern string   Regular expressions pattern matching the log. (only use with --format=regexp) (default "^(\\S+)\\s\\S+\\s+(\\S+\\s+)+\\[(?P<time>[^]]+)\\]\\s\"(?P<method>\\S*)\\s?(?P<uri>(?:[^\"]*(?:\\\\\")?)*)\\s([^\"]*)\"\\s(?P<status>\\S+)\\s(?P<body_bytes>\\S+)\\s\"((?:[^\"]*(?:\\\\\")?)*)\"\\s\"(?:.+)\"\\s(?P<response_time>\\S+)(?:\\s(?P<request_time>\\S+))?$")
  191.   -r, --reverse          Sort results in reverse order

ltsv

  • Parses a log in LTSV format
  • By default, the following labels are parsed:
    • time
      • datetime
    • method
      • HTTP Method
    • uri
      • URI
    • status
      • HTTP Status Code
    • apptime
      • Response time from the upstream server
    • reqtime
      • Request Processing Time (Response time after receiving a request)
  • The --xxx-label option can you change the name to any label
  1. $ cat example/logs/ltsv_access.log
  2. time:2015-09-06T05:58:05+09:00    method:POST    uri:/foo/bar?token=xxx&uuid=1234    status:200    size:12    apptime:0.057
  3. time:2015-09-06T05:58:41+09:00    method:POST    uri:/foo/bar?token=yyy    status:200    size:34    apptime:0.100
  4. time:2015-09-06T06:00:42+09:00    method:GET    uri:/foo/bar?token=zzz    status:200    size:56    apptime:0.123
  5. time:2015-09-06T06:00:43+09:00    method:GET    uri:/foo/bar    status:400    size:15    apptime:-
  6. time:2015-09-06T05:58:44+09:00    method:POST    uri:/foo/bar?token=yyy    status:200    size:34    apptime:0.234
  7. time:2015-09-06T05:58:44+09:00    method:POST    uri:/hoge/piyo?id=yyy    status:200    size:34    apptime:0.234
  8. time:2015-09-06T05:58:05+09:00    method:POST    uri:/foo/bar?token=xxx&uuid=1234    status:200    size:12    apptime:0.057
  9. time:2015-09-06T05:58:41+09:00    method:POST    uri:/foo/bar?token=yyy    status:200    size:34    apptime:0.100
  10. time:2015-09-06T06:00:42+09:00    method:GET    uri:/foo/bar?token=zzz    status:200    size:56    apptime:0.123
  11. time:2015-09-06T06:00:43+09:00    method:GET    uri:/foo/bar    status:400    size:15    apptime:-
  12. time:2015-09-06T06:00:43+09:00    method:GET    uri:/diary/entry/1234    status:200    size:15    apptime:0.135
  13. time:2015-09-06T06:00:43+09:00    method:GET    uri:/diary/entry/5678    status:200    size:30    apptime:0.432
  14. time:2015-09-06T06:00:43+09:00    method:GET    uri:/foo/bar/5xx    status:504    size:15    apptime:60.000
  16. $ cat example/logs/ltsv_access.log | alp ltsv
  17. +-------+-----+-----+-----+-----+-----+--------+-------------------+--------+--------+--------+--------+--------+--------+--------+--------+-----------+-----------+-----------+-----------+
  18. | COUNT | 1XX | 2XX | 3XX | 4XX | 5XX | METHOD |        URI        |  MIN   |  MAX   |  SUM   |  AVG   |  P90   |  P95   |  P99   | STDDEV | MIN(BODY) | MAX(BODY) | SUM(BODY) | AVG(BODY) |
  19. +-------+-----+-----+-----+-----+-----+--------+-------------------+--------+--------+--------+--------+--------+--------+--------+--------+-----------+-----------+-----------+-----------+
  20. |     1 |   0 |   1 |   0 |   0 |   0 | GET    | /req              |  0.321 |  0.321 |  0.321 |  0.321 |  0.321 |  0.321 |  0.321 |  0.000 |    15.000 |    15.000 |    15.000 |    15.000 |
  21. |     1 |   0 |   1 |   0 |   0 |   0 | POST   | /hoge/piyo        |  0.234 |  0.234 |  0.234 |  0.234 |  0.234 |  0.234 |  0.234 |  0.000 |    34.000 |    34.000 |    34.000 |    34.000 |
  22. |     1 |   0 |   1 |   0 |   0 |   0 | GET    | /diary/entry/1234 |  0.135 |  0.135 |  0.135 |  0.135 |  0.135 |  0.135 |  0.135 |  0.000 |    15.000 |    15.000 |    15.000 |    15.000 |
  23. |     1 |   0 |   1 |   0 |   0 |   0 | GET    | /diary/entry/5678 |  0.432 |  0.432 |  0.432 |  0.432 |  0.432 |  0.432 |  0.432 |  0.000 |    30.000 |    30.000 |    30.000 |    30.000 |
  24. |     1 |   0 |   0 |   0 |   0 |   1 | GET    | /foo/bar/5xx      | 60.000 | 60.000 | 60.000 | 60.000 | 60.000 | 60.000 | 60.000 |  0.000 |    15.000 |    15.000 |    15.000 |    15.000 |
  25. |     2 |   0 |   2 |   0 |   0 |   0 | GET    | /foo/bar          |  0.123 |  0.123 |  0.246 |  0.123 |  0.123 |  0.123 |  0.123 |  0.000 |    56.000 |    56.000 |   112.000 |    56.000 |
  26. |     5 |   0 |   5 |   0 |   0 |   0 | POST   | /foo/bar          |  0.057 |  0.234 |  0.548 |  0.110 |  0.234 |  0.234 |  0.234 |  0.065 |    12.000 |    34.000 |   126.000 |    25.200 |
  27. +-------+-----+-----+-----+-----+-----+--------+-------------------+--------+--------+--------+--------+--------+--------+--------+--------+-----------+-----------+-----------+-----------+

Log format

Apache

  1. LogFormat "time:%t\tforwardedfor:%{X-Forwarded-For}i\thost:%h\treq:%r\tstatus:%>s\tmethod:%m\turi:%U%q\tsize:%B\treferer:%{Referer}i\tua:%{User-Agent}i\treqtime_microsec:%D\tapptime:%D\tcache:%{X-Cache}o\truntime:%{X-Runtime}o\tvhost:%{Host}i" ltsv

Nginx

  1. log_format ltsv "time:$time_local"
  2.                 "\thost:$remote_addr"
  3.                 "\tforwardedfor:$http_x_forwarded_for"
  4.                 "\treq:$request"
  5.                 "\tstatus:$status"
  6.                 "\tmethod:$request_method"
  7.                 "\turi:$request_uri"
  8.                 "\tsize:$body_bytes_sent"
  9.                 "\treferer:$http_referer"
  10.                 "\tua:$http_user_agent"
  11.                 "\treqtime:$request_time"
  12.                 "\tcache:$upstream_http_x_cache"
  13.                 "\truntime:$upstream_http_x_runtime"
  14.                 "\tapptime:$upstream_response_time"
  15.                 "\tvhost:$host";

H2O

  1. access-log:
  2.   format: "time:%t\thost:%h\tua:\"%{User-agent}i\"\tstatus:%s\treq:%r\turi:%U\tapptime:%{duration}x\tsize:%b\tmethod:%m"

json

  • Parse a log with one JSON per line
  • By default, the following keys are parsed:
    • time
      • datetime
    • method
      • HTTP Method
    • uri
      • URI
    • status
      • HTTP Status Code
    • response_time
      • Response time from the upstream server
    • request_time
      • Request Processing Time (Response time after receiving a request)
  • The --xxx-key option can you change the name to any key
  1. $ cat example/logs/json_access.log
  2. {"time":"2015-09-06T05:58:05+09:00","method":"POST","uri":"/foo/bar?token=xxx&uuid=1234","status":200,"body_bytes":12,"response_time":0.057}
  3. {"time":"2015-09-06T05:58:41+09:00","method":"POST","uri":"/foo/bar?token=yyy","status":200,"body_bytes":34,"response_time":0.100}
  4. {"time":"2015-09-06T06:00:42+09:00","method":"GET","uri":"/foo/bar?token=zzz","status":200,"body_bytes":56,"response_time":0.123}
  5. {"time":"2015-09-06T06:00:43+09:00","method":"GET","uri":"/foo/bar","status":400,"body_bytes":15,"response_time":"-"}
  6. {"time":"2015-09-06T05:58:44+09:00","method":"POST","uri":"/foo/bar?token=yyy","status":200,"body_bytes":34,"response_time":0.234}
  7. {"time":"2015-09-06T05:58:44+09:00","method":"POST","uri":"/hoge/piyo?id=yyy","status":200,"body_bytes":34,"response_time":0.234}
  8. {"time":"2015-09-06T05:58:05+09:00","method":"POST","uri":"/foo/bar?token=xxx&uuid=1234","status":200,"body_bytes":12,"response_time":0.057}
  9. {"time":"2015-09-06T05:58:41+09:00","method":"POST","uri":"/foo/bar?token=yyy","status":200,"body_bytes":34,"response_time":0.100}
  10. {"time":"2015-09-06T06:00:42+09:00","method":"GET","uri":"/foo/bar?token=zzz","status":200,"body_bytes":56,"response_time":0.123}
  11. {"time":"2015-09-06T06:00:43+09:00","method":"GET","uri":"/foo/bar","status":400,"body_bytes":15,"response_time":"-"}
  12. {"time":"2015-09-06T06:00:43+09:00","method":"GET","uri":"/diary/entry/1234","status":200,"body_bytes":15,"response_time":0.135}
  13. {"time":"2015-09-06T06:00:43+09:00","method":"GET","uri":"/diary/entry/5678","status":200,"body_bytes":30,"response_time":0.432}
  14. {"time":"2015-09-06T06:00:43+09:00","method":"GET","uri":"/foo/bar/5xx","status":504,"body_bytes":15,"response_time":60.000}
  16. $ cat example/logs/json_access.log | alp json
  17. +-------+-----+-----+-----+-----+-----+--------+-------------------+--------+--------+--------+--------+--------+--------+--------+--------+-----------+-----------+-----------+-----------+
  18. | COUNT | 1XX | 2XX | 3XX | 4XX | 5XX | METHOD |        URI        |  MIN   |  MAX   |  SUM   |  AVG   |  P90   |  P95   |  P99   | STDDEV | MIN(BODY) | MAX(BODY) | SUM(BODY) | AVG(BODY) |
  19. +-------+-----+-----+-----+-----+-----+--------+-------------------+--------+--------+--------+--------+--------+--------+--------+--------+-----------+-----------+-----------+-----------+
  20. |     1 |   0 |   1 |   0 |   0 |   0 | GET    | /req              |  0.321 |  0.321 |  0.321 |  0.321 |  0.321 |  0.321 |  0.321 |  0.000 |    15.000 |    15.000 |    15.000 |    15.000 |
  21. |     1 |   0 |   1 |   0 |   0 |   0 | POST   | /hoge/piyo        |  0.234 |  0.234 |  0.234 |  0.234 |  0.234 |  0.234 |  0.234 |  0.000 |    34.000 |    34.000 |    34.000 |    34.000 |
  22. |     1 |   0 |   1 |   0 |   0 |   0 | GET    | /diary/entry/1234 |  0.135 |  0.135 |  0.135 |  0.135 |  0.135 |  0.135 |  0.135 |  0.000 |    15.000 |    15.000 |    15.000 |    15.000 |
  23. |     1 |   0 |   1 |   0 |   0 |   0 | GET    | /diary/entry/5678 |  0.432 |  0.432 |  0.432 |  0.432 |  0.432 |  0.432 |  0.432 |  0.000 |    30.000 |    30.000 |    30.000 |    30.000 |
  24. |     1 |   0 |   0 |   0 |   0 |   1 | GET    | /foo/bar/5xx      | 60.000 | 60.000 | 60.000 | 60.000 | 60.000 | 60.000 | 60.000 |  0.000 |    15.000 |    15.000 |    15.000 |    15.000 |
  25. |     2 |   0 |   2 |   0 |   0 |   0 | GET    | /foo/bar          |  0.123 |  0.123 |  0.246 |  0.123 |  0.123 |  0.123 |  0.123 |  0.000 |    56.000 |    56.000 |   112.000 |    56.000 |
  26. |     5 |   0 |   5 |   0 |   0 |   0 | POST   | /foo/bar          |  0.057 |  0.234 |  0.548 |  0.110 |  0.234 |  0.234 |  0.234 |  0.065 |    12.000 |    34.000 |   126.000 |    25.200 |
  27. +-------+-----+-----+-----+-----+-----+--------+-------------------+--------+--------+--------+--------+--------+--------+--------+--------+-----------+-----------+-----------+-----------+

Log format

Apache

  1. LogFormat "{\"time\":\"%t\",\"forwardedfor\":\"%{X-Forwarded-For}i\",\"host\":\"%h\",\"req\":\"%r\",\"status\":%>s,\"method\":\"%m\",\"uri\":\"%U%q\",\"body_bytes\":%B,\"referer\":\"%{Referer}i\",\"ua\":\"%{User-Agent}i\",\"reqtime_microsec\":%D,\"response_time\":%D,\"cache\":\"%{X-Cache}o\",\"runtime\":\"%{X-Runtime}o\",\"vhost\":\"%{Host}i\"}" json

Nginx

  1.     log_format json escape=json '{"time":"$time_local",'
  2.                                 '"host":"$remote_addr",'
  3.                                 '"forwardedfor":"$http_x_forwarded_for",'
  4.                                 '"req":"$request",'
  5.                                 '"status":"$status",'
  6.                                 '"method":"$request_method",'
  7.                                 '"uri":"$request_uri",'
  8.                                 '"body_bytes":$body_bytes_sent,'
  9.                                 '"referer":"$http_referer",'
  10.                                 '"ua":"$http_user_agent",'
  11.                                 '"request_time":$request_time,'
  12.                                 '"cache":"$upstream_http_x_cache",'
  13.                                 '"runtime":"$upstream_http_x_runtime",'
  14.                                 '"response_time":"$upstream_response_time",'
  15.                                 '"vhost":"$host"}';

H2O

  1. access-log:
  2.   escape: json
  3.   format: '{"time":"%t","host":"%h","ua":"%{User-agent}i","status":%s,"req":"%r","uri":"%U","response_time":%{duration}x,"body_bytes":%b,"method":"%m"}'

regexp

  • Parses the log to match the regular expression
  • By default, the following named capture groups are parsed:
    • time
      • datetime
    • method
      • HTTP Method
    • uri
      • URI
    • status
      • HTTP Status Code
    • response_time
      • Response time from the upstream server
    • request_time
      • Request Processing Time (Response time after receiving a request)
  • The --xxx-subexp option can you change the name to any named capture groups
  1. $ cat example/logs/combined_access.log
  2. 127.0.0.1 - - [06/Sep/2015:05:58:05 +0900] "POST /foo/bar?token=xxx&uuid=1234 HTTP/1.1" 200 12 "-" "curl/7.54.0" "-" 0.057
  3. 127.0.0.1 - - [06/Sep/2015:05:58:41 +0900] "POST /foo/bar?token=yyy HTTP/1.1" 200 34 "-" "curl/7.54.0" "-" 0.100
  4. 127.0.0.1 - - [06/Sep/2015:06:00:42 +0900] "GET /foo/bar?token=zzz HTTP/1.1" 200 56 "-" "curl/7.54.0" "-" 0.123
  5. 127.0.0.1 - - [06/Sep/2015:06:00:43 +0900] "GET /foo/bar HTTP/1.1" 400 15 "-" "curl/7.54.0" "-" -
  6. 127.0.0.1 - - [06/Sep/2015:05:58:44 +0900] "POST /foo/bar?token=yyy HTTP/1.1" 200 34 "-" "curl/7.54.0" "-" 0.234
  7. 127.0.0.1 - - [06/Sep/2015:05:58:44 +0900] "POST /hoge/piyo?id=yyy HTTP/1.1" 200 34 "-" "curl/7.54.0" "-" 0.234
  8. 127.0.0.1 - - [06/Sep/2015:05:58:05 +0900] "POST foo/bar?token=xxx&uuid=1234 HTTP/1.1" 200 12 "-" "curl/7.54.0" "-" 0.057
  9. 127.0.0.1 - - [06/Sep/2015:05:58:41 +0900] "POST /foo/bar?token=yyy HTTP/1.1" 200 34 "-" "curl/7.54.0" "-" 0.100
  10. 127.0.0.1 - - [06/Sep/2015:06:00:42 +0900] "GET /foo/bar?token=zzz HTTP/1.1" 200 56 "-" "curl/7.54.0" "-" 0.123
  11. 127.0.0.1 - - [06/Sep/2015:06:00:43 +0900] "GET /foo/bar HTTP/1.1" 400 15 "-" "curl/7.54.0" "-" -
  12. 127.0.0.1 - - [06/Sep/2015:06:00:43 +0900] "GET /diary/entry/1234 HTTP/1.1" 200 15 "-" "curl/7.54.0" "-" 0.135
  13. 127.0.0.1 - - [06/Sep/2015:06:00:43 +0900] "GET /diary/entry/5678 HTTP/1.1" 200 30 "-" "curl/7.54.0" "-" 0.432
  14. 127.0.0.1 - - [06/Sep/2015:06:00:43 +0900] "GET /foo/bar/5xx HTTP/1.1" 504 15 "-" "curl/7.54.0" "-" 60.000
  16. $ cat example/logs/combined_access.log | alp regexp
  17. +-------+-----+-----+-----+-----+-----+--------+-------------------+--------+--------+--------+--------+--------+--------+--------+--------+-----------+-----------+-----------+-----------+
  18. | COUNT | 1XX | 2XX | 3XX | 4XX | 5XX | METHOD |        URI        |  MIN   |  MAX   |  SUM   |  AVG   |  P90   |  P95   |  P99   | STDDEV | MIN(BODY) | MAX(BODY) | SUM(BODY) | AVG(BODY) |
  19. +-------+-----+-----+-----+-----+-----+--------+-------------------+--------+--------+--------+--------+--------+--------+--------+--------+-----------+-----------+-----------+-----------+
  20. |     1 |   0 |   1 |   0 |   0 |   0 | GET    | /req              |  0.321 |  0.321 |  0.321 |  0.321 |  0.321 |  0.321 |  0.321 |  0.000 |    15.000 |    15.000 |    15.000 |    15.000 |
  21. |     1 |   0 |   1 |   0 |   0 |   0 | POST   | /hoge/piyo        |  0.234 |  0.234 |  0.234 |  0.234 |  0.234 |  0.234 |  0.234 |  0.000 |    34.000 |    34.000 |    34.000 |    34.000 |
  22. |     1 |   0 |   1 |   0 |   0 |   0 | GET    | /diary/entry/1234 |  0.135 |  0.135 |  0.135 |  0.135 |  0.135 |  0.135 |  0.135 |  0.000 |    15.000 |    15.000 |    15.000 |    15.000 |
  23. |     1 |   0 |   1 |   0 |   0 |   0 | GET    | /diary/entry/5678 |  0.432 |  0.432 |  0.432 |  0.432 |  0.432 |  0.432 |  0.432 |  0.000 |    30.000 |    30.000 |    30.000 |    30.000 |
  24. |     1 |   0 |   0 |   0 |   0 |   1 | GET    | /foo/bar/5xx      | 60.000 | 60.000 | 60.000 | 60.000 | 60.000 | 60.000 | 60.000 |  0.000 |    15.000 |    15.000 |    15.000 |    15.000 |
  25. |     2 |   0 |   2 |   0 |   0 |   0 | GET    | /foo/bar          |  0.123 |  0.123 |  0.246 |  0.123 |  0.123 |  0.123 |  0.123 |  0.000 |    56.000 |    56.000 |   112.000 |    56.000 |
  26. |     5 |   0 |   5 |   0 |   0 |   0 | POST   | /foo/bar          |  0.057 |  0.234 |  0.548 |  0.110 |  0.234 |  0.234 |  0.234 |  0.065 |    12.000 |    34.000 |   126.000 |    25.200 |
  27. +-------+-----+-----+-----+-----+-----+--------+-------------------+--------+--------+--------+--------+--------+--------+--------+--------+-----------+-----------+-----------+-----------+

Log format

Apache

  1. LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\" %D" combined_plus

Nginx

  1.     log_format combined_plus '$remote_addr - $remote_user [$time_local] '
  2.                              '"$request" $status $body_bytes_sent '
  3.                              '"$http_referer" "$http_user_agent" $upstream_response_time $request_time';

H2O

  1. access-log:
  2.   format: "%h %l %u %t \"%r\" %s %b \"%{Referer}i\" \"%{User-agent}i\" %{duration}x"

pcap

  • Parses the pcap file to extract HTTP request/response packets to analyze the stats
    • Note that the actual response time may time duration from the actual response time because the difference in the timestamp of packet capturing is regarded as the real response time
    • The IP address and TCP port number of the server are required to distinguish between HTTP requests/responses to the server
  • Able to specify the IP address of the HTTP server with the --pcap-server-ip option
    • This option can be specified more than once
    • By default, it automatically obtains the IP address from the network interfaces of its own host and uses it
    • However, 127.0.0.1 and ::1 will be the defaults in environments where permissions to retrieve network interface information are restricted.
  • Able to specify the TCP port of the HTTP server with the --pcap-server-port option
    • The default server port number is 80.
  • Cannot be used with --pos. (not yet supported)
  1. $ sudo tcpdump -i lo port 5000 -s0 -w http.cap -Z $USER
  2. tcpdump: listening on lo, link-type EN10MB (Ethernet), capture size 262144 bytes
  3. 10000 packets captured
  4. 20000 packets received by filter
  5. 0 packets dropped by kernel
  7. $ alp pcap --file=http.cap --pcap-server-port=5000
  8. +-------+-----+-----+-----+-----+-----+--------+-------------------+--------+--------+--------+--------+--------+--------+--------+--------+-----------+-----------+-----------+-----------+
  9. | COUNT | 1XX | 2XX | 3XX | 4XX | 5XX | METHOD |        URI        |  MIN   |  MAX   |  SUM   |  AVG   |  P90   |  P95   |  P99   | STDDEV | MIN(BODY) | MAX(BODY) | SUM(BODY) | AVG(BODY) |
  10. +-------+-----+-----+-----+-----+-----+--------+-------------------+--------+--------+--------+--------+--------+--------+--------+--------+-----------+-----------+-----------+-----------+
  11. |     1 |   0 |   1 |   0 |   0 |   0 | GET    | /req              |  0.321 |  0.321 |  0.321 |  0.321 |  0.321 |  0.321 |  0.321 |  0.000 |    15.000 |    15.000 |    15.000 |    15.000 |
  12. |     1 |   0 |   1 |   0 |   0 |   0 | POST   | /hoge/piyo        |  0.234 |  0.234 |  0.234 |  0.234 |  0.234 |  0.234 |  0.234 |  0.000 |    34.000 |    34.000 |    34.000 |    34.000 |
  13. |     1 |   0 |   1 |   0 |   0 |   0 | GET    | /diary/entry/1234 |  0.135 |  0.135 |  0.135 |  0.135 |  0.135 |  0.135 |  0.135 |  0.000 |    15.000 |    15.000 |    15.000 |    15.000 |
  14. |     1 |   0 |   1 |   0 |   0 |   0 | GET    | /diary/entry/5678 |  0.432 |  0.432 |  0.432 |  0.432 |  0.432 |  0.432 |  0.432 |  0.000 |    30.000 |    30.000 |    30.000 |    30.000 |
  15. |     1 |   0 |   0 |   0 |   0 |   1 | GET    | /foo/bar/5xx      | 60.000 | 60.000 | 60.000 | 60.000 | 60.000 | 60.000 | 60.000 |  0.000 |    15.000 |    15.000 |    15.000 |    15.000 |
  16. |     2 |   0 |   2 |   0 |   0 |   0 | GET    | /foo/bar          |  0.123 |  0.123 |  0.246 |  0.123 |  0.123 |  0.123 |  0.123 |  0.000 |    56.000 |    56.000 |   112.000 |    56.000 |
  17. |     5 |   0 |   5 |   0 |   0 |   0 | POST   | /foo/bar          |  0.057 |  0.234 |  0.548 |  0.110 |  0.234 |  0.234 |  0.234 |  0.065 |    12.000 |    34.000 |   126.000 |    25.200 |
  18. +-------+-----+-----+-----+-----+-----+--------+-------------------+--------+--------+--------+--------+--------+--------+--------+--------+-----------+-----------+-----------+-----------+

diff

  • Show the difference between the two profile results
  • + means an increasing number of requests and body size, and response time is slower
  • - means a decreasing number of requests and body size, and response time is faster
  1. $ cat /path/to/access.log | alp json --dump dumpfile1.yaml
  3. $ cat /path/to/access.log | alp json --dump dumpfile2.yaml
  5. $ alp diff dumpfile1.yaml dumpfile2.yaml -o count,2xx,method,uri,min,max,sum,avg,p90 --show-footers
  6. +---------+---------+--------+-------------------+-----------------+-----------------+-----------------+-----------------+-----------------+
  7. |  COUNT  |   2XX   | METHOD |        URI        |       MIN       |       MAX       |       SUM       |       AVG       |       P90       |
  8. +---------+---------+--------+-------------------+-----------------+-----------------+-----------------+-----------------+-----------------+
  9. | 1       | 1       | GET    | /req              | 0.221 (-0.100)  | 0.221 (-0.100)  | 0.221 (-0.100)  | 0.221 (-0.100)  | 0.221 (-0.100)  |
  10. | 1       | 1       | GET    | /new              | 0.221           | 0.221           | 0.221           | 0.221           | 0.221           |
  11. | 1       | 1       | POST   | /hoge/piyo        | 0.134 (-0.100)  | 0.134 (-0.100)  | 0.134 (-0.100)  | 0.134 (-0.100)  | 0.134 (-0.100)  |
  12. | 1       | 1       | GET    | /diary/entry/1234 | 0.035 (-0.100)  | 0.035 (-0.100)  | 0.035 (-0.100)  | 0.035 (-0.100)  | 0.035 (-0.100)  |
  13. | 1       | 0       | GET    | /foo/bar/5xx      | 59.000 (-1.000) | 59.000 (-1.000) | 59.000 (-1.000) | 59.000 (-1.000) | 59.000 (-1.000) |
  14. | 2 (+1)  | 2 (+1)  | GET    | /diary/entry/5678 | 0.332 (-0.100)  | 2.332 (+1.900)  | 2.664 (+2.232)  | 1.332 (+0.900)  | 2.332 (+1.900)  |
  15. | 2       | 2       | GET    | /foo/bar          | 0.023 (-0.100)  | 0.023 (-0.100)  | 0.046 (-0.200)  | 0.023 (-0.100)  | 0.023 (-0.100)  |
  16. | 5       | 5       | POST   | /foo/bar          | 0.047 (-0.010)  | 0.134 (-0.100)  | 0.378 (-0.170)  | 0.076 (-0.034)  | 0.134 (-0.100)  |
  17. +---------+---------+--------+-------------------+-----------------+-----------------+-----------------+-----------------+-----------------+
  18. | 14 (+2) | 13 (+2) |
  19. +---------+---------+--------+-------------------+-----------------+-----------------+-----------------+-----------------+-----------------+

Global options

See: Usage samples

  • -c, --config
    • The configuration file
    • YAML
  • --file=FILE
    • The access log file
  • -d, --dump=DUMP
    • File path for creating the profile results to a file
  • -l, --load=LOAD
    • File path to read the results of the profile created with the -d, --dump option
    • Can expect it to work fast if you change the --sort and --reverse options for the same profile results
  • --sort=count
    • Output the results in sorted order
    • Sort in ascending order
    • max, min, sum, avg
    • max-body, min-body, sum-body, avg-body
    • p90, p95, p99, stddev
    • uri
    • method
    • count
    • The default is count
    • p90, p95, and p99 are modified by the values specified in --percentiles
  • -r, --reverse
    • Sort in desecending order
  • -q, --query-string
    • URIs up to and including query strings are included in the profile
  • --qs-ignore-values
    • Ignore the value of the query string
    • It’s not enabled unless use with -q, --query-string
  • --decode-uri
    • Decode the URI
  • --format=table
    • Print the profile results in a table, Markdown, TSV, CSV and HTML format
    • The default is table format
  • --noheaders
    • Print no header when TSV and CSV format
  • --show-footers
    • Print the total number of each 1xx ~ 5xx in the footer of the table or Markdown format
  • --limit=5000
    • Maximum number of profile results to be printed
    • This setting is to avoid using too much memory
    • The default is 5000 lines
  • --location="Local"
    • The timezone of the time specified in the filter condition.
    • Default is localhost timezone
  • -o, --output="all"
    • Specify the profile results to be print, separated by commas
    • count,1xx, 2xx, 3xx, 4xx, 5xx, method, uri, min, max, sum, avg, p90, p95, p99, stddev, min_body, max_body, sum_body, avg_body
      • p90, p95, and p99 are modified by the values specified in --percentiles
    • The default is all
  • -m, --matching-groups=PATTERN,...
    • Treat URIs that match regular expressions as the same URI
    • Evaluate in the specified order. If matched, no further evaluation is performed.
    • See URI matching groups
  • -f, --filters=FILTERS
    • Filters the targets for profile
    • See Filter
  • --pos=POSITION_FILE
    • Stores the number of bytes to which the file has been read.
    • If the number of bytes is stored in the POSITION_FILE, the data after that number of bytes will be profiled
    • You can profile without truncating the file
      • Also, it is expected to work fast because it seeks and skips files
  • --nosave-pos
    • Data after the number of bytes specified by --pos is profiled, but the number of bytes reads is not stored
  • --percentiles
    • Specifies the percentile values to output, separated by commas
    • The default is 90,95,99

URI matching groups

Consider the following cases like /diary/entry/1234 and /diary/entry/5678.
If you simply profile URIs with different parameters on the same route, they will be profiled by parameter, but you may want to profile them by the route.

  1. $ cat example/logs/ltsv_access.log | alp ltsv --filters "Uri matches '^/diary/entry'"
  2. +-------+-----+-----+-----+-----+-----+--------+-------------------+-------+-------+-------+-------+-------+-------+-------+--------+-----------+-----------+-----------+-----------+
  3. | COUNT | 1XX | 2XX | 3XX | 4XX | 5XX | METHOD |        URI        |  MIN  |  MAX  |  SUM  |  AVG  |  P90  |  P95  |  P99  | STDDEV | MIN(BODY) | MAX(BODY) | SUM(BODY) | AVG(BODY) |
  4. +-------+-----+-----+-----+-----+-----+--------+-------------------+-------+-------+-------+-------+-------+-------+-------+--------+-----------+-----------+-----------+-----------+
  5. |     1 |   0 |   1 |   0 |   0 |   0 | GET    | /diary/entry/1234 | 0.135 | 0.135 | 0.135 | 0.135 | 0.135 | 0.135 | 0.135 |  0.000 |    15.000 |    15.000 |    15.000 |    15.000 |
  6. |     1 |   0 |   1 |   0 |   0 |   0 | GET    | /diary/entry/5678 | 0.432 | 0.432 | 0.432 | 0.432 | 0.432 | 0.432 | 0.432 |  0.000 |    30.000 |    30.000 |    30.000 |    30.000 |
  7. +-------+-----+-----+-----+-----+-----+--------+-------------------+-------+-------+-------+-------+-------+-------+-------+--------+-----------+-----------+-----------+-----------+
  1. $ cat example/logs/ltsv_access.log | alp ltsv --filters "Uri matches '^/diary/entry'" -m "/diary/entry/.+"
  2. +-------+-----+-----+-----+-----+-----+--------+-----------------+-------+-------+-------+-------+-------+-------+-------+--------+-----------+-----------+-----------+-----------+
  3. | COUNT | 1XX | 2XX | 3XX | 4XX | 5XX | METHOD |       URI       |  MIN  |  MAX  |  SUM  |  AVG  |  P90  |  P95  |  P99  | STDDEV | MIN(BODY) | MAX(BODY) | SUM(BODY) | AVG(BODY) |
  4. +-------+-----+-----+-----+-----+-----+--------+-----------------+-------+-------+-------+-------+-------+-------+-------+--------+-----------+-----------+-----------+-----------+
  5. |     2 |   0 |   2 |   0 |   0 |   0 | GET    | /diary/entry/.+ | 0.135 | 0.432 | 0.567 | 0.283 | 0.432 | 0.432 | 0.432 |  0.148 |    15.000 |    30.000 |    45.000 |    22.500 |
  6. +-------+-----+-----+-----+-----+-----+--------+-----------------+-------+-------+-------+-------+-------+-------+-------+--------+-----------+-----------+-----------+-----------+

In such a case, there is an option -m, --matching-groups=PATTERN,....
You can also specify multiple items separated by commas.

Filter

It is a function to include or exclude targets according to the conditions.

Variables

Filter on the following variables:.

Operators

The following operators are available:.

  • +, -, *, /, %, **(pow)
  • ==, !=, <, >, <=, >=
  • not, !
  • and, &&
  • or, ||
  • matches
    • e.g.
      • Uri matches "PATTERN"
      • not(Uri matches "PATTERN")
  • contains
    • e.g.
      • Uri contains "STRING"
      • not(Uri contains "STRING")
  • startsWith
    • e.g.
      • Uri startsWith "PREFIX"
      • not(Uri startsWith "PREFIX")
  • endsWith
    • e.g.
      • Uri endsWith "SUFFIX"
      • not(Uri endsWith "SUFFIX")
  • in
    • e.g.
      • Method in ["GET", "POST"]
      • Method not in ["GET", "POST"]

See: https://github.com/antonmedv/expr/blob/master/docs/Language-Definition.md

Functions

  • TimeAgo(duration)
    • now - duration
    • ns, us or µs, ms, s, m, h
    • e.g.
      • Time >= TimeAgo("5m")
  • BetweenTime(val, start, end)
    • Like SQL’s BETWEEN, returns start <= val && val <= end
    • e.g.
      • BetweenTime(Time, "2019-08-06T00:00:00", "2019-08-06T00:05:00")

Usage samples

See: Usage samples

Donation

Donations are welcome as always!
:heart: Sponsor