kahttp - Keep-alive http test program

This test program is intended for testing many simultaneous
(keep-alive) http connections. It is not intended for throughput or
latency measurements but is designed to test the limit for how many
simultaneous client connectons a http server or proxy can handle.

Build

./build.sh image
ls ./image/kahttp

Usage

Local usage;

# Start server;
kahttp -server -address [::]:5080
# In another shell;
wget -q -O - http://127.0.0.1:5080/
kahttp -address http://127.0.0.1:5080/ -monitor -rate 400 -nclients 40 -timeout 10s | jq .
# In yet another shell;
tcpdump -ni lo 'tcp[tcpflags] == tcp-syn'
# Or for ipv6;
tcpdump -ni lo tcp and 'ip6[13+40]&0x2!=0'

One way to verify that keep-alive is used done is to trace the SYN
packets with tcpdump. If there are an initial burst of and then
silence keep-alive is used. A steady stream os SYNs indicates that
keep-alive is not used.

Deploy a kahttp server on Kubernetes;

kubectl apply -f https://raw.githubusercontent.com/Nordix/kahttp/master/kahttp.yaml

NOTE: The server is very primitive and is intended for simple tests only.

The server will use an internal self-signed certificate for https by
default. See below for an instruction howto use a k8s tls secret.

Measure keep-alive connections

Kahttp starts a number of http clients according to the -nclients
parameter. Each of these clients will make periodic http GET requests
towards the URL specified by the -address parameter. The -rate
parameter specifies the total number of http requests per
second. This is divided among the clients so a rate of 400 and 40
clients will result in 10 requests per second per client. The
keep-alive time is limited (but different) on servers so to make sure
connections are kept a certain rate per client must be maintained.

The key metric for measuring keep-alive connections is Dials. This
is the total number of actual TCP connects that the clients do. Below
is an example where all connections are kept;

> kahttp -address http://127.0.0.1:5080/ -monitor -rate 400 -nclients 40 -timeout 10s | jq .
Clients act/fail/Dials: 40/0/40, Packets send/rec/dropped: 399/399/0
Clients act/fail/Dials: 40/0/40, Packets send/rec/dropped: 799/799/0
Clients act/fail/Dials: 40/0/40, Packets send/rec/dropped: 1199/1199/0
Clients act/fail/Dials: 40/0/40, Packets send/rec/dropped: 1600/1600/0
Clients act/fail/Dials: 40/0/40, Packets send/rec/dropped: 2000/2000/0
Clients act/fail/Dials: 40/0/40, Packets send/rec/dropped: 2400/2400/0
Clients act/fail/Dials: 40/0/40, Packets send/rec/dropped: 2800/2800/0
Clients act/fail/Dials: 40/0/40, Packets send/rec/dropped: 3200/3200/0
Clients act/fail/Dials: 40/0/40, Packets send/rec/dropped: 3600/3600/0
{
  "Started": "2019-04-02T13:02:06.815903366+02:00",
  "Duration": 10000112870,
  "Rate": 400,
  "Clients": 40,
  "Dials": 40,
  "FailedConnections": 0,
  "Sent": 3999,
  "Received": 3999,
  "Dropped": 0,
  "FailedConnects": 0
}

In this case we have 40 clients making 3999 requests in total but the
number of Dials is 40, so each client make only one connection which
is kept alive for the entire run.

This shows the case where keep-alive is disabled using the
-disable_ka flag;

# kahttp -nclients 10 -monitor -address https://10.0.0.2/ -host_stats -disable_ka | jq .
Clients act/fail/Dials: 10/0/10, Packets send/rec/dropped: 10/10/0
Clients act/fail/Dials: 10/0/20, Packets send/rec/dropped: 20/20/0
Clients act/fail/Dials: 10/0/30, Packets send/rec/dropped: 30/30/0
Clients act/fail/Dials: 10/0/40, Packets send/rec/dropped: 40/40/0
Clients act/fail/Dials: 10/0/50, Packets send/rec/dropped: 50/50/0
Clients act/fail/Dials: 10/0/60, Packets send/rec/dropped: 60/60/0
Clients act/fail/Dials: 10/0/70, Packets send/rec/dropped: 70/70/0
Clients act/fail/Dials: 10/0/80, Packets send/rec/dropped: 80/80/0
Clients act/fail/Dials: 10/0/90, Packets send/rec/dropped: 90/90/0
{
  "Started": "2019-04-06T08:18:09.44414217Z",
  "Duration": 9897737610,
  "Rate": 10,
  "Clients": 10,
  "Dials": 100,
  "FailedConnections": 0,
  "Sent": 100,
  "Received": 100,
  "Dropped": 0,
  "FailedConnects": 0,
  "Hosts": {
    "kahttp-deployment-ff8b6966-2cjw6": 26,
    "kahttp-deployment-ff8b6966-j45fw": 24,
    "kahttp-deployment-ff8b6966-plvqn": 26,
    "kahttp-deployment-ff8b6966-t867d": 24
  }
}

The number of Dials is equal to the number of Sent requests. This
means that no connection is kept alive.

Server host statistics

The example above uses the -host_stats to get server host
statistics. Server host statistics only works if the server is
kahttp. The kahttp server returns the hostname of the server in
the body and in the X-Kahttp-Server-Host: http header. The kahttp
client collect the statistics and presents the number of requests to
each host.

This functions is not usable for keep-alive testing (the main purpose
of kahttp) but may be used for instance to test a
canary
setup. When -host_stats is used you should most likely also disable
keep-alive with -disable_ka.

Server access logging

Access to the server can be logged (to stdout). This may be useful for
debugging and tests of low intensity traffic such as health probing.

NOTE: Extensive logging will affect performance.

Server access logging is initiated with the -log-access parameter or
the $LOG_ACCESS environment variable. The parameter specifies a
sub-path to be logged or “/“ to log everything.

$ ./image/kahttp -server -address :7000 -log-access /metrics
2020-07-11 13:02:39; localhost:7000/metrics from 127.0.0.1:43660
2020-07-11 13:02:39; localhost:7000/metrics from 127.0.0.1:43662
2020-07-11 13:02:39; localhost:7000/metrics from 127.0.0.1:43664
...

The log format is “date; from “.

Multiple source addresses

For testing of very many connections it is essential that the
connections originates from many sources. If just one (prominent)
source is used there will likely be a shortage of connection resources
since the src/dest addresses are always the same, only the ports can
be varied. The resource problem does not have to occur on the
endpoints but can be in network elements (NE) in between such as
load-balancers or NAT boxes.

To use many source addresses you must first assign a CIDR to the
loopback interface as described for
mconnect. Then
use;

kahttp -address http://127.0.0.1:5080/ -monitor -rate 400 -nclients 40 \
  -timeout 10s -scrcidr 222.222.222.0/24

Set source address

Some times the source address must be set for instance to use a
specific ip-family as described in #6.

The -scrcidr can be used for this purpose but with masks /32 and /128;

$ kahttp kahttp -address http://vm-001/cgi-bin/info -monitor -timeout 5s -srccidr 192.168.1.3/32
... (ipv4 used)
$ kahttp kahttp -address http://vm-001/cgi-bin/info -monitor -timeout 5s -srccidr 1000::1:c0a8:103/128
... (ipv6 used)

Https

Kahttp disables certificate verification because the server is most
likely using a self-signed certificate. There is currently no way of
enabling certificate verification in the kahttp client.

The kahttp server will expose a https server if both -https_key
and -https_cert are specified (or the corresponding environment
variables $KAHTTP_KEY and $KAHTTP_CERT are set). The address
(including the port) can be specified with the -https_addr option
(default “:5443”).

Implementation is based on this
description.

Example;

export KAHTTP_KEY=/tmp/server.key
export KAHTTP_CERT=/tmp/server.crt
openssl genrsa -out $KAHTTP_KEY 2048
openssl req -new -x509 -sha256 -key $KAHTTP_KEY -out $KAHTTP_CERT -days 3650
kahttp -server -address [::]:5080
# In another shell;
wget --no-check-certificate -qO- https://[::1]:5443/
# Or;
kahttp -address https://[::1]:5443/ -monitor -rate 400 -nclients 40 -timeout 10s | jq .

If you use another client, e.g. curl you can verify the self-signed
certificate. First the “Common name” used must match the request
url. Use the --resolv option, for example;

curl -v --cacert /tmp/server.crt --resolv kahttp.localdomain:5443:[::1] \
  https://kahttp.localdomain:5443/

Client certificate, mTLS

Kahttp accepts and verifies client certificates if given
(tls.VerifyClientCertIfGiven). Http requests without client
certificate or with a correct certificate are accepted but requests
with an invalid certificate are rejected.

curl -v --cacert /tmp/server.crt --resolv kahttp.localdomain:5443:[::1] \
  --cert /tmp/server.crt --key /tmp/server.key \
  https://kahttp.localdomain:5443/
...
* TLSv1.2 (OUT), TLS handshake, CERT verify (15):

The same certificate as for the server is presented as a client
certificate which is valid and the (faked) header
CERT verify indicates that a client certificate is used.

If an invalid client certificate is used the conneciton fails;

curl --cacert /tmp/server.crt --resolv kahttp.localdomain:5443:[::1] \
  --cert /tmp/kahttp.crt --key /tmp/kahttp.key \
  https://kahttp.localdomain:5443/
...
curl: (35) error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate

Http2

Curl uses http/2 by default, use the --http1.1 option to enforce
http1. Use the -http2 flag to make the kahttp client to use
http2. Http2 used on a clear-text connection (http) is converted to h2c.

curl --insecure --http1.1 --resolv kahttp.localdomain:5443:[::1] https://[::1]:5443/
kahttp -address https://[::1]:5443/ -monitor -http2 -rate 400 -nclients 40 -timeout 10s | jq .
# This will use h2c;
curl -D - --http2 http://[::1]:5443/
# This will not work;
kahttp -address http://[::1]:5443/ -http2

Use a k8s tls secret

First create a k8s tls secret from the crt and key files;

kubectl create secret tls kahttp-secret --key /cert/kahttp.key --cert /cert/kahttp.crt

In the kahttp manifest mount the secret and set the KAHTTP_KEY and
KAHTTP_CERT variables appropriately (see
kahttp-secret.yaml);

...
        env:
          - name: KAHTTP_KEY
            value: "/cert/tls.key"
          - name: KAHTTP_CERT
            value: "/cert/tls.crt"
        volumeMounts:
          - name: cert
            mountPath: "/cert"
            readOnly: true
      volumes:
        - name: cert
          secret:
            secretName: kahttp-secret