docklogbeat

DEPRECATED: Please use a more robust solution, such as fluentd on Kubernetes. This repository
is no longer maintained. It is archived as an example for future users.

docklogbeat is a tool which combines filebeat
and docker-gen to transport data
from Docker container logs to Logstash and Elasticsearch.

It is intended as a more reliable replacement for logspout.

The implementation was inspired by this blog post.

Instructions

Run the docklogbeat container on each physical host in your cluster. It will
watch for new Docker containers to be spawned. If a container has a specific
label, it will automatically begin shipping the logs to your Logstash or
Elasticsearch cluster.

An example:

$ docker run -d \
    -v /var/lib/docker:/var/lib/docker:ro \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -e LOGSTASH_HOSTS=logstash1.example.com:5044,logstash2.example.com:5044 \
    docklogbeat

Any containers with the label docklogbeat=true will have their Docker logs
sent to the configured Logstash / Elasticsearch cluster.

An example:

$ docker run -l docklogbeat=true ...

Limitations

The docker-gen tool takes about 2 seconds to notice new containers and
regenerate the filebeat configuration. If your containers are short lived,
they may not have their log output sent to your Logstash / Elasticsearch cluster.

This container will read all log messages from Log Producing Containers when it
starts, similar to docker logs --tail=all. If you need to restart docklogbeat
without producing duplicate messages, you must make sure to store
/var/lib/filebeat/registry on a persistent volume.

Rancher Integration

This container automatically detects if it is being run on a Rancher
cluster and sets the DOCKLOGBEAT_REGISTRY_FILE_NAME to a unique value based on the
hostname of the physical host running the container. This allows docklogbeat to be
restarted safely without producing duplicate messages.

Environment Variables

These environment variables can be used to configure the docklogbeat container.

• LOGSTASH_HOSTS - A comma-separated lists of the Elastic Beats servers which will receive logs.
• DOCKLOGBEAT_REGISTRY_FILE_NAME - File to store the filebeat registry (default: /var/lib/filebeat/registry).
• DOCKLOGBEAT_LABEL_KEY - Container Filter Label Key (default: docklogbeat).
• DOCKLOGBEAT_LABEL_VALUE - Container Filter Label Value (default: true).
• DOCKLOGBEAT_SCAN_FRECUENCY - Container Scan Logs Frecuency (default: 11s).

Labels for Log Producing Containers

• docklogbeat - must be set to true for docklogbeat to collect this container’s logs.
• docklogbeat_document_type - set the document_type field for this container (optional).