项目作者: CIRCL

项目描述 :
pcapdj - dispatch pcap files
高级语言: C
项目地址: git://github.com/CIRCL/pcapdj.git
创建时间: 2013-07-08T09:01:05Z
项目社区:https://github.com/CIRCL/pcapdj

开源协议:GNU Affero General Public License v3.0

下载


pcapdj

pcapdj - dispatch pcap files

Network captures often result in very large files. Therefore, tools like
tcpdump or dumpcap offer features of file rotation either after a fixed
size or a fixed amount of time.

When these files are analyzed focusing on stateful protocols such as TCP,
TCP sessions could have been established in one pcap file and continue in
the next pcap files. When these TCP sessions have to be properly reassembled,
then either the TCP reassembly tool has to support multiple pcap files as
input or the pcap files have to merged in a single file using for instance a
tool such as editcap. However, in this case, very large files are the results,
that were tried to be avoided with the file rotation.

PCAPDJ processes a list of pcap files and write each individual packet in a
named pipe. A second process reads these individual packets and does some
processing. A third process, does some cleanup operations and controls pcapdj.

Each pcap file that is processed by pcapdj must be authorized by a third process.
When a pcap file is not acknowledged the file descriptor to the named pipe
is not closed and hence the other program processing pcap data does not end and
keeps its internal states. Before a next file is authorized, other tasks could be done,
such as removing duplicated files or already processed pcap files in order to free disk space.

Building PCAPDJ

Dependencies

In an Ubuntu 16.04 Operating system the following packages must be installed.

apt-get install libwsutil-dev libwiretap-dev libhiredis-dev libglib2.0-dev

Compiling

unzip pcapdj-master.zip

cd pcapdj-master

make

Use case with Suricata

The Suricata IDS can be configured to extract all HTTP payloads [1].
This feature is used in the following example in conjunction with pcapdj
feeding suricata with multiple pcap files.

Import the pcap files that should be processed by pcapdj

  1. #!/usr/bin/python
  2. import redis
  3. import os
  4. root="mypcapfiledir"
  5. red = redis.Redis()
  6. files = os.listdir(root)
  7. files.sort()
  8. for rf in files:
  9. f = root + "/"+rf
  10. if f.endswith('pcap') == True:
  11. # Compressed files are supported too
  12. # if f.endswith('pcap.gz') == True:
  13. red.rpush("PCAPDJ_IN_QUEUE",f)

Create a name pipe that is shared between pcapdj and suricata

  1. mkfifo /tmp/pcapbuffer

Launch pcapdj

  1. ./pcapdj -b /tmp/pcapbuffer
  2. redis_server = 127.0.0.1
  3. redis_port = 6379
  4. named pipe = /tmp/pcapbuffer
  5. Waiting for other peer (IDS, tcp-reassembly engine, etc)...
  6. PCAPDJ waits for the consumer of the fifo bufer. In this case suricata.

Launch suricata

  1. suricata -r /tmp/pcapbuffer

Until now no packets are put in the buffer because pcapdj needs an
authorization. PCAPDJ says that it is ready to process the pcapfile 1.pcap and
that it waits for this authorization. For doing so, pcapdj puts the next file
it wants to process in a queue called PCAPDJ_NEXT and it searches for the given
filename in the PCAPDJ_AUTH set. This way several pcadj processes can be managed
by the same authorization script.

  1. [INFO] Next file to process /tmp/testpcaps/1.pcap
  2. [INFO] Waiting authorization to process file /tmp/testpcaps/2.pcap

Launch the controller script that authorizes each pcap file that is put
in the named pipe.

  1. while True:
  2. #Check if some data is ready to be process
  3. pcapname = red.lpop("PCAPDJ_NEXT")
  4. if pcapname != None:
  5. print "Authorized file ",pcapname
  6. red.sadd("PCAPDJ_AUTH", pcapname)

Wait until pcapdj and suricata are done

Additional Features

On the experimental branch two new features were implemented based on
a signal handler.

Suspending PCAPDJ

If PCAPDJ is processing a large file and the resources are at the point of
being exhausted, the command kill -SIGUSR1 can be executed.
PCAPDJ stops feeding the fifo buffer and resources can be manually freed
without terminating the consumer program.

Once, the machine is cleaned up, PCAPDJ can be resumed by sending a second
time the SIGUSR1 signal.

Display Accounting Data

When PCAPDJ is running for a while, it might be interesting to determine
what is happening. The signal SIGUSR2 can be sent to PCAPDJ. PCAPDJ
shows then following information on standard output.

  • A timestamp when PCAPDJ started
  • The number of seconds elapsed since PCAPDJ started
  • The internal state of PCAPDJ
  • The number of times PCAPDJ has been suspended
  • The number of files PCAPDJ processed
  • The number of packets PCAPDJ processed
  • The sum of the cap length fields
  • The sum of the length fields. If the sum of lengths is different from
    the sum of cap lengths then the capture is incomplete.

An example is shown below:

  1. [STATS] Start time:2013-06-09 09:17:50
  2. [STATS] Uptime:322 (seconds)
  3. [STATS] Internal state:Waiting for authorization
  4. [STATS] Number of suspensions:0
  5. [STATS] Number of files:1
  6. [STATS] Number of packets:2968
  7. [STATS] Number of cap_lengths:330581
  8. [STATS] Number of lengths:330581

References

[1] http://blog.inliniac.net/2011/11/29/file-extraction-in-suricata/