TA-netflow-logstash for Splunk

This CIM compliant TA can be used with Splunk Enterprise Security and
provides field extractions, aliases, and tags for Netflow v5, v9 and IPFIX data
that has been collected by Logstash.

This fills the Traffic Center panel under Security Domains -> Network in Splunk
Enterprise Security:

Installation

1. Install this TA on your Splunk (Enterprise Security) search head. Make sure to rename it TA-netflow-logstash otherwise ES won’t eat it.
2. Install Logstash on an instance that will receive netflow data from the various netflow probes
3. Install a Splunk Universal Forwarder on the Logstash instance
4. Disable any existing Splunk Add-on for Netflow (Splunk_TA_flowfix)

Configuration

1. Configure Logstash to output a .json file for the received netflow data, for example with the following config file:

    input {
      udp {
        port => 2055
        codec => netflow {
          versions => [5,9,10]
        }
      }
    }
    output {
      file {
        codec => "json_lines"
        path => "/var/log/netflow/netflow.json"
      }
    }

2. Have the Splunk Universal Forwarder index the netflow.json file, for example with the following Splunk inputs.conf:

    [monitor:///var/log/netflow/netflow.json]
    disabled = false
    sourcetype = netflow_raw
    index=netflow_raw

3. Create 2 Splunk indexes:

   • netflow_raw

     This index will temporarily hold the Netflow data, so you can keep it small at e.g. 1GB.

   • netflow

     This index will hold the flow objects built from the netflow_raw index, so size it for proper retention. This index will be filled through a scheduled Splunk search that runs every minute.

Netflow probes

Logstash has been tested with the following netflow probes:

CIM

The TA provides fields compatible with the Splunk Common Information Model (CIM):

• duration
• src_ip
• dest_ip
• src_port
• dest_port
• bytes_in
• pkts_in
• transport
• protocol_version
• icmp_type

Netflow fields

The original fields are also still available through the netflow. prefix, followed by their Netflow field names:

nprobe (v10 output)

• version
• sourceIPv4Address
• destinationIPv4Address
• ipNextHopIPv4Address
• sourceTransportPort
• destinationTransportPort
• tcpControlBits
• ingressInterface
• egressInterface
• packetDeltaCount
• octetDeltaCount
• flowStartMilliseconds
• flowEndMilliseconds
• protocolIdentifier
• ipClassOfService
• flowEndReason
• tcpOptions

ipt_NETFLOW (v10 output)

• version
• sourceIPv4Address
• destinationIPv4Address
• ipNextHopIPv4Address
• sourceTransportPort
• destinationTransportPort
• tcpControlBits
• ingressInterface
• egressInterface
• packetDeltaCount
• octetDeltaCount
• flowStartMilliseconds
• flowEndMilliseconds
• protocolIdentifier
• ipClassOfService
• flowEndReason

nprobe (v9 output)

• version
• flow_seq_num
• flowset_id
• in_bytes
• in_pkts
• protocol
• src_tos
• tcp_flags
• l4_src_port
• ipv4_src_addr
• src_mask
• input_snmp
• l4_dst_port
• ipv4_dst_addr
• dst_mask
• output_snmp
• ipv4_next_hop
• src_as
• dst_as
• last_switched
• first_switched

softflowd (v9 output)

• version
• flow_seq_num
• flowset_id
• ipv4_src_addr | ipv6_dst_addr
• 83.149.69.228
• ipv4_dst_addr | ipv6_dst_addr
• last_switched
• first_switched
• in_bytes
• in_pkts
• input_snmp
• output_snmp
• l4_src_port
• l4_dst_port
• protocol
• tcp_flags
• ip_protocol_version

fprobe (v5 output)

TODO

Contribute

Any feedback in the form of patches, feature requests, bug reports or just an email is most welcome.
If you want to provide patches, please do so through a Pull Request.
If you have any bug reports or feature requests, please submit them to the issue tracker.