Directionality of "in" and "out" flows

[candlerb]

In a standard unidirectional netflow, for traffic from "src" to "dst", the counters are stored in

                table->packets
                table->bytes

Then to support bi-directional flows, there are these extensions for "dst" to "src":

                table->out_packets // Extension EX_OUT_PKG_4/8
                table->out_bytes   // Extension EX_OUT_BYTES_4/8 

Therefore "src" to "dst" is considered as "in", and "dst" to "src" is considered as "out". I believe this is the opposite to how most people would think of it.

This becomes confusing when dealing with devices which do actually generate bi-directional flows.

• In the case of Cisco ASA NSEL, it gives NF_F_FWD_FLOW_DELTA_BYTES and NF_F_REV_FLOW_DELTA_BYTES. FWD refers to the "src" to "dst" direction, and REV to the "dst" to "src" direction. But these are stored as "in" and "out" respectively by nfdump.

  The result is: when you are looking at flows for (e.g.) a user connecting to a webserver, and looking separately at "ibytes" and "obytes", the outbound request is counted as "ibytes" and the downloaded traffic from webserver to user is counted as "obytes".

• Netflow version 9 has NF9_IN_BYTES and NF9_OUT_BYTES. I don't have a source to test these with, but I would expect "OUT" refers to the src to dst direction. (Need to check whether unidirectional flows set NF9_OUT_BYTES only)

To make this consistent, and assuming people would be happier with src to dst being "out" rather than "in" I have some suggestions:

1. Rather than swapping "in" and "out" throughout nfdump, maybe rename them to "fwd" and "rev" respectively. The default, for unidirectional flows, is "fwd".
2. Then change the aggregate query types like "ibytes" and "obytes" to use "rev" and "fwd" respectively. This would be change of behaviour (results swapped around compared to before); however people who use unidirectional flows and who query just "bytes" would be unaffected, now that Aggregate using in+out bytes for bidirectional flows #28 has been merged.
3. Check that when you use nfdump itself to synthesise bidirectional flows, it gets fwd and rev the right way round (I suspect it does already)
4. Store NF9_IN_BYTES as "rev" and NF9_OUT_BYTES as "fwd". This is unfortunately a swap and means that historical data will not be compatible. I don't see a way to avoid that.

[candlerb]

See also discussion on nfdump-users

[noa-appletech]

I am having the same problem with exported ASA flows.

[philhagen]

I agree that this is unclear. Rather than "in" and "out, or "fwd" and "rev", I propose these be named "src" and "dst". This would match the originator IP address of each byte count. For example:
Src IP: 1.2.3.4
Dst IP: 9.8.7.6
Src Bytes: 1,024,412 (i.e. bytes sent from 1.2.3.4 to 9.8.7.6)
Dst Bytes: 105 (i.e. bytes sent from 9.8.7.6 to 1.2.3.4)

The default in this model is "src bytes", which is consistent with unidirectional flows.

[candlerb]

Interesting.

Thinking about this, you could say the underlying problem is with the labels "src" and "dst" themselves. These very terms imply that packets are travelling from source to destination! Unfortunately, I don't think we are going to be able to rename them to something generic like "A" and "B". The terms src and dst are baked into the netflow protocols; and in any case the 90% use case is for unidirectional flows, which are from src to dst.

Now, my initial reaction was that it's clearer that "fwd" means packets from source to destination (the natural flow when talking about a "source" and a "destination"), and "rev" means packets in the reverse direction, i.e. destination to source - it's the reverse of what the terms "source" and "destination" imply.

Thinking about NSEL-type bidirectional flows, the "src" is actually the side which initiated the UDP/TCP exchange, and "dst" is the side which answered it. Let's think how would this work in practice. Suppose a client on the outside made an inbound HTTP connection through my firewall to an internal webserver. The http request would be counted as "src bytes" and the response as "dst bytes". I'm not sure it's immediately clear that "dst bytes" means the bytes downloaded by the client; it means bytes generated by dst, not bytes received at the dst.

Perhaps some other pair of labels can be found which implies this asymmetry. "request bytes" and "response bytes"? "originate bytes" and "answer bytes"? "up" and "down" ??

I agree that the terms "in" / "out" would be too confusing: if there's someone on the outside of firewall who originates a connection into your network, the first packet is "out" from their perspective but "inbound" from ours. I'm happy to rule that out.

Aside: I am guessing that when nfdump synthesises bidirectional flows, it's the first packet of the flow which determines which end is "src" and which is "dst"?

[candlerb]

Perhaps some other pair of labels can be found which implies this asymmetry

"send" and "receive"?

[philhagen]

I'm not sure "send" and "receive" would work, though - who is sending and who is receiving? Since the "src" is a characteristic assigned by either the original NetFlow/NSEL/etc or nfdump, the output UI can address this. For example:

$ nfdump -N -B -R netflow/ -o 'fmt:%ts %te %sap %dap %pr %ibyt %obyt'
Date first seen         Date last seen               Src IP Addr:Port       Dst IP Addr:Port  Proto  In Byte Out Byte
2013-05-07 14:31:18.094 2013-05-07 14:32:07.990      10.3.59.157:55464     65.39.178.58:80    6         9220   277204
2013-05-07 14:31:18.401 2013-05-07 14:32:36.517      10.3.59.157:55465     65.39.178.58:80    6         2154    37201
2013-05-07 14:31:19.117 2013-05-07 14:32:36.518      10.3.59.157:55466     65.39.178.58:80    6         2953    40850

and

$ nfdump -N -b -R netflow/ -o 'fmt:%ts %te %sap %dap %pr %ibyt %obyt'
Date first seen         Date last seen               Src IP Addr:Port       Dst IP Addr:Port  Proto  In Byte Out Byte
2013-05-07 14:31:18.094 2013-05-07 14:32:07.990     65.39.178.58:80         10.3.59.157:55464 6       277204     9220
2013-05-07 14:31:18.401 2013-05-07 14:32:36.517     65.39.178.58:80         10.3.59.157:55465 6        37201     2154
2013-05-07 14:31:19.117 2013-05-07 14:32:36.518     65.39.178.58:80         10.3.59.157:55466 6        40850     2953
2013-05-07 14:31:19.118 2013-05-07 14:32:36.518     65.39.178.58:80         10.3.59.157:55467 6        17242     1377
2013-05-07 14:31:19.152 2013-05-07 14:32:36.517     65.39.178.58:80         10.3.59.157:55468 6        68382     4778
2013-05-07 14:31:19.164 2013-05-07 14:31:24.909     65.39.178.58:80         10.3.59.157:55469 6           92      172

The In Byte and Out Byte fields are consistent with their respective Src IP and Dst IP values and track with them when the so-called "Source" and "Destination" flip. However, it's their terminology that is problematic. In both cases, though, the "In Byte" refers to data sent by the "Src IP" and the "Out Byte" refers to data sent by the "Dst IP". This is counterintuitive. "Input" data is not typically associated with a "Source" host and vice versa.

This should not be a problem for NSEL-style flows, as the "src" and "dst" are explicitly assigned rather than implicitly assumed as with unidirectional flows represented with bidirectional aggregation.

[candlerb]

I'm not sure "send" and "receive" would work, though - who is sending and who is receiving?

Yeah, you're right - we expect a source to be sending, and a destination to be receiving.

I still think forward and reverse or return are unambiguous when using src/dst terminology. Or how about "s2d" and "d2s" to be completely explicit? This would give for example:

$ nfdump -N -b -R netflow/ -o 'fmt:%ts %te %sap %dap %pr %ibyt %obyt'
Date first seen         Date last seen               Src IP Addr:Port       Dst IP Addr:Port  Proto  s2d Byte d2s Byte
2013-05-07 14:31:18.094 2013-05-07 14:32:07.990     65.39.178.58:80          10.3.59.157:55464 6       277204     9220
2013-05-07 14:31:18.401 2013-05-07 14:32:36.517     65.39.178.58:80          10.3.59.157:55465 6        37201     2154
2013-05-07 14:31:19.117 2013-05-07 14:32:36.518     65.39.178.58:80          10.3.59.157:55466 6        40850     2953
2013-05-07 14:31:19.118 2013-05-07 14:32:36.518     65.39.178.58:80          10.3.59.157:55467 6        17242     1377
2013-05-07 14:31:19.152 2013-05-07 14:32:36.517     65.39.178.58:80          10.3.59.157:55468 6        68382     4778
2013-05-07 14:31:19.164 2013-05-07 14:31:24.909     65.39.178.58:80          10.3.59.157:55469 6           92      172

I certainly agree that calling from src to dst "In" is counter-intuitive - that's what I was saying when I opened the ticket in the first place.

As for -b and -B behaviour: indeed it cannot always work, and it's also right that if src/dst change places then the counters should change places too.

But we still need a suitable label for "src to dst" and "dst to src", which is better than "in" and "out" respectively.

[philhagen]

I could see "s2d" and "d2s" - that's entirely unambiguous. Good solution!

And yes, as long as the directionality is swapped to match what's represented as "src" or "dst", we're good to go. Would really like to see this reflected in a future version!

[phaag]

sorry for being late ...

The story is a longer one. Back in the good old days, flows were uni-directional and therefore it was clear the the number of packets/bytes could not mistakenly mixed up. The history of nfcapd/nfdump was focused on these unid-dir netflow data and the CISCO definitions. The respective tags are:

According the CISCO definition:
IN_BYTES: Incoming counter with length N x 8 bits for number of bytes associated with an IP Flow. Default: 4.
and the same for IN_PKTS respectively.

According to these definitions and the following versions of CISCO, nfdump stores IN_BYTES in the respective structures internally and names them respectively in the output. This did not confuse as long as also v9 was uni-dir only. The question is, what is out and what is in. In the CISCO perspective it is in the view of the interface, where netflow is enabled and collected - this mostly matches with the view of the destination IP in many situations.

With NSEL/ASA these got a new dimension:
Some NSEL flows have/had no time stamps and byte/packet counters, as these where not really flows but events. Then e.g. 8.1 introduced:
NF_F_FLOW_BYTES: The total number of bytes of the flow

For historical reason and to map these counters in netflow structures these values are internally mapped to in_bytes as it was with IN_BYTES in v9. With 9.1 we have a mixture of flows and events and the definition is:
NF_F_FWD_FLOW_DELTA_BYTES The delta number of bytes from source to destination.

And here I agree, it is clear, that it is no longer the perspective from the destination but by definition from the source. As such it is a nfcapd bug for ASA 9.1 which incorrectly collects these counters.

So, how to fix this
In general, I can simple remap the counters but then I introduce an inconsistency with previous versions and collected data. At least this will result in a major version change, which affects NSEL/ASA users. If I leave it like it is, and introduce s2d and d2s tags - which can be done anyway, we willingly keep the bug for NSEL/ASA, which is not really nice too.
For standard netflow v9, we are still on the merci of the CISCO view what is in and out which is an interface view from the exporter and most of the time identical with the destination IP view.

As I have no ASA/NSEL device neither for testing nor for development and operational work, I rely on user input, how I should proceed here.

I would prefer to remap _NF_F_FWD_FLOW_DELTA_BYTES internally, add s2d and d2s tokens to make it more clear for NSEL/ASA and increase the major version to 1.7. The question is, should s2d and d2s also be available for standard v9 flows, as it is still in interface perspective and not clearly defined. The pacth would not affect standard v9 anyway.

Comments and proposals are welcome.

[philhagen]

Thanks for the detailed explanation! Very helpful in understanding the nuances of these terms.

I do not have an ASA/NSEL device either, but I may be able to get one. I will ask around and let you know if I am successful.

With regard to applying s2d and d2s, I think those tokens should be straightforward for any bidirectional flow (explicit or calculated/derived). I would have to see it in practice to say if that terminology is useful and valid or not, but my gut feeling is that it will be.

Would the s2d and d2s tokens be used in addition to the existing token values or in place of them?

[candlerb]

@phaag:

In the CISCO perspective it is in the view of the interface, where netflow is enabled and collected

Aha, I had forgotten that. It makes a lot of sense now as to why "src to dest" is counted as "in".

This raises another question. In modern versions of IOS, you can enable netflow in both directions on the same interface. Would one set of flows have NF9_IN_BYTES and the other set NF9_OUT_BYTES?

I believe these are unidirectional flows, and I thought it was just a "bytes" counter, plus a flag which indicates in/out (since you can write filters in nfdump like [in|out] if <num>)

Now, if both in and out are unidirectional flows, then of course the traffic would be "src" to "dest" by definition. As long as it's all written as NF9_IN_BYTES then there would be no problem treating this as "s2d".

I guess it would be possible to set something up with dynamips to test this.

@philhagen:

Would the s2d and d2s tokens be used in addition to the existing token values or in place of them?

My gut feeling is that they should replace them. It seems silly to have 4 different sets of counters. "s2d" would be the new name for "in", and "d2s" would be the new name for "out".

This leaves the issue of pre-existing NSEL records being the wrong way round. Personally, I would be happy to take that hit; I care more that it is right going forward, than the quality of the historical data. But it's not my call.

I don't know if you can identify the source as NSEL within the record, but bumping the version number would probably be good enough for anyone who cares. That is: if you see a bidirectional flow, and you happen to know it was collected from NSEL, and it has the old version number, then you know that s2d/d2s are the wrong way round.

[candlerb]

(Aside: I was wrong about in/out being a flag, there are separate fields for inif and outif)

As I had the dynamips bits more or less ready, I decided to run a test. This is a 7200 with IOS 15.1

flow exporter EXPORTER-1
 description Export to pc1
 destination 10.10.1.1
 template data timeout 60
!
flow monitor FLOW-MONITOR-V4
 record netflow ipv4 original-input
 exporter EXPORTER-1
 cache timeout active 300
!
interface FastEthernet0/0
 ip flow monitor FLOW-MONITOR-V4 input
 ip flow monitor FLOW-MONITOR-V4 output

This is Netflow v9 by default:

rtr1#sh flow exporter EXPORTER-1
Flow Exporter EXPORTER-1:
  Description:              Export to pc1
  Export protocol:          NetFlow Version 9
  Transport Configuration:
    Destination IP address: 10.10.1.1
    Source IP address:      10.10.1.254
    Transport Protocol:     UDP
    Destination Port:       9995
    Source Port:            59799
    DSCP:                   0x0
    TTL:                    255
    Output Features:        Not Used

I now have both the nfdump files, and raw pcap files taken at the same time. Looking at some short HTTPS sessions:

# nfdump -M /var/nfsen/profiles-data/live/upstream1  -T  -r 2016/12/02/nfcapd.201612022055 -c 100 'host 147.28.0.62'
Date first seen          Duration Proto      Src IP Addr:Port          Dst IP Addr:Port   Packets    Bytes Flows
2016-12-02 20:55:23.812     1.028 TCP          10.10.1.1:55130 ->      147.28.0.62:443         13     1204     1
2016-12-02 20:55:23.960     1.040 TCP        147.28.0.62:443   ->        10.10.1.1:55130       14     9904     1
2016-12-02 20:55:24.996     0.016 TCP          10.10.1.1:55130 ->      147.28.0.62:443          3      120     1
2016-12-02 20:59:01.092     1.016 TCP          10.10.1.1:55140 ->      147.28.0.62:443         11     1127     1
2016-12-02 20:59:01.228     1.044 TCP        147.28.0.62:443   ->        10.10.1.1:55140       11     5278     1
2016-12-02 20:59:02.256     0.024 TCP          10.10.1.1:55140 ->      147.28.0.62:443          3      120     1
2016-12-02 20:59:12.304     0.708 TCP          10.10.1.1:55142 ->      147.28.0.62:443          9     1122     1
2016-12-02 20:59:12.448     0.704 TCP        147.28.0.62:443   ->        10.10.1.1:55142       10     4752     1
2016-12-02 20:59:13.172     0.000 TCP          10.10.1.1:55142 ->      147.28.0.62:443          2       80     1
2016-12-02 20:59:34.868     1.024 TCP          10.10.1.1:55146 ->      147.28.0.62:443         33     2377     1
2016-12-02 20:59:35.020     1.020 TCP        147.28.0.62:443   ->        10.10.1.1:55146       57    74425     1
2016-12-02 20:59:36.056     0.000 TCP          10.10.1.1:55146 ->      147.28.0.62:443          2       80     1
2016-12-02 20:59:40.408     1.016 TCP          10.10.1.1:55148 ->      147.28.0.62:443         33     2377     1
2016-12-02 20:59:40.560     1.004 TCP        147.28.0.62:443   ->        10.10.1.1:55148       57    74425     1
2016-12-02 20:59:41.588     0.000 TCP          10.10.1.1:55148 ->      147.28.0.62:443          2       80     1
Summary: total flows: 15, total bytes: 177471, total packets: 260, avg bps: 5507, avg pps: 1, avg bpp: 682
Time window: 2016-12-02 20:51:14 - 2016-12-02 20:59:41
Total flows processed: 65, Blocks skipped: 0, Bytes read: 4524
Sys: 0.004s flows/second: 16250.0    Wall: 0.000s flows/second: 66666.7

And decoding the packets with tshark:

# tshark -r /tmp/netflow.pcap -nnV -d udp.port==9995,cflow
...
        Flow 3
            SrcAddr: 10.10.1.1
            DstAddr: 147.28.0.62
            SamplerID: 0
            InputInt: 2
            SrcPort: 55140
            DstPort: 443
            IP ToS: 0x00
            Protocol: TCP (6)
            SrcMask: 24
            DstMask: 0
            TCP Flags: 0x1b
            SrcAS: 0
            DstAS: 0
            NextHop: 10.10.0.254
            Octets: 1127
            Packets: 11
            [Duration: 1.016000000 seconds (switched)]
                StartTime: 1683.140000000 seconds
                EndTime: 1684.156000000 seconds
            OutputInt: 1
        Flow 4
            SrcAddr: 147.28.0.62
            DstAddr: 10.10.1.1
            SamplerID: 0
            InputInt: 1
            SrcPort: 443
            DstPort: 55140
            IP ToS: 0x80
            Protocol: TCP (6)
            SrcMask: 0
            DstMask: 24
            TCP Flags: 0x1b
            SrcAS: 0
            DstAS: 0
            NextHop: 10.10.1.1
            Octets: 5278
            Packets: 11
            [Duration: 1.044000000 seconds (switched)]
                StartTime: 1683.276000000 seconds
                EndTime: 1684.320000000 seconds
            OutputInt: 2
...

OK, that's as expected:

• Flows are unidirectional
• There are just "Octets" and "Packets" counters, which I think must be stored as IN_XXX by nfdump (i.e. "s2d" in the proposed new terminology)
• I'm capturing on interface 1 only, but both in and out. So some flows have InputInt: 1 and OutputInt: 2, and some are the other way round

[faisalts]

i have question regarding the bidirectional i would like to have the in / out pps,pbs,bpp and in/out duration
because i aggragted the dtat by -b in nfdump and i just got in/out packts, bytes
could any one help please

[candlerb]

Sorry I don't understand what you're saying.

Can you show:

• The nfdump command you typed
• The output you got
• What output you wanted instead

[faisalts]

this is the command nfdump -R /var/cache/nfdump/nfcapd.201701312347:nfcapd.201702010022 -o "fmt:%ts %td %pr %sap -> %dap %flg %ipkt %opkt %ibyt %obyt %fl " -b | more

Date first seen Duration Proto Src IP Addr:Port Dst IP Addr:Port Flags In Pkt Out Pkt In Byte Out Byte Flows
2017-01-31 23:43:44.419 0.001 UDP 10.0.2.15:45567 -> 141.163.201.222:53 ...... 1 1 59 110 2
2017-02-01 00:14:41.020 0.002 UDP 10.0.2.15:28923 -> 141.163.201.221:53 ...... 1 1 59 110 2
2017-01-31 23:42:28.941 0.001 UDP 10.0.2.15:13984 -> 141.163.201.222:53 ...... 1 1 61 77 2
2017-01-31 23:42:39.939 0.000 UDP 10.0.2.15:51353 -> 141.163.201.222:53 ...... 1 1 58 103 2
2017-02-01 00:06:14.159 0.001 UDP 10.0.2.15:48540 -> 141.163.201.222:53 ...... 1 1 75 131 2
2017-02-01 00:05:15.814 0.001 UDP 10.0.2.15:52237 -> 141.163.201.222:53 ...... 1 1 60 76 2
2017-01-31 23:42:37.100 0.002 UDP 10.0.2.15:40806 -> 141.163.201.222:53 ...... 1 1 64 101 2
2017-02-01 00:13:36.523 0.001 UDP 10.0.2.15:53783 -> 141.163.201.222:53 ...... 1 1 70 120 2
2017-01-31 23:47:58.204 741.499 UDP 10.0.2.15:48096 -> 141.163.201.222:53 ...... 2 2 146 281 4
2017-02-01 00:13:47.254 0.001 UDP 10.0.2.15:1082 -> 141.163.201.222:53 ...... 1 1 79 125 2
2017-02-01 00:06:21.946 0.001 UDP 10.0.2.15:18808 -> 141.163.201.222:53 ...... 1 1 61 123 2
2017-01-31 23:52:34.181 0.027 UDP 10.0.2.15:64704 -> 141.163.201.222:53 ...... 1 1 78 123 2
2017-02-01 00:13:39.228 0.001 UDP 10.0.2.15:12241 -> 141.163.201.222:53 ...... 1 1 71 142 2
2017-02-01 00:09:24.215 0.002 UDP 10.0.2.15:58853 -> 141.163.246.124:53 ...... 1 1 78 123 2
2017-01-31 23:48:08.784 0.001 UDP 10.0.2.15:34507 -> 141.163.201.222:53 ...... 1 1 71 130 2
2017-01-31 23:49:10.647 0.001 UDP 10.0.2.15:19681 -> 141.163.201.222:53 ...... 1 1 61 123 2
2017-01-31 23:48:07.035 0.001 UDP 10.0.2.15:18082 -> 141.163.201.222:53 ...... 1 1 70 120 2
2017-02-01 00:18:26.056 0.002 UDP 10.0.2.15:52711 -> 141.163.201.221:53 ...... 1 1 59 122 2
2017-01-31 23:44:36.969 0.001 UDP 10.0.2.15:36373 -> 141.163.201.222:53 ...... 1 1 57 114 2
2017-01-31 23:42:30.493 2303.402 TCP 10.0.2.15:53692 -> 216.58.213.78:443 .AP.SF 1588 2109 400493 3.2 M 2
2017-01-31 23:42:28.945 0.001 UDP 10.0.2.15:1218 -> 141.163.201.222:53 ...... 1 1 71 116 2
2017-01-31 23:47:08.729 0.003 UDP 10.0.2.15:12922 -> 141.163.201.222:53 ...... 1 1 59 110 2

--now for bidirectional format i have (inpackts,outpackts,in bytes,outbytes) this is not enogh for me i would like to have(inpps,outpps,inpbs,outpbs,inbpp,outbpp,induration,outduration)
--and also i would like to convert the value e.g. 2.3 M to 2300000 for calculation
Thank you

[candlerb]

I think extra formats like %ibps would have to be added, but you can calculate these yourself e.g. from %ibyt and %td

Try option -N to get plain numbers without converting to 2.3 M.

(In future, support questions would probably be best submitted via the mailing list, rather than added to an existing ticket)

[faisalts]

thank you
i have tried %ibps but it did not work i will try to calculated . thank you -N is working perfectly and converted to plain numbers

./nfdump -b -w - | nfdump -R /var/cache/nfdump/nfcapd.201701312347:nfcapd.201702010022 -o "fmt:%ts %td %pr %sap -> %dap %flg %ipkt %opkt %ibyt %obyt %obps %pps %bpp %fl " -b | more
Output format parse error at: %obps %pps %bpp %fl
-bash: ./nfdump: No such file or directory

you can see the error when i use %obps

[candlerb]

Like I said, there are various extra formats like %ibps, %obps which are not implemented. For the full list see here

Calculate %obps by dividing %obyt by %td (and multiply by 8 for bits per second rather than bytes per second)

[faisalts]

please is there any way by NFDUMP to convert a time from ISO 8601 format to Unix(epoch) format
if any one could help
Thanks

[candlerb]

Please don't ask questions on tickets. Support requests should go to the nfdump-users mailing list.

[aksppc]

Hi all!
Can someone help me with swaping in and out packets/bytes in nfdump sources?
Where and what i need to edit?

[phaag]

I think the original question of this thread should be answered. Please do do ask not related questions to issues for other topics. Please open a new issue?
Thanks