Explain the output in the manual page. What is iperf3 _doing_, and what is it _saying_?

[thx1111]

iperf3 3.1.3-1
Arch Linux

It would be really polite to actually describe the program output in the manual page. For instance:
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 4.04 MBytes 3.39 Mbits/sec 0 sender
[ 4] 0.00-10.00 sec 2.28 MBytes 1.91 Mbits/sec receiver

"sender" and "receiver" - who do you suppose they are? Perhaps one is the client and the other is the server? But which is which? Or, do "sender" and "receiver" refer to "sending" and "receiving", which would have a different meaning entirely? But then still, from what point of view? ... It's just not that hard to say something like "server sending" and "server receiving", or "client sending" and "client receiving". Or, is the server sending at a different rate than the client can receive? Perhaps the client can send faster than the server can receive? Or does iperf3 keep track of that? Or were there MTU issues in the channel? Does iperf3 check for that before running the test?

BTW - "Mbits/sec" is not a "Bandwidth", but a "Channel Capacity" or "Communication Rate". The "Bandwidth" would refer to something like "20MHz" or "40MHz", which would be interesting to notice when measuring the Communication Rate of a WiFi network interface.

[yonirotvel]

I agree with it all.

[bltierney]

anyone want to do a 1st draft of this for us?

[eriitguy]

iperf 3.x - Understanding test results (Draft)

1. Description
2. Run tests
3. Interpreting the results

Description

iperf tool can be used to measure network bandwidth capacity

Run tests

iperf 3.x can be used in two modes:

1. Server mode
2. Client mode

Usage:

iperf [-s|-c host] [options]
#
iperf [-h|--help] [-v|--version]

Server mode, can be started using -s or --server parameters:

iperf3 -s

or

iperf3 --server

I that mode iperf listening on the default port 5201 for connections.

Custom port for listening can be specified using -p or --port parameters:

iperf3 -s -p 5002

After server is started we should use iperf in client mode, in order to connect to it and verify connection speed.

Client mode can be started using -c or --client parameters, it also require a <host> to which it should connect, this can be an IP or hostname:

iperf3 -c ping.online.net

iperf3 -c 192.168.0.1

If iperf server is running on non-default port, we can specify it during startup, using -p or --port parameters:

iperf3 -c ping.online.net -p 5202

We should be aware that, when the client connect to the server it send the data and in such case, by default Upload speed is measured.
If we want to measure Download speed, we can start iperf in client and reverse mode, by using -R or --reverse parameters:

iperf3 -c ping.online.net -p 5202 -R

Interpreting the results

Result we get should be interpreted based on the:

1. iperf client mode
2. results from the iperf client or server side

iperf client mode

As mentioned above, by default iperf in client mode sending the data to the server. In such case results we got during the test:

[  4]   0.00-10.00  sec   466 MBytes   391 Mbits/sec                  sender
[  4]   0.00-10.00  sec   466 MBytes   391 Mbits/sec                  receiver

Should be interpreted from the following points:
sender - is iperf client, Upload speed from iperf client to iperf server is measured
receiver - is iperf server, Download speed on iperf server from iperf client is measured

If you start iperf client in reverse mode, iperf server will send data and iperf client will receive. In such case results we got during the test:

[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec   569 MBytes   478 Mbits/sec                  sender
[  4]   0.00-10.00  sec   569 MBytes   478 Mbits/sec                  receiver

Should be interpreted from the following points:
sender - is iperf server, Upload speed from iperf server to iperf client is measured
receiver - is iperf client, Download speed on iperf client from iperf server is measured

results from the iperf client or server side

Default mode test:

iperf3 -s
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-10.00  sec  0.00 Bytes  0.00 bits/sec                  sender
[  5]   0.00-10.00  sec   466 MBytes   391 Mbits/sec                  receiver

iperf3 -c 192.168.0.1
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec   466 MBytes   391 Mbits/sec                  sender
[  4]   0.00-10.00  sec   466 MBytes   391 Mbits/sec                  receiver

During this test iperf client was a sender and in the results we can see:
From client logs:
sender - is iperf client, Upload speed from iperf client to iperf server is measured
receiver - is iperf server, Download speed on iperf server from iperf client is measured
From server logs:
sender - is iperf client, Upload speed from iperf client to iperf server is measured
receiver - is iperf server, Download speed on iperf server from iperf client is measured

Reverse mode test:

iperf3 -s
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-10.01  sec   569 MBytes   477 Mbits/sec                  sender
[  5]   0.00-10.01  sec  0.00 Bytes  0.00 bits/sec                  receiver

iperf3 -c 192.168.0.1 -R
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec   569 MBytes   478 Mbits/sec                  sender
[  4]   0.00-10.00  sec   569 MBytes   478 Mbits/sec                  receiver

During this test iperf server was a sender and in the results we can see:
From client logs:
sender - is iperf server, Upload speed from iperf server to iperf client is measured
receiver - is iperf client, Download speed on iperf client from iperf server is measured
From server logs:
sender - is iperf server, Upload speed from iperf server to iperf client is measured
receiver - is iperf client, Download speed on iperf client from iperf server is measured

Any comments, tips and notes are welcome.

Thank you!

[bmah888]

Thanks @eriitguy ! I only skimmed this so far, need to make another pass through it. I also need to figure out where's the best place to put this information (e.g. README.md, manual page, Web site, or some combination of the three). (I know the OP asked for it in the manual page, but I want to really consider where's the best place to put this.)

[bmah888]

@eriitguy: A belated thank-you. I edited your text and shortened it a bit to fit into the context of a manual page (which is really intended to be more of a quick-reference to a program). That's what I committed yestserday. Thanks again!

[eriitguy]

@bmah888, thank you for possibility to contribute!
This was just an attempt of explanation for my colleagues if we correctly understanding results.

[CMCDragonkai]

From the client mode, the sender line I understand represents the "upload bandwidth". But the receiver line, if it means the "download bandwidth" from the server's perspective, then what would be causing them to have different values for bandwidth?

[catanon]

As @eriitguy wrote:

results from the iperf client or server side
Default mode test:

iperf3 -s
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-10.00  sec  0.00 Bytes  0.00 bits/sec                  sender

From server logs:
sender - is iperf client, Upload speed from iperf client to iperf server is measured

I'm not sure if understand. How can the transfer/bandwidth be 0 if it is the upload speed of iperf client?

[condamoor-sap]

why don't we add the ip address of the server and client in the corresponding output rows so it is clear who is sending and receiving..

[SarenT]

and what is Retr? I can't seem to find what it means.

[groda]

and what is Retr? I can't seem to find what it means.

Found an answer here: what "Retr" field mean, apparently retr stands for retrasmission, and the Retr column shows the number of TCP segments retransmitted.
A high retransmission number signals some issue in the network (congestion or corruption).

[joaopedrolourencoaffonso]

Why the value at server log and client log are different? Shouldn't they be equal?

[namett]

i have a same question. Why the senders's transfer is not equal receiver's transfer
[ ID] Interval Transfer Bandwidth Retr
[ 5] 0.00-20.00 sec 24.5 MBytes 10.3 Mbits/sec 0 sender
[ 5] 0.00-20.00 sec 23.4 MBytes 9.82 Mbits/sec receiver

[davidBar-On]

For UDP this may be because of lost UDP packets, so X was sent but only Y arrived. Another reason for both UDP and TCP is that sent packets may still be buffered when the test ends/terminates, so they don't get the chance to arrive. See related issues #382, #1071, #1367.

[ryanwwest]

It seems like the UDP loss in packets could explain differences in sender/receiver numbers, but not if either the server's sender or receiver average values are 0. I've noticed the same behavior (in TCP as well), where the other three values (from the server receiver/sender and client receiver/sender) are all a substantial, if not equal number of Mbits/sec.

This seems like a program default and I don't think it's been answered yet why.

[davidBar-On]

t seems like the UDP loss in packets could explain differences in sender/receiver numbers, but not if either the server's sender or receiver average values are 0. ...
This seems like a program default and I don't think it's been answered yet why.

@ryanwwest, can you give an example of the scenario and data you are referring to?

[dharmatma]

Hello

Maybe I missed in the manual but my question is that if the result reflect the TCP / UDP payload throughput?
The maximum TCP payload throughput with IPv4 and 1500 Bytes MTU on a 1Gbps link is 949.28 Mbps. Similarly the same composition gives on a 100 Mbps link 94.928 Mbps
However when I am conducting a test with two i7 computers with Intel 219 network cards with a directly connected Cat6 ethernet cable I get only 941 Mbps on a 1 Gbps link

iperf -c 10.0.0.2
------------------------------------------------------------
Client connecting to 10.0.0.2, TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[384] local 10.0.0.1 port 50982 connected with 10.0.0.2 port 5001
[ ID] Interval       Transfer     Bandwidth
[384]  0.0-10.0 sec  1.10 GBytes   941 Mbits/sec

and when I limit the connection to 100 Mbps I get exactly 10 times less 94.1 Mbps

iperf -c 10.0.0.2
------------------------------------------------------------
Client connecting to 10.0.0.2, TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[352] local 10.0.0.1 port 65177 connected with 10.0.0.2 port 5001
[ ID] Interval       Transfer     Bandwidth
[352]  0.0-10.1 sec   113 MBytes  94.1 Mbits/sec

I tried to increase the window size and use multiple connections but they didn't change the result.
Does the iPerf result reflect the TCP payload througput or I need to interpret it somehow elese? If it does represent the TCP payload througput why the result is 941 Mbps and not 949.28 Mbps?

Thank you!

[davidBar-On]

Hi @dharmatma, note that this is iperf3 issue and it seems that you are using iperf2 (although I am not sure there is a difference between the two tools regarding your question).

Can you explain how did you limit limit the connection to 100 Mbps? I assume that somehow you limit the connection throughput, as you didn't use the iperf2 --bandwidth option to limit the iperf throughput. My guess is that by limiting the iperf throughput using --bandwidth 100M you will get almost exactly 100Mbps.

[dharmatma]

Hello

I didn't say there is an issue. This is only a question: What does exacly the iperf/iperf3 througput represent?
The website says:

iPerf3 is a tool for active measurements of the maximum achievable bandwidth on IP networks.
IP networks have different layers each provides different theoretical maximum bandwidth.
Based on the numbers I received during my tests above I can't tell the answer.
In the above example I run iperf in TCP mode where the bandwidth command is not applicable, therefore I change my Ethernet interface settings from Auto speed (1Gbps) to 100 Mbps.

The measurement can represent the troughput of different layers of the OSI ISO model, which I haven't find clearly described in the documentation on the official website.
Using a 1Gbps Ethernet with 1500 Bytes MTU, IPv4 and TCP the max throughuput are the following based on which layer the measurement is applicable.
L1 (Phy) > 1Gbps - Is the measurement applicable on this layer? Definitely not.
L2 (Data) Excluding: L1 (20 Bytes) overhead > 986.99 Mbps - Is the measurement applicable on this layer? Unlikely.
L3 (Net - IPv4) Excluding: L2 (18 Bytes) + L1 overhead > 975.29 Mbps - Is the measurement applicable on this layer? Migth be.
L4 (Transport - TCP) Excluding: L3 (20 Bytes) + L2 + L1 overheads > 962.28 Mbps - Is the measurement applicable on this layer? Migth be.
L5+ (Application - TCP payload) Excluding: L4 (20 Bytes) + L3 + L2 + L1 overheads > 949.28 Mbps - Is the measurement applicable on this layer? Migth be.

So the question is very simple: For the above described environment (1Gbps Ethernet with 1500 Bytes MTU, IPv4 and TCP) what is the theoretical maximum number iperf can report?

Thank you!

[davidBar-On]

Hello, this is a good question .... Assuming that your numbers are correct (I didn't check), I am actually surprised that you got 941Mbps out of the calculated 949Mbps. What is missing in your calculation is the processing overhead. That includes the iperf overhead, TCP stack overhead, Network-card overhead, buffers, etc. Therefore, the maximum throughput depends on the environment where the test is running.

There are some answers to this question on the internet, e.g. this and this.

(Note that this discussion is in a closed issue, so probably it will not get much attention from others. If the discussions I refereed to are not enough, you may try re-send your question as a Discussion, or event send it to forums like Stack Overflow.)

[dharmatma]

Not missing, but purposefully left out, becuase processing overheads, as you also say, are test environment related and my calculations represent the theoretical maximum achievable numbers, and I didn't diluted with test environment bottlenecks.
Similarly the links you provided are not factoring in test enviromnet values. Thank you for the links. The first link is mixing up Frame size vs MTU, however the second link is correctly calculating L2 througput 986 Mbps, therefore it is matching with the calculations I provided before, but unfortunately they don't provide info about what iperf is doing.

Most of today's computers (even consumer grade ones) are easily capable to transmit on 1Gbps rate on Ethernet cards (not to speak about 100Mbps), and the test enviroment I am using belongs to this category, I validated it with two other (paid) measurement software and with these software I am getting the theoretical maximum numbers, provided before, on each layer.
I placed iperf in this same test environment, and the numbers are different that is why my question is: How does iperf calculate the througput. Once I will know this number I can see if my test environment is indeed not perfect for iperf so I will further fine tune, but to do that I need know how the result is calculated by iperf.
I agree with you maybe there is bottleneck in the iperf software and the numbers I received is the absolute max iperf can produce.

Howerver I really would like to exclude my or any test setup, the only reason I set it up to attempt to find answer for the title of this topic "What is iperf3 doing, and what is it saying?" beacuse I couldn't find the answer in the documentation.

Thank you for bringing to my attention that this topic is closed, since I was able to reply I didn't pay attention to the status. It is a good suggetion to start a new topic and continue there.