3 HOWTO for the linux packet generator
4 ------------------------------------
6 Enable CONFIG_NET_PKTGEN to compile and build pktgen either in-kernel
7 or as a module. A module is preferred; modprobe pktgen if needed. Once
8 running, pktgen creates a thread for each CPU with affinity to that CPU.
9 Monitoring and controlling is done via /proc. It is easiest to select a
10 suitable sample script and configure that.
15 root 129 0.3 0.0 0 0 ? SW 2003 523:20 [pktgen/0]
16 root 130 0.3 0.0 0 0 ? SW 2003 509:50 [pktgen/1]
19 For monitoring and control pktgen creates:
20 /proc/net/pktgen/pgctrl
21 /proc/net/pktgen/kpktgend_X
25 Tuning NIC for max performance
26 ==============================
28 The default NIC settings are (likely) not tuned for pktgen's artificial
29 overload type of benchmarking, as this could hurt the normal use-case.
31 Specifically increasing the TX ring buffer in the NIC:
32 # ethtool -G ethX tx 1024
34 A larger TX ring can improve pktgen's performance, while it can hurt
35 in the general case, 1) because the TX ring buffer might get larger
36 than the CPU's L1/L2 cache, 2) because it allows more queueing in the
37 NIC HW layer (which is bad for bufferbloat).
39 One should hesitate to conclude that packets/descriptors in the HW
40 TX ring cause delay. Drivers usually delay cleaning up the
41 ring-buffers for various performance reasons, and packets stalling
42 the TX ring might just be waiting for cleanup.
44 This cleanup issue is specifically the case for the driver ixgbe
45 (Intel 82599 chip). This driver (ixgbe) combines TX+RX ring cleanups,
46 and the cleanup interval is affected by the ethtool --coalesce setting
47 of parameter "rx-usecs".
49 For ixgbe use e.g. "30" resulting in approx 33K interrupts/sec (1/30*10^6):
50 # ethtool -C ethX rx-usecs 30
55 /proc/net/pktgen/kpktgend_0
58 Result: OK: add_device=eth1
60 Most important are the devices assigned to the thread. Note that a
61 device can only belong to one thread.
67 The Params section holds configured information. The Current section
68 holds running statistics. The Result is printed after a run or after
69 interruption. Example:
73 Params: count 10000000 min_pkt_size: 60 max_pkt_size: 60
74 frags: 0 delay: 0 clone_skb: 1000000 ifname: eth1
76 dst_min: 10.10.11.2 dst_max:
78 src_mac: 00:00:00:00:00:00 dst_mac: 00:04:23:AC:FD:82
79 udp_src_min: 9 udp_src_max: 9 udp_dst_min: 9 udp_dst_max: 9
80 src_mac_count: 0 dst_mac_count: 0
83 pkts-sofar: 10000000 errors: 39664
84 started: 1103053986245187us stopped: 1103053999346329us idle: 880401us
85 seq_num: 10000011 cur_dst_mac_offset: 0 cur_src_mac_offset: 0
86 cur_saddr: 0x10a0a0a cur_daddr: 0x20b0a0a
87 cur_udp_dst: 9 cur_udp_src: 9
89 Result: OK: 13101142(c12220741+d880401) usec, 10000000 (60byte,0frags)
90 763292pps 390Mb/sec (390805504bps) errors: 39664
92 Configuring threads and devices
93 ================================
94 This is done via the /proc interface, and most easily done via pgset
95 as defined in the sample scripts.
99 pgset "clone_skb 1" sets the number of copies of the same packet
100 pgset "clone_skb 0" use single SKB for all transmits
101 pgset "burst 8" uses xmit_more API to queue 8 copies of the same
102 packet and update HW tx queue tail pointer once.
103 "burst 1" is the default
104 pgset "pkt_size 9014" sets packet size to 9014
105 pgset "frags 5" packet will consist of 5 fragments
106 pgset "count 200000" sets number of packets to send, set to zero
107 for continuous sends until explicitly stopped.
109 pgset "delay 5000" adds delay to hard_start_xmit(). nanoseconds
111 pgset "dst 10.0.0.1" sets IP destination address
112 (BEWARE! This generator is very aggressive!)
114 pgset "dst_min 10.0.0.1" Same as dst
115 pgset "dst_max 10.0.0.254" Set the maximum destination IP.
116 pgset "src_min 10.0.0.1" Set the minimum (or only) source IP.
117 pgset "src_max 10.0.0.254" Set the maximum source IP.
118 pgset "dst6 fec0::1" IPV6 destination address
119 pgset "src6 fec0::2" IPV6 source address
120 pgset "dstmac 00:00:00:00:00:00" sets MAC destination address
121 pgset "srcmac 00:00:00:00:00:00" sets MAC source address
123 pgset "queue_map_min 0" Sets the min value of tx queue interval
124 pgset "queue_map_max 7" Sets the max value of tx queue interval, for multiqueue devices
125 To select queue 1 of a given device,
126 use queue_map_min=1 and queue_map_max=1
128 pgset "src_mac_count 1" Sets the number of MACs we'll range through.
129 The 'minimum' MAC is what you set with srcmac.
131 pgset "dst_mac_count 1" Sets the number of MACs we'll range through.
132 The 'minimum' MAC is what you set with dstmac.
134 pgset "flag [name]" Set a flag to determine behaviour. Current flags
135 are: IPSRC_RND # IP source is random (between min/max)
136 IPDST_RND # IP destination is random
137 UDPSRC_RND, UDPDST_RND,
138 MACSRC_RND, MACDST_RND
140 MPLS_RND, VID_RND, SVID_RND
142 QUEUE_MAP_RND # queue map random
143 QUEUE_MAP_CPU # queue map mirrors smp_processor_id()
145 IPSEC # IPsec encapsulation (needs CONFIG_XFRM)
146 NODE_ALLOC # node specific memory allocation
147 NO_TIMESTAMP # disable timestamping
149 pgset spi SPI_VALUE Set specific SA used to transform packet.
151 pgset "udp_src_min 9" set UDP source port min, If < udp_src_max, then
152 cycle through the port range.
154 pgset "udp_src_max 9" set UDP source port max.
155 pgset "udp_dst_min 9" set UDP destination port min, If < udp_dst_max, then
156 cycle through the port range.
157 pgset "udp_dst_max 9" set UDP destination port max.
159 pgset "mpls 0001000a,0002000a,0000000a" set MPLS labels (in this example
160 outer label=16,middle label=32,
161 inner label=0 (IPv4 NULL)) Note that
162 there must be no spaces between the
163 arguments. Leading zeros are required.
164 Do not set the bottom of stack bit,
165 that's done automatically. If you do
166 set the bottom of stack bit, that
167 indicates that you want to randomly
168 generate that address and the flag
169 MPLS_RND will be turned on. You
170 can have any mix of random and fixed
171 labels in the label stack.
173 pgset "mpls 0" turn off mpls (or any invalid argument works too!)
175 pgset "vlan_id 77" set VLAN ID 0-4095
176 pgset "vlan_p 3" set priority bit 0-7 (default 0)
177 pgset "vlan_cfi 0" set canonical format identifier 0-1 (default 0)
179 pgset "svlan_id 22" set SVLAN ID 0-4095
180 pgset "svlan_p 3" set priority bit 0-7 (default 0)
181 pgset "svlan_cfi 0" set canonical format identifier 0-1 (default 0)
183 pgset "vlan_id 9999" > 4095 remove vlan and svlan tags
184 pgset "svlan 9999" > 4095 remove svlan tag
187 pgset "tos XX" set former IPv4 TOS field (e.g. "tos 28" for AF11 no ECN, default 00)
188 pgset "traffic_class XX" set former IPv6 TRAFFIC CLASS (e.g. "traffic_class B8" for EF no ECN, default 00)
190 pgset stop aborts injection. Also, ^C aborts generator.
192 pgset "rate 300M" set rate to 300 Mb/s
193 pgset "ratep 1000000" set rate to 1Mpps
195 pgset "xmit_mode netif_receive" RX inject into stack netif_receive_skb()
196 Works with "burst" but not with "clone_skb".
197 Default xmit_mode is "start_xmit".
202 A collection of small tutorial scripts for pktgen is in the
203 samples/pktgen directory:
205 pktgen.conf-1-1 # 1 CPU 1 dev
206 pktgen.conf-1-2 # 1 CPU 2 dev
207 pktgen.conf-2-1 # 2 CPU's 1 dev
208 pktgen.conf-2-2 # 2 CPU's 2 dev
209 pktgen.conf-1-1-rdos # 1 CPU 1 dev w. route DoS
210 pktgen.conf-1-1-ip6 # 1 CPU 1 dev ipv6
211 pktgen.conf-1-1-ip6-rdos # 1 CPU 1 dev ipv6 w. route DoS
212 pktgen.conf-1-1-flows # 1 CPU 1 dev multiple flows.
214 Run in shell: ./pktgen.conf-X-Y
215 This does all the setup including sending.
220 Note that when adding devices to a specific CPU it is a good idea to
221 also assign /proc/irq/XX/smp_affinity so that the TX interrupts are bound
222 to the same CPU. This reduces cache bouncing when freeing skbs.
226 Default IPsec transformation with ESP encapsulation plus transport mode
227 can be enabled by simply setting:
232 To avoid breaking existing testbed scripts for using AH type and tunnel mode,
233 you can use "pgset spi SPI_VALUE" to specify which transformation mode
237 Current commands and configuration options
238 ==========================================
240 ** Pgcontrol commands:
330 xmit_mode <start_xmit|netif_receive>
342 ftp://robur.slu.se/pub/Linux/net-development/pktgen-testing/
343 ftp://robur.slu.se/pub/Linux/net-development/pktgen-testing/examples/
345 Paper from Linux-Kongress in Erlangen 2004.
346 ftp://robur.slu.se/pub/Linux/net-development/pktgen-testing/pktgen_paper.pdf
349 Grant Grundler for testing on IA-64 and parisc, Harald Welte, Lennert Buytenhek
350 Stephen Hemminger, Andi Kleen, Dave Miller and many others.
353 Good luck with the linux net-development.