Blame view
Documentation/networking/ixgb.txt
16.6 KB
d7064f4c1 Documentation/net... |
1 2 |
Linux Base Driver for 10 Gigabit Intel(R) Ethernet Network Connection ===================================================================== |
1da177e4c Linux-2.6.12-rc2 |
3 |
|
d7064f4c1 Documentation/net... |
4 |
March 14, 2011 |
1da177e4c Linux-2.6.12-rc2 |
5 6 7 8 9 10 11 |
Contents ======== - In This Release - Identifying Your Adapter |
2115a6432 ixgb: update read... |
12 |
- Building and Installation |
1da177e4c Linux-2.6.12-rc2 |
13 14 |
- Command Line Parameters - Improving Performance |
2115a6432 ixgb: update read... |
15 16 |
- Additional Configurations - Known Issues/Troubleshooting |
1da177e4c Linux-2.6.12-rc2 |
17 |
- Support |
2115a6432 ixgb: update read... |
18 |
|
1da177e4c Linux-2.6.12-rc2 |
19 20 |
In This Release =============== |
2115a6432 ixgb: update read... |
21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 |
This file describes the ixgb Linux Base Driver for the 10 Gigabit Intel(R) Network Connection. This driver includes support for Itanium(R)2-based systems. For questions related to hardware requirements, refer to the documentation supplied with your 10 Gigabit adapter. All hardware requirements listed apply to use with Linux. The following features are available in this kernel: - Native VLANs - Channel Bonding (teaming) - SNMP Channel Bonding documentation can be found in the Linux kernel source: /Documentation/networking/bonding.txt The driver information previously displayed in the /proc filesystem is not supported in this release. Alternatively, you can use ethtool (version 1.6 |
322d3ed16 ixgb: remove refe... |
39 |
or later), lspci, and iproute2 to obtain the same information. |
2115a6432 ixgb: update read... |
40 41 42 |
Instructions on updating ethtool can be found in the section "Additional Configurations" later in this document. |
1da177e4c Linux-2.6.12-rc2 |
43 |
|
1da177e4c Linux-2.6.12-rc2 |
44 45 46 |
Identifying Your Adapter ======================== |
2115a6432 ixgb: update read... |
47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 |
The following Intel network adapters are compatible with the drivers in this release: Controller Adapter Name Physical Layer ---------- ------------ -------------- 82597EX Intel(R) PRO/10GbE LR/SR/CX4 10G Base-LR (1310 nm optical fiber) Server Adapters 10G Base-SR (850 nm optical fiber) 10G Base-CX4(twin-axial copper cabling) For more information on how to identify your adapter, go to the Adapter & Driver ID Guide at: http://support.intel.com/support/network/sb/CS-012904.htm Building and Installation ========================= select m for "Intel(R) PRO/10GbE support" located at: Location: -> Device Drivers -> Network device support (NETDEVICES [=y]) -> Ethernet (10000 Mbit) (NETDEV_10000 [=y]) 1. make modules && make modules_install 2. Load the module: modprobe ixgb <parameter>=<value> The insmod command can be used if the full path to the driver module is specified. For example: insmod /lib/modules/<KERNEL VERSION>/kernel/drivers/net/ixgb/ixgb.ko With 2.6 based kernels also make sure that older ixgb drivers are removed from the kernel, before loading the new module: |
1da177e4c Linux-2.6.12-rc2 |
83 |
|
2115a6432 ixgb: update read... |
84 |
rmmod ixgb; modprobe ixgb |
1da177e4c Linux-2.6.12-rc2 |
85 |
|
2115a6432 ixgb: update read... |
86 87 |
3. Assign an IP address to the interface by entering the following, where x is the interface number: |
1da177e4c Linux-2.6.12-rc2 |
88 |
|
322d3ed16 ixgb: remove refe... |
89 |
ip addr add ethx <IP_address> |
2115a6432 ixgb: update read... |
90 91 92 93 94 95 |
4. Verify that the interface works. Enter the following, where <IP_address> is the IP address for another machine on the same subnet as the interface that is being tested: ping <IP_address> |
1da177e4c Linux-2.6.12-rc2 |
96 |
|
1da177e4c Linux-2.6.12-rc2 |
97 98 99 |
Command Line Parameters ======================= |
2115a6432 ixgb: update read... |
100 101 102 |
If the driver is built as a module, the following optional parameters are used by entering them on the command line with the modprobe command using this syntax: |
1da177e4c Linux-2.6.12-rc2 |
103 104 |
modprobe ixgb [<option>=<VAL1>,<VAL2>,...] |
2115a6432 ixgb: update read... |
105 |
For example, with two 10GbE PCI adapters, entering: |
1da177e4c Linux-2.6.12-rc2 |
106 |
|
2115a6432 ixgb: update read... |
107 |
modprobe ixgb TxDescriptors=80,128 |
1da177e4c Linux-2.6.12-rc2 |
108 |
|
2115a6432 ixgb: update read... |
109 |
loads the ixgb driver with 80 TX resources for the first adapter and 128 TX |
1da177e4c Linux-2.6.12-rc2 |
110 111 112 |
resources for the second adapter. The default value for each parameter is generally the recommended setting, |
2115a6432 ixgb: update read... |
113 |
unless otherwise noted. |
1da177e4c Linux-2.6.12-rc2 |
114 115 116 117 |
FlowControl Valid Range: 0-3 (0=none, 1=Rx only, 2=Tx only, 3=Rx&Tx) Default: Read from the EEPROM |
2115a6432 ixgb: update read... |
118 119 120 121 |
If EEPROM is not detected, default is 1 This parameter controls the automatic generation(Tx) and response(Rx) to Ethernet PAUSE frames. There are hardware bugs associated with enabling Tx flow control so beware. |
1da177e4c Linux-2.6.12-rc2 |
122 123 124 125 |
RxDescriptors Valid Range: 64-512 Default Value: 512 |
2115a6432 ixgb: update read... |
126 127 128 129 130 |
This value is the number of receive descriptors allocated by the driver. Increasing this value allows the driver to buffer more incoming packets. Each descriptor is 16 bytes. A receive buffer is also allocated for each descriptor and can be either 2048, 4056, 8192, or 16384 bytes, depending on the MTU setting. When the MTU size is 1500 or less, the |
1da177e4c Linux-2.6.12-rc2 |
131 |
receive buffer size is 2048 bytes. When the MTU is greater than 1500 the |
2115a6432 ixgb: update read... |
132 |
receive buffer size will be either 4056, 8192, or 16384 bytes. The |
1da177e4c Linux-2.6.12-rc2 |
133 134 135 136 |
maximum MTU size is 16114. RxIntDelay Valid Range: 0-65535 (0=off) |
2115a6432 ixgb: update read... |
137 138 139 140 141 142 143 |
Default Value: 72 This value delays the generation of receive interrupts in units of 0.8192 microseconds. Receive interrupt reduction can improve CPU efficiency if properly tuned for specific network traffic. Increasing this value adds extra latency to frame reception and can end up decreasing the throughput of TCP traffic. If the system is reporting dropped receives, this value may be set too high, causing the driver to |
1da177e4c Linux-2.6.12-rc2 |
144 145 146 147 148 149 |
run out of available receive descriptors. TxDescriptors Valid Range: 64-4096 Default Value: 256 This value is the number of transmit descriptors allocated by the driver. |
2115a6432 ixgb: update read... |
150 |
Increasing this value allows the driver to queue more transmits. Each |
1da177e4c Linux-2.6.12-rc2 |
151 152 153 154 155 156 157 |
descriptor is 16 bytes. XsumRX Valid Range: 0-1 Default Value: 1 A value of '1' indicates that the driver should enable IP checksum offload for received packets (both UDP and TCP) to the adapter hardware. |
1da177e4c Linux-2.6.12-rc2 |
158 159 160 |
Improving Performance ===================== |
2115a6432 ixgb: update read... |
161 162 163 164 165 166 |
With the 10 Gigabit server adapters, the default Linux configuration will very likely limit the total available throughput artificially. There is a set of configuration changes that, when applied together, will increase the ability of Linux to transmit and receive data. The following enhancements were originally acquired from settings published at http://www.spec.org/web99/ for various submitted results using Linux. |
1da177e4c Linux-2.6.12-rc2 |
167 |
|
2115a6432 ixgb: update read... |
168 169 |
NOTE: These changes are only suggestions, and serve as a starting point for tuning your network performance. |
1da177e4c Linux-2.6.12-rc2 |
170 171 |
The changes are made in three major ways, listed in order of greatest effect: |
322d3ed16 ixgb: remove refe... |
172 |
- Use ip link to modify the mtu (maximum transmission unit) and the txqueuelen |
1da177e4c Linux-2.6.12-rc2 |
173 174 |
parameter. - Use sysctl to modify /proc parameters (essentially kernel tuning) |
2115a6432 ixgb: update read... |
175 |
- Use setpci to modify the MMRBC field in PCI-X configuration space to increase |
1da177e4c Linux-2.6.12-rc2 |
176 |
transmit burst lengths on the bus. |
2115a6432 ixgb: update read... |
177 178 179 180 181 182 |
NOTE: setpci modifies the adapter's configuration registers to allow it to read up to 4k bytes at a time (for transmits). However, for some systems the behavior after modifying this register may be undefined (possibly errors of some kind). A power-cycle, hard reset or explicitly setting the e6 register back to 22 (setpci -d 8086:1a48 e6.b=22) may be required to get back to a stable configuration. |
1da177e4c Linux-2.6.12-rc2 |
183 184 185 |
- COPY these lines and paste them into ixgb_perf.sh: #!/bin/bash |
2115a6432 ixgb: update read... |
186 187 |
echo "configuring network performance , edit this file to change the interface or device ID of 10GbE card" |
1da177e4c Linux-2.6.12-rc2 |
188 |
# set mmrbc to 4k reads, modify only Intel 10GbE device IDs |
2115a6432 ixgb: update read... |
189 190 191 192 193 |
# replace 1a48 with appropriate 10GbE device's ID installed on the system, # if needed. setpci -d 8086:1a48 e6.b=2e # set the MTU (max transmission unit) - it requires your switch and clients # to change as well. |
1da177e4c Linux-2.6.12-rc2 |
194 195 |
# set the txqueuelen # your ixgb adapter should be loaded as eth1 for this to work, change if needed |
322d3ed16 ixgb: remove refe... |
196 |
ip li set dev eth1 mtu 9000 txqueuelen 1000 up |
2115a6432 ixgb: update read... |
197 198 |
# call the sysctl utility to modify /proc/sys entries sysctl -p ./sysctl_ixgb.conf |
1da177e4c Linux-2.6.12-rc2 |
199 200 201 202 203 204 205 206 207 |
- END ixgb_perf.sh - COPY these lines and paste them into sysctl_ixgb.conf: # some of the defaults may be different for your kernel # call this file with sysctl -p <this file> # these are just suggested values that worked well to increase throughput in # several network benchmark tests, your mileage may vary ### IPV4 specific settings |
2115a6432 ixgb: update read... |
208 209 210 211 212 213 214 215 216 217 218 |
# turn TCP timestamp support off, default 1, reduces CPU use net.ipv4.tcp_timestamps = 0 # turn SACK support off, default on # on systems with a VERY fast bus -> memory interface this is the big gainer net.ipv4.tcp_sack = 0 # set min/default/max TCP read buffer, default 4096 87380 174760 net.ipv4.tcp_rmem = 10000000 10000000 10000000 # set min/pressure/max TCP write buffer, default 4096 16384 131072 net.ipv4.tcp_wmem = 10000000 10000000 10000000 # set min/pressure/max TCP buffer space, default 31744 32256 32768 net.ipv4.tcp_mem = 10000000 10000000 10000000 |
1da177e4c Linux-2.6.12-rc2 |
219 220 |
### CORE settings (mostly for socket and UDP effect) |
2115a6432 ixgb: update read... |
221 222 223 224 225 226 227 228 229 230 231 232 |
# set maximum receive socket buffer size, default 131071 net.core.rmem_max = 524287 # set maximum send socket buffer size, default 131071 net.core.wmem_max = 524287 # set default receive socket buffer size, default 65535 net.core.rmem_default = 524287 # set default send socket buffer size, default 65535 net.core.wmem_default = 524287 # set maximum amount of option memory buffers, default 10240 net.core.optmem_max = 524287 # set number of unprocessed input packets before kernel starts dropping them; default 300 net.core.netdev_max_backlog = 300000 |
1da177e4c Linux-2.6.12-rc2 |
233 |
- END sysctl_ixgb.conf |
2115a6432 ixgb: update read... |
234 235 236 |
Edit the ixgb_perf.sh script if necessary to change eth1 to whatever interface your ixgb driver is using and/or replace '1a48' with appropriate 10GbE device's ID installed on the system. |
1da177e4c Linux-2.6.12-rc2 |
237 |
|
2115a6432 ixgb: update read... |
238 239 |
NOTE: Unless these scripts are added to the boot process, these changes will only last only until the next system reboot. |
1da177e4c Linux-2.6.12-rc2 |
240 241 242 243 |
Resolving Slow UDP Traffic -------------------------- |
2115a6432 ixgb: update read... |
244 245 246 247 248 |
If your server does not seem to be able to receive UDP traffic as fast as it can receive TCP traffic, it could be because Linux, by default, does not set the network stack buffers as large as they need to be to support high UDP transfer rates. One way to alleviate this problem is to allow more memory to be used by the IP stack to store incoming data. |
1da177e4c Linux-2.6.12-rc2 |
249 |
|
2115a6432 ixgb: update read... |
250 |
For instance, use the commands: |
1da177e4c Linux-2.6.12-rc2 |
251 252 253 |
sysctl -w net.core.rmem_max=262143 and sysctl -w net.core.rmem_default=262143 |
2115a6432 ixgb: update read... |
254 255 256 |
to increase the read buffer memory max and default to 262143 (256k - 1) from defaults of max=131071 (128k - 1) and default=65535 (64k - 1). These variables will increase the amount of memory used by the network stack for receives, and |
1da177e4c Linux-2.6.12-rc2 |
257 |
can be increased significantly more if necessary for your application. |
2115a6432 ixgb: update read... |
258 259 260 261 262 263 264 265 |
Additional Configurations ========================= Configuring the Driver on Different Distributions ------------------------------------------------- Configuring a network driver to load properly when the system is started is distribution dependent. Typically, the configuration process involves adding |
d7064f4c1 Documentation/net... |
266 267 268 |
an alias line to /etc/modprobe.conf as well as editing other system startup scripts and/or configuration files. Many popular Linux distributions ship with tools to make these changes for you. To learn the proper way to |
2115a6432 ixgb: update read... |
269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 |
configure a network device for your system, refer to your distribution documentation. If during this process you are asked for the driver or module name, the name for the Linux Base Driver for the Intel 10GbE Family of Adapters is ixgb. Viewing Link Messages --------------------- Link messages will not be displayed to the console if the distribution is restricting system messages. In order to see network driver link messages on your console, set dmesg to eight by entering the following: dmesg -n 8 NOTE: This setting is not saved across reboots. Jumbo Frames ------------ The driver supports Jumbo Frames for all adapters. Jumbo Frames support is enabled by changing the MTU to a value larger than the default of 1500. |
322d3ed16 ixgb: remove refe... |
289 |
The maximum value for the MTU is 16114. Use the ip command to |
2115a6432 ixgb: update read... |
290 |
increase the MTU size. For example: |
322d3ed16 ixgb: remove refe... |
291 |
ip li set dev ethx mtu 9000 |
2115a6432 ixgb: update read... |
292 293 294 |
The maximum MTU setting for Jumbo Frames is 16114. This value coincides with the maximum Jumbo Frames size of 16128. |
d7064f4c1 Documentation/net... |
295 |
ethtool |
2115a6432 ixgb: update read... |
296 297 |
------- The driver utilizes the ethtool interface for driver configuration and |
68f20d948 Documentation/net... |
298 |
diagnostics, as well as displaying statistical information. The ethtool |
2115a6432 ixgb: update read... |
299 300 301 |
version 1.6 or later is required for this functionality. The latest release of ethtool can be found from |
68f20d948 Documentation/net... |
302 |
http://ftp.kernel.org/pub/software/network/ethtool/ |
2115a6432 ixgb: update read... |
303 |
|
68f20d948 Documentation/net... |
304 305 306 |
NOTE: The ethtool version 1.6 only supports a limited set of ethtool options. Support for a more complete ethtool feature set can be enabled by upgrading to the latest version. |
2115a6432 ixgb: update read... |
307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 |
NAPI ---- NAPI (Rx polling mode) is supported in the ixgb driver. NAPI is enabled or disabled based on the configuration of the kernel. see CONFIG_IXGB_NAPI See www.cyberus.ca/~hadi/usenix-paper.tgz for more information on NAPI. Known Issues/Troubleshooting ============================ NOTE: After installing the driver, if your Intel Network Connection is not working, verify in the "In This Release" section of the readme that you have installed the correct driver. Intel(R) PRO/10GbE CX4 Server Adapter Cable Interoperability Issue with Fujitsu XENPAK Module in SmartBits Chassis --------------------------------------------------------------------- Excessive CRC errors may be observed if the Intel(R) PRO/10GbE CX4 Server adapter is connected to a Fujitsu XENPAK CX4 module in a SmartBits chassis using 15 m/24AWG cable assemblies manufactured by Fujitsu or Leoni. The CRC errors may be received either by the Intel(R) PRO/10GbE CX4 Server adapter or the SmartBits. If this situation occurs using a different cable assembly may resolve the issue. CX4 Server Adapter Cable Interoperability Issues with HP Procurve 3400cl Switch Port ------------------------------------------------------------------------ Excessive CRC errors may be observed if the Intel(R) PRO/10GbE CX4 Server adapter is connected to an HP Procurve 3400cl switch port using short cables (1 m or shorter). If this situation occurs, using a longer cable may resolve the issue. Excessive CRC errors may be observed using Fujitsu 24AWG cable assemblies that Are 10 m or longer or where using a Leoni 15 m/24AWG cable assembly. The CRC errors may be received either by the CX4 Server adapter or at the switch. If this situation occurs, using a different cable assembly may resolve the issue. Jumbo Frames System Requirement ------------------------------- Memory allocation failures have been observed on Linux systems with 64 MB of RAM or less that are running Jumbo Frames. If you are using Jumbo Frames, your system may require more than the advertised minimum requirement of 64 MB of system memory. Performance Degradation with Jumbo Frames ----------------------------------------- Degradation in throughput performance may be observed in some Jumbo frames environments. If this is observed, increasing the application's socket buffer size and/or increasing the /proc/sys/net/ipv4/tcp_*mem entry values may help. See the specific application manual and /usr/src/linux*/Documentation/ networking/ip-sysctl.txt for more details. Allocating Rx Buffers when Using Jumbo Frames --------------------------------------------- Allocating Rx buffers when using Jumbo Frames on 2.6.x kernels may fail if the available memory is heavily fragmented. This issue may be seen with PCI-X adapters or with packet split disabled. This can be reduced or eliminated by changing the amount of available memory for receive buffer allocation, by increasing /proc/sys/vm/min_free_kbytes. Multiple Interfaces on Same Ethernet Broadcast Network ------------------------------------------------------ Due to the default ARP behavior on Linux, it is not possible to have one system on two IP networks in the same Ethernet broadcast domain (non-partitioned switch) behave as expected. All Ethernet interfaces will respond to IP traffic for any IP address assigned to the system. This results in unbalanced receive traffic. If you have multiple interfaces in a server, do either of the following: - Turn on ARP filtering by entering: echo 1 > /proc/sys/net/ipv4/conf/all/arp_filter - Install the interfaces in separate broadcast domains - either in different switches or in a switch partitioned to VLANs. UDP Stress Test Dropped Packet Issue -------------------------------------- Under small packets UDP stress test with 10GbE driver, the Linux system may drop UDP packets due to the fullness of socket buffers. You may want to change the driver's Flow Control variables to the minimum value for controlling packet reception. Tx Hangs Possible Under Stress ------------------------------ Under stress conditions, if TX hangs occur, turning off TSO "ethtool -K eth0 tso off" may resolve the problem. |
1da177e4c Linux-2.6.12-rc2 |
404 405 |
Support ======= |
2115a6432 ixgb: update read... |
406 |
For general information, go to the Intel support website at: |
1da177e4c Linux-2.6.12-rc2 |
407 408 |
http://support.intel.com |
2115a6432 ixgb: update read... |
409 410 411 |
or the Intel Wired Networking project hosted by Sourceforge at: http://sourceforge.net/projects/e1000 |
1da177e4c Linux-2.6.12-rc2 |
412 |
If an issue is identified with the released source code on the supported |
2115a6432 ixgb: update read... |
413 414 |
kernel with a supported adapter, email the specific information related to the issue to e1000-devel@lists.sf.net |