Eric Lee / linux-smarc-t335x-v3.2

Download as
Browse Code ยป

Commit f11970e383acd6f505f492f1bc07fb1a4d884829

Authored by Neil Horman
Committed by David S. Miller
1 parent 6dcbbe25dc
Exists in master and in 4 other branches v3.2_AM335xPSP_04.06.00.11, v3.2_NEWT335xPSP_04.06.00.11, v3.2_SBC_SMARTMEN, v3.2_SMARCT335xPSP_04.06.00.11

net: make dev_disable_lro use physical device if passed a vlan dev (v2) 

If the device passed into dev_disable_lro is a vlan, then repoint the dev
poniter so that we actually modify the underlying physical device.

Signed-of-by: Neil Horman <nhorman@tuxdriver.com>
CC: davem@davemloft.net
CC: bhutchings@solarflare.com

Signed-off-by: David S. Miller <davem@davemloft.net>

Showing 1 changed file with 7 additions and 0 deletions Inline Diff

1 /* 1 /*
2 * NET3 Protocol independent device support routines. 2 * NET3 Protocol independent device support routines.
3 * 3 *
4 * This program is free software; you can redistribute it and/or 4 * This program is free software; you can redistribute it and/or
5 * modify it under the terms of the GNU General Public License 5 * modify it under the terms of the GNU General Public License
6 * as published by the Free Software Foundation; either version 6 * as published by the Free Software Foundation; either version
7 * 2 of the License, or (at your option) any later version. 7 * 2 of the License, or (at your option) any later version.
8 * 8 *
9 * Derived from the non IP parts of dev.c 1.0.19 9 * Derived from the non IP parts of dev.c 1.0.19
10 * Authors: Ross Biro 10 * Authors: Ross Biro
11 * Fred N. van Kempen, <waltje@uWalt.NL.Mugnet.ORG> 11 * Fred N. van Kempen, <waltje@uWalt.NL.Mugnet.ORG>
12 * Mark Evans, <evansmp@uhura.aston.ac.uk> 12 * Mark Evans, <evansmp@uhura.aston.ac.uk>
13 * 13 *
14 * Additional Authors: 14 * Additional Authors:
15 * Florian la Roche <rzsfl@rz.uni-sb.de> 15 * Florian la Roche <rzsfl@rz.uni-sb.de>
16 * Alan Cox <gw4pts@gw4pts.ampr.org> 16 * Alan Cox <gw4pts@gw4pts.ampr.org>
17 * David Hinds <dahinds@users.sourceforge.net> 17 * David Hinds <dahinds@users.sourceforge.net>
18 * Alexey Kuznetsov <kuznet@ms2.inr.ac.ru> 18 * Alexey Kuznetsov <kuznet@ms2.inr.ac.ru>
19 * Adam Sulmicki <adam@cfar.umd.edu> 19 * Adam Sulmicki <adam@cfar.umd.edu>
20 * Pekka Riikonen <priikone@poesidon.pspt.fi> 20 * Pekka Riikonen <priikone@poesidon.pspt.fi>
21 * 21 *
22 * Changes: 22 * Changes:
23 * D.J. Barrow : Fixed bug where dev->refcnt gets set 23 * D.J. Barrow : Fixed bug where dev->refcnt gets set
24 * to 2 if register_netdev gets called 24 * to 2 if register_netdev gets called
25 * before net_dev_init & also removed a 25 * before net_dev_init & also removed a
26 * few lines of code in the process. 26 * few lines of code in the process.
27 * Alan Cox : device private ioctl copies fields back. 27 * Alan Cox : device private ioctl copies fields back.
28 * Alan Cox : Transmit queue code does relevant 28 * Alan Cox : Transmit queue code does relevant
29 * stunts to keep the queue safe. 29 * stunts to keep the queue safe.
30 * Alan Cox : Fixed double lock. 30 * Alan Cox : Fixed double lock.
31 * Alan Cox : Fixed promisc NULL pointer trap 31 * Alan Cox : Fixed promisc NULL pointer trap
32 * ???????? : Support the full private ioctl range 32 * ???????? : Support the full private ioctl range
33 * Alan Cox : Moved ioctl permission check into 33 * Alan Cox : Moved ioctl permission check into
34 * drivers 34 * drivers
35 * Tim Kordas : SIOCADDMULTI/SIOCDELMULTI 35 * Tim Kordas : SIOCADDMULTI/SIOCDELMULTI
36 * Alan Cox : 100 backlog just doesn't cut it when 36 * Alan Cox : 100 backlog just doesn't cut it when
37 * you start doing multicast video 8) 37 * you start doing multicast video 8)
38 * Alan Cox : Rewrote net_bh and list manager. 38 * Alan Cox : Rewrote net_bh and list manager.
39 * Alan Cox : Fix ETH_P_ALL echoback lengths. 39 * Alan Cox : Fix ETH_P_ALL echoback lengths.
40 * Alan Cox : Took out transmit every packet pass 40 * Alan Cox : Took out transmit every packet pass
41 * Saved a few bytes in the ioctl handler 41 * Saved a few bytes in the ioctl handler
42 * Alan Cox : Network driver sets packet type before 42 * Alan Cox : Network driver sets packet type before
43 * calling netif_rx. Saves a function 43 * calling netif_rx. Saves a function
44 * call a packet. 44 * call a packet.
45 * Alan Cox : Hashed net_bh() 45 * Alan Cox : Hashed net_bh()
46 * Richard Kooijman: Timestamp fixes. 46 * Richard Kooijman: Timestamp fixes.
47 * Alan Cox : Wrong field in SIOCGIFDSTADDR 47 * Alan Cox : Wrong field in SIOCGIFDSTADDR
48 * Alan Cox : Device lock protection. 48 * Alan Cox : Device lock protection.
49 * Alan Cox : Fixed nasty side effect of device close 49 * Alan Cox : Fixed nasty side effect of device close
50 * changes. 50 * changes.
51 * Rudi Cilibrasi : Pass the right thing to 51 * Rudi Cilibrasi : Pass the right thing to
52 * set_mac_address() 52 * set_mac_address()
53 * Dave Miller : 32bit quantity for the device lock to 53 * Dave Miller : 32bit quantity for the device lock to
54 * make it work out on a Sparc. 54 * make it work out on a Sparc.
55 * Bjorn Ekwall : Added KERNELD hack. 55 * Bjorn Ekwall : Added KERNELD hack.
56 * Alan Cox : Cleaned up the backlog initialise. 56 * Alan Cox : Cleaned up the backlog initialise.
57 * Craig Metz : SIOCGIFCONF fix if space for under 57 * Craig Metz : SIOCGIFCONF fix if space for under
58 * 1 device. 58 * 1 device.
59 * Thomas Bogendoerfer : Return ENODEV for dev_open, if there 59 * Thomas Bogendoerfer : Return ENODEV for dev_open, if there
60 * is no device open function. 60 * is no device open function.
61 * Andi Kleen : Fix error reporting for SIOCGIFCONF 61 * Andi Kleen : Fix error reporting for SIOCGIFCONF
62 * Michael Chastain : Fix signed/unsigned for SIOCGIFCONF 62 * Michael Chastain : Fix signed/unsigned for SIOCGIFCONF
63 * Cyrus Durgin : Cleaned for KMOD 63 * Cyrus Durgin : Cleaned for KMOD
64 * Adam Sulmicki : Bug Fix : Network Device Unload 64 * Adam Sulmicki : Bug Fix : Network Device Unload
65 * A network device unload needs to purge 65 * A network device unload needs to purge
66 * the backlog queue. 66 * the backlog queue.
67 * Paul Rusty Russell : SIOCSIFNAME 67 * Paul Rusty Russell : SIOCSIFNAME
68 * Pekka Riikonen : Netdev boot-time settings code 68 * Pekka Riikonen : Netdev boot-time settings code
69 * Andrew Morton : Make unregister_netdevice wait 69 * Andrew Morton : Make unregister_netdevice wait
70 * indefinitely on dev->refcnt 70 * indefinitely on dev->refcnt
71 * J Hadi Salim : - Backlog queue sampling 71 * J Hadi Salim : - Backlog queue sampling
72 * - netif_rx() feedback 72 * - netif_rx() feedback
73 */ 73 */
74 74
75 #include <asm/uaccess.h> 75 #include <asm/uaccess.h>
76 #include <asm/system.h> 76 #include <asm/system.h>
77 #include <linux/bitops.h> 77 #include <linux/bitops.h>
78 #include <linux/capability.h> 78 #include <linux/capability.h>
79 #include <linux/cpu.h> 79 #include <linux/cpu.h>
80 #include <linux/types.h> 80 #include <linux/types.h>
81 #include <linux/kernel.h> 81 #include <linux/kernel.h>
82 #include <linux/hash.h> 82 #include <linux/hash.h>
83 #include <linux/slab.h> 83 #include <linux/slab.h>
84 #include <linux/sched.h> 84 #include <linux/sched.h>
85 #include <linux/mutex.h> 85 #include <linux/mutex.h>
86 #include <linux/string.h> 86 #include <linux/string.h>
87 #include <linux/mm.h> 87 #include <linux/mm.h>
88 #include <linux/socket.h> 88 #include <linux/socket.h>
89 #include <linux/sockios.h> 89 #include <linux/sockios.h>
90 #include <linux/errno.h> 90 #include <linux/errno.h>
91 #include <linux/interrupt.h> 91 #include <linux/interrupt.h>
92 #include <linux/if_ether.h> 92 #include <linux/if_ether.h>
93 #include <linux/netdevice.h> 93 #include <linux/netdevice.h>
94 #include <linux/etherdevice.h> 94 #include <linux/etherdevice.h>
95 #include <linux/ethtool.h> 95 #include <linux/ethtool.h>
96 #include <linux/notifier.h> 96 #include <linux/notifier.h>
97 #include <linux/skbuff.h> 97 #include <linux/skbuff.h>
98 #include <net/net_namespace.h> 98 #include <net/net_namespace.h>
99 #include <net/sock.h> 99 #include <net/sock.h>
100 #include <linux/rtnetlink.h> 100 #include <linux/rtnetlink.h>
101 #include <linux/proc_fs.h> 101 #include <linux/proc_fs.h>
102 #include <linux/seq_file.h> 102 #include <linux/seq_file.h>
103 #include <linux/stat.h> 103 #include <linux/stat.h>
104 #include <net/dst.h> 104 #include <net/dst.h>
105 #include <net/pkt_sched.h> 105 #include <net/pkt_sched.h>
106 #include <net/checksum.h> 106 #include <net/checksum.h>
107 #include <net/xfrm.h> 107 #include <net/xfrm.h>
108 #include <linux/highmem.h> 108 #include <linux/highmem.h>
109 #include <linux/init.h> 109 #include <linux/init.h>
110 #include <linux/kmod.h> 110 #include <linux/kmod.h>
111 #include <linux/module.h> 111 #include <linux/module.h>
112 #include <linux/netpoll.h> 112 #include <linux/netpoll.h>
113 #include <linux/rcupdate.h> 113 #include <linux/rcupdate.h>
114 #include <linux/delay.h> 114 #include <linux/delay.h>
115 #include <net/wext.h> 115 #include <net/wext.h>
116 #include <net/iw_handler.h> 116 #include <net/iw_handler.h>
117 #include <asm/current.h> 117 #include <asm/current.h>
118 #include <linux/audit.h> 118 #include <linux/audit.h>
119 #include <linux/dmaengine.h> 119 #include <linux/dmaengine.h>
120 #include <linux/err.h> 120 #include <linux/err.h>
121 #include <linux/ctype.h> 121 #include <linux/ctype.h>
122 #include <linux/if_arp.h> 122 #include <linux/if_arp.h>
123 #include <linux/if_vlan.h> 123 #include <linux/if_vlan.h>
124 #include <linux/ip.h> 124 #include <linux/ip.h>
125 #include <net/ip.h> 125 #include <net/ip.h>
126 #include <linux/ipv6.h> 126 #include <linux/ipv6.h>
127 #include <linux/in.h> 127 #include <linux/in.h>
128 #include <linux/jhash.h> 128 #include <linux/jhash.h>
129 #include <linux/random.h> 129 #include <linux/random.h>
130 #include <trace/events/napi.h> 130 #include <trace/events/napi.h>
131 #include <trace/events/net.h> 131 #include <trace/events/net.h>
132 #include <trace/events/skb.h> 132 #include <trace/events/skb.h>
133 #include <linux/pci.h> 133 #include <linux/pci.h>
134 #include <linux/inetdevice.h> 134 #include <linux/inetdevice.h>
135 #include <linux/cpu_rmap.h> 135 #include <linux/cpu_rmap.h>
136 136
137 #include "net-sysfs.h" 137 #include "net-sysfs.h"
138 138
139 /* Instead of increasing this, you should create a hash table. */ 139 /* Instead of increasing this, you should create a hash table. */
140 #define MAX_GRO_SKBS 8 140 #define MAX_GRO_SKBS 8
141 141
142 /* This should be increased if a protocol with a bigger head is added. */ 142 /* This should be increased if a protocol with a bigger head is added. */
143 #define GRO_MAX_HEAD (MAX_HEADER + 128) 143 #define GRO_MAX_HEAD (MAX_HEADER + 128)
144 144
145 /* 145 /*
146 * The list of packet types we will receive (as opposed to discard) 146 * The list of packet types we will receive (as opposed to discard)
147 * and the routines to invoke. 147 * and the routines to invoke.
148 * 148 *
149 * Why 16. Because with 16 the only overlap we get on a hash of the 149 * Why 16. Because with 16 the only overlap we get on a hash of the
150 * low nibble of the protocol value is RARP/SNAP/X.25. 150 * low nibble of the protocol value is RARP/SNAP/X.25.
151 * 151 *
152 * NOTE: That is no longer true with the addition of VLAN tags. Not 152 * NOTE: That is no longer true with the addition of VLAN tags. Not
153 * sure which should go first, but I bet it won't make much 153 * sure which should go first, but I bet it won't make much
154 * difference if we are running VLANs. The good news is that 154 * difference if we are running VLANs. The good news is that
155 * this protocol won't be in the list unless compiled in, so 155 * this protocol won't be in the list unless compiled in, so
156 * the average user (w/out VLANs) will not be adversely affected. 156 * the average user (w/out VLANs) will not be adversely affected.
157 * --BLG 157 * --BLG
158 * 158 *
159 * 0800 IP 159 * 0800 IP
160 * 8100 802.1Q VLAN 160 * 8100 802.1Q VLAN
161 * 0001 802.3 161 * 0001 802.3
162 * 0002 AX.25 162 * 0002 AX.25
163 * 0004 802.2 163 * 0004 802.2
164 * 8035 RARP 164 * 8035 RARP
165 * 0005 SNAP 165 * 0005 SNAP
166 * 0805 X.25 166 * 0805 X.25
167 * 0806 ARP 167 * 0806 ARP
168 * 8137 IPX 168 * 8137 IPX
169 * 0009 Localtalk 169 * 0009 Localtalk
170 * 86DD IPv6 170 * 86DD IPv6
171 */ 171 */
172 172
173 #define PTYPE_HASH_SIZE (16) 173 #define PTYPE_HASH_SIZE (16)
174 #define PTYPE_HASH_MASK (PTYPE_HASH_SIZE - 1) 174 #define PTYPE_HASH_MASK (PTYPE_HASH_SIZE - 1)
175 175
176 static DEFINE_SPINLOCK(ptype_lock); 176 static DEFINE_SPINLOCK(ptype_lock);
177 static struct list_head ptype_base[PTYPE_HASH_SIZE] __read_mostly; 177 static struct list_head ptype_base[PTYPE_HASH_SIZE] __read_mostly;
178 static struct list_head ptype_all __read_mostly; /* Taps */ 178 static struct list_head ptype_all __read_mostly; /* Taps */
179 179
180 /* 180 /*
181 * The @dev_base_head list is protected by @dev_base_lock and the rtnl 181 * The @dev_base_head list is protected by @dev_base_lock and the rtnl
182 * semaphore. 182 * semaphore.
183 * 183 *
184 * Pure readers hold dev_base_lock for reading, or rcu_read_lock() 184 * Pure readers hold dev_base_lock for reading, or rcu_read_lock()
185 * 185 *
186 * Writers must hold the rtnl semaphore while they loop through the 186 * Writers must hold the rtnl semaphore while they loop through the
187 * dev_base_head list, and hold dev_base_lock for writing when they do the 187 * dev_base_head list, and hold dev_base_lock for writing when they do the
188 * actual updates. This allows pure readers to access the list even 188 * actual updates. This allows pure readers to access the list even
189 * while a writer is preparing to update it. 189 * while a writer is preparing to update it.
190 * 190 *
191 * To put it another way, dev_base_lock is held for writing only to 191 * To put it another way, dev_base_lock is held for writing only to
192 * protect against pure readers; the rtnl semaphore provides the 192 * protect against pure readers; the rtnl semaphore provides the
193 * protection against other writers. 193 * protection against other writers.
194 * 194 *
195 * See, for example usages, register_netdevice() and 195 * See, for example usages, register_netdevice() and
196 * unregister_netdevice(), which must be called with the rtnl 196 * unregister_netdevice(), which must be called with the rtnl
197 * semaphore held. 197 * semaphore held.
198 */ 198 */
199 DEFINE_RWLOCK(dev_base_lock); 199 DEFINE_RWLOCK(dev_base_lock);
200 EXPORT_SYMBOL(dev_base_lock); 200 EXPORT_SYMBOL(dev_base_lock);
201 201
202 static inline struct hlist_head *dev_name_hash(struct net *net, const char *name) 202 static inline struct hlist_head *dev_name_hash(struct net *net, const char *name)
203 { 203 {
204 unsigned hash = full_name_hash(name, strnlen(name, IFNAMSIZ)); 204 unsigned hash = full_name_hash(name, strnlen(name, IFNAMSIZ));
205 return &net->dev_name_head[hash_32(hash, NETDEV_HASHBITS)]; 205 return &net->dev_name_head[hash_32(hash, NETDEV_HASHBITS)];
206 } 206 }
207 207
208 static inline struct hlist_head *dev_index_hash(struct net *net, int ifindex) 208 static inline struct hlist_head *dev_index_hash(struct net *net, int ifindex)
209 { 209 {
210 return &net->dev_index_head[ifindex & (NETDEV_HASHENTRIES - 1)]; 210 return &net->dev_index_head[ifindex & (NETDEV_HASHENTRIES - 1)];
211 } 211 }
212 212
213 static inline void rps_lock(struct softnet_data *sd) 213 static inline void rps_lock(struct softnet_data *sd)
214 { 214 {
215 #ifdef CONFIG_RPS 215 #ifdef CONFIG_RPS
216 spin_lock(&sd->input_pkt_queue.lock); 216 spin_lock(&sd->input_pkt_queue.lock);
217 #endif 217 #endif
218 } 218 }
219 219
220 static inline void rps_unlock(struct softnet_data *sd) 220 static inline void rps_unlock(struct softnet_data *sd)
221 { 221 {
222 #ifdef CONFIG_RPS 222 #ifdef CONFIG_RPS
223 spin_unlock(&sd->input_pkt_queue.lock); 223 spin_unlock(&sd->input_pkt_queue.lock);
224 #endif 224 #endif
225 } 225 }
226 226
227 /* Device list insertion */ 227 /* Device list insertion */
228 static int list_netdevice(struct net_device *dev) 228 static int list_netdevice(struct net_device *dev)
229 { 229 {
230 struct net *net = dev_net(dev); 230 struct net *net = dev_net(dev);
231 231
232 ASSERT_RTNL(); 232 ASSERT_RTNL();
233 233
234 write_lock_bh(&dev_base_lock); 234 write_lock_bh(&dev_base_lock);
235 list_add_tail_rcu(&dev->dev_list, &net->dev_base_head); 235 list_add_tail_rcu(&dev->dev_list, &net->dev_base_head);
236 hlist_add_head_rcu(&dev->name_hlist, dev_name_hash(net, dev->name)); 236 hlist_add_head_rcu(&dev->name_hlist, dev_name_hash(net, dev->name));
237 hlist_add_head_rcu(&dev->index_hlist, 237 hlist_add_head_rcu(&dev->index_hlist,
238 dev_index_hash(net, dev->ifindex)); 238 dev_index_hash(net, dev->ifindex));
239 write_unlock_bh(&dev_base_lock); 239 write_unlock_bh(&dev_base_lock);
240 return 0; 240 return 0;
241 } 241 }
242 242
243 /* Device list removal 243 /* Device list removal
244 * caller must respect a RCU grace period before freeing/reusing dev 244 * caller must respect a RCU grace period before freeing/reusing dev
245 */ 245 */
246 static void unlist_netdevice(struct net_device *dev) 246 static void unlist_netdevice(struct net_device *dev)
247 { 247 {
248 ASSERT_RTNL(); 248 ASSERT_RTNL();
249 249
250 /* Unlink dev from the device chain */ 250 /* Unlink dev from the device chain */
251 write_lock_bh(&dev_base_lock); 251 write_lock_bh(&dev_base_lock);
252 list_del_rcu(&dev->dev_list); 252 list_del_rcu(&dev->dev_list);
253 hlist_del_rcu(&dev->name_hlist); 253 hlist_del_rcu(&dev->name_hlist);
254 hlist_del_rcu(&dev->index_hlist); 254 hlist_del_rcu(&dev->index_hlist);
255 write_unlock_bh(&dev_base_lock); 255 write_unlock_bh(&dev_base_lock);
256 } 256 }
257 257
258 /* 258 /*
259 * Our notifier list 259 * Our notifier list
260 */ 260 */
261 261
262 static RAW_NOTIFIER_HEAD(netdev_chain); 262 static RAW_NOTIFIER_HEAD(netdev_chain);
263 263
264 /* 264 /*
265 * Device drivers call our routines to queue packets here. We empty the 265 * Device drivers call our routines to queue packets here. We empty the
266 * queue in the local softnet handler. 266 * queue in the local softnet handler.
267 */ 267 */
268 268
269 DEFINE_PER_CPU_ALIGNED(struct softnet_data, softnet_data); 269 DEFINE_PER_CPU_ALIGNED(struct softnet_data, softnet_data);
270 EXPORT_PER_CPU_SYMBOL(softnet_data); 270 EXPORT_PER_CPU_SYMBOL(softnet_data);
271 271
272 #ifdef CONFIG_LOCKDEP 272 #ifdef CONFIG_LOCKDEP
273 /* 273 /*
274 * register_netdevice() inits txq->_xmit_lock and sets lockdep class 274 * register_netdevice() inits txq->_xmit_lock and sets lockdep class
275 * according to dev->type 275 * according to dev->type
276 */ 276 */
277 static const unsigned short netdev_lock_type[] = 277 static const unsigned short netdev_lock_type[] =
278 {ARPHRD_NETROM, ARPHRD_ETHER, ARPHRD_EETHER, ARPHRD_AX25, 278 {ARPHRD_NETROM, ARPHRD_ETHER, ARPHRD_EETHER, ARPHRD_AX25,
279 ARPHRD_PRONET, ARPHRD_CHAOS, ARPHRD_IEEE802, ARPHRD_ARCNET, 279 ARPHRD_PRONET, ARPHRD_CHAOS, ARPHRD_IEEE802, ARPHRD_ARCNET,
280 ARPHRD_APPLETLK, ARPHRD_DLCI, ARPHRD_ATM, ARPHRD_METRICOM, 280 ARPHRD_APPLETLK, ARPHRD_DLCI, ARPHRD_ATM, ARPHRD_METRICOM,
281 ARPHRD_IEEE1394, ARPHRD_EUI64, ARPHRD_INFINIBAND, ARPHRD_SLIP, 281 ARPHRD_IEEE1394, ARPHRD_EUI64, ARPHRD_INFINIBAND, ARPHRD_SLIP,
282 ARPHRD_CSLIP, ARPHRD_SLIP6, ARPHRD_CSLIP6, ARPHRD_RSRVD, 282 ARPHRD_CSLIP, ARPHRD_SLIP6, ARPHRD_CSLIP6, ARPHRD_RSRVD,
283 ARPHRD_ADAPT, ARPHRD_ROSE, ARPHRD_X25, ARPHRD_HWX25, 283 ARPHRD_ADAPT, ARPHRD_ROSE, ARPHRD_X25, ARPHRD_HWX25,
284 ARPHRD_PPP, ARPHRD_CISCO, ARPHRD_LAPB, ARPHRD_DDCMP, 284 ARPHRD_PPP, ARPHRD_CISCO, ARPHRD_LAPB, ARPHRD_DDCMP,
285 ARPHRD_RAWHDLC, ARPHRD_TUNNEL, ARPHRD_TUNNEL6, ARPHRD_FRAD, 285 ARPHRD_RAWHDLC, ARPHRD_TUNNEL, ARPHRD_TUNNEL6, ARPHRD_FRAD,
286 ARPHRD_SKIP, ARPHRD_LOOPBACK, ARPHRD_LOCALTLK, ARPHRD_FDDI, 286 ARPHRD_SKIP, ARPHRD_LOOPBACK, ARPHRD_LOCALTLK, ARPHRD_FDDI,
287 ARPHRD_BIF, ARPHRD_SIT, ARPHRD_IPDDP, ARPHRD_IPGRE, 287 ARPHRD_BIF, ARPHRD_SIT, ARPHRD_IPDDP, ARPHRD_IPGRE,
288 ARPHRD_PIMREG, ARPHRD_HIPPI, ARPHRD_ASH, ARPHRD_ECONET, 288 ARPHRD_PIMREG, ARPHRD_HIPPI, ARPHRD_ASH, ARPHRD_ECONET,
289 ARPHRD_IRDA, ARPHRD_FCPP, ARPHRD_FCAL, ARPHRD_FCPL, 289 ARPHRD_IRDA, ARPHRD_FCPP, ARPHRD_FCAL, ARPHRD_FCPL,
290 ARPHRD_FCFABRIC, ARPHRD_IEEE802_TR, ARPHRD_IEEE80211, 290 ARPHRD_FCFABRIC, ARPHRD_IEEE802_TR, ARPHRD_IEEE80211,
291 ARPHRD_IEEE80211_PRISM, ARPHRD_IEEE80211_RADIOTAP, ARPHRD_PHONET, 291 ARPHRD_IEEE80211_PRISM, ARPHRD_IEEE80211_RADIOTAP, ARPHRD_PHONET,
292 ARPHRD_PHONET_PIPE, ARPHRD_IEEE802154, 292 ARPHRD_PHONET_PIPE, ARPHRD_IEEE802154,
293 ARPHRD_VOID, ARPHRD_NONE}; 293 ARPHRD_VOID, ARPHRD_NONE};
294 294
295 static const char *const netdev_lock_name[] = 295 static const char *const netdev_lock_name[] =
296 {"_xmit_NETROM", "_xmit_ETHER", "_xmit_EETHER", "_xmit_AX25", 296 {"_xmit_NETROM", "_xmit_ETHER", "_xmit_EETHER", "_xmit_AX25",
297 "_xmit_PRONET", "_xmit_CHAOS", "_xmit_IEEE802", "_xmit_ARCNET", 297 "_xmit_PRONET", "_xmit_CHAOS", "_xmit_IEEE802", "_xmit_ARCNET",
298 "_xmit_APPLETLK", "_xmit_DLCI", "_xmit_ATM", "_xmit_METRICOM", 298 "_xmit_APPLETLK", "_xmit_DLCI", "_xmit_ATM", "_xmit_METRICOM",
299 "_xmit_IEEE1394", "_xmit_EUI64", "_xmit_INFINIBAND", "_xmit_SLIP", 299 "_xmit_IEEE1394", "_xmit_EUI64", "_xmit_INFINIBAND", "_xmit_SLIP",
300 "_xmit_CSLIP", "_xmit_SLIP6", "_xmit_CSLIP6", "_xmit_RSRVD", 300 "_xmit_CSLIP", "_xmit_SLIP6", "_xmit_CSLIP6", "_xmit_RSRVD",
301 "_xmit_ADAPT", "_xmit_ROSE", "_xmit_X25", "_xmit_HWX25", 301 "_xmit_ADAPT", "_xmit_ROSE", "_xmit_X25", "_xmit_HWX25",
302 "_xmit_PPP", "_xmit_CISCO", "_xmit_LAPB", "_xmit_DDCMP", 302 "_xmit_PPP", "_xmit_CISCO", "_xmit_LAPB", "_xmit_DDCMP",
303 "_xmit_RAWHDLC", "_xmit_TUNNEL", "_xmit_TUNNEL6", "_xmit_FRAD", 303 "_xmit_RAWHDLC", "_xmit_TUNNEL", "_xmit_TUNNEL6", "_xmit_FRAD",
304 "_xmit_SKIP", "_xmit_LOOPBACK", "_xmit_LOCALTLK", "_xmit_FDDI", 304 "_xmit_SKIP", "_xmit_LOOPBACK", "_xmit_LOCALTLK", "_xmit_FDDI",
305 "_xmit_BIF", "_xmit_SIT", "_xmit_IPDDP", "_xmit_IPGRE", 305 "_xmit_BIF", "_xmit_SIT", "_xmit_IPDDP", "_xmit_IPGRE",
306 "_xmit_PIMREG", "_xmit_HIPPI", "_xmit_ASH", "_xmit_ECONET", 306 "_xmit_PIMREG", "_xmit_HIPPI", "_xmit_ASH", "_xmit_ECONET",
307 "_xmit_IRDA", "_xmit_FCPP", "_xmit_FCAL", "_xmit_FCPL", 307 "_xmit_IRDA", "_xmit_FCPP", "_xmit_FCAL", "_xmit_FCPL",
308 "_xmit_FCFABRIC", "_xmit_IEEE802_TR", "_xmit_IEEE80211", 308 "_xmit_FCFABRIC", "_xmit_IEEE802_TR", "_xmit_IEEE80211",
309 "_xmit_IEEE80211_PRISM", "_xmit_IEEE80211_RADIOTAP", "_xmit_PHONET", 309 "_xmit_IEEE80211_PRISM", "_xmit_IEEE80211_RADIOTAP", "_xmit_PHONET",
310 "_xmit_PHONET_PIPE", "_xmit_IEEE802154", 310 "_xmit_PHONET_PIPE", "_xmit_IEEE802154",
311 "_xmit_VOID", "_xmit_NONE"}; 311 "_xmit_VOID", "_xmit_NONE"};
312 312
313 static struct lock_class_key netdev_xmit_lock_key[ARRAY_SIZE(netdev_lock_type)]; 313 static struct lock_class_key netdev_xmit_lock_key[ARRAY_SIZE(netdev_lock_type)];
314 static struct lock_class_key netdev_addr_lock_key[ARRAY_SIZE(netdev_lock_type)]; 314 static struct lock_class_key netdev_addr_lock_key[ARRAY_SIZE(netdev_lock_type)];
315 315
316 static inline unsigned short netdev_lock_pos(unsigned short dev_type) 316 static inline unsigned short netdev_lock_pos(unsigned short dev_type)
317 { 317 {
318 int i; 318 int i;
319 319
320 for (i = 0; i < ARRAY_SIZE(netdev_lock_type); i++) 320 for (i = 0; i < ARRAY_SIZE(netdev_lock_type); i++)
321 if (netdev_lock_type[i] == dev_type) 321 if (netdev_lock_type[i] == dev_type)
322 return i; 322 return i;
323 /* the last key is used by default */ 323 /* the last key is used by default */
324 return ARRAY_SIZE(netdev_lock_type) - 1; 324 return ARRAY_SIZE(netdev_lock_type) - 1;
325 } 325 }
326 326
327 static inline void netdev_set_xmit_lockdep_class(spinlock_t *lock, 327 static inline void netdev_set_xmit_lockdep_class(spinlock_t *lock,
328 unsigned short dev_type) 328 unsigned short dev_type)
329 { 329 {
330 int i; 330 int i;
331 331
332 i = netdev_lock_pos(dev_type); 332 i = netdev_lock_pos(dev_type);
333 lockdep_set_class_and_name(lock, &netdev_xmit_lock_key[i], 333 lockdep_set_class_and_name(lock, &netdev_xmit_lock_key[i],
334 netdev_lock_name[i]); 334 netdev_lock_name[i]);
335 } 335 }
336 336
337 static inline void netdev_set_addr_lockdep_class(struct net_device *dev) 337 static inline void netdev_set_addr_lockdep_class(struct net_device *dev)
338 { 338 {
339 int i; 339 int i;
340 340
341 i = netdev_lock_pos(dev->type); 341 i = netdev_lock_pos(dev->type);
342 lockdep_set_class_and_name(&dev->addr_list_lock, 342 lockdep_set_class_and_name(&dev->addr_list_lock,
343 &netdev_addr_lock_key[i], 343 &netdev_addr_lock_key[i],
344 netdev_lock_name[i]); 344 netdev_lock_name[i]);
345 } 345 }
346 #else 346 #else
347 static inline void netdev_set_xmit_lockdep_class(spinlock_t *lock, 347 static inline void netdev_set_xmit_lockdep_class(spinlock_t *lock,
348 unsigned short dev_type) 348 unsigned short dev_type)
349 { 349 {
350 } 350 }
351 static inline void netdev_set_addr_lockdep_class(struct net_device *dev) 351 static inline void netdev_set_addr_lockdep_class(struct net_device *dev)
352 { 352 {
353 } 353 }
354 #endif 354 #endif
355 355
356 /******************************************************************************* 356 /*******************************************************************************
357 357
358 Protocol management and registration routines 358 Protocol management and registration routines
359 359
360 *******************************************************************************/ 360 *******************************************************************************/
361 361
362 /* 362 /*
363 * Add a protocol ID to the list. Now that the input handler is 363 * Add a protocol ID to the list. Now that the input handler is
364 * smarter we can dispense with all the messy stuff that used to be 364 * smarter we can dispense with all the messy stuff that used to be
365 * here. 365 * here.
366 * 366 *
367 * BEWARE!!! Protocol handlers, mangling input packets, 367 * BEWARE!!! Protocol handlers, mangling input packets,
368 * MUST BE last in hash buckets and checking protocol handlers 368 * MUST BE last in hash buckets and checking protocol handlers
369 * MUST start from promiscuous ptype_all chain in net_bh. 369 * MUST start from promiscuous ptype_all chain in net_bh.
370 * It is true now, do not change it. 370 * It is true now, do not change it.
371 * Explanation follows: if protocol handler, mangling packet, will 371 * Explanation follows: if protocol handler, mangling packet, will
372 * be the first on list, it is not able to sense, that packet 372 * be the first on list, it is not able to sense, that packet
373 * is cloned and should be copied-on-write, so that it will 373 * is cloned and should be copied-on-write, so that it will
374 * change it and subsequent readers will get broken packet. 374 * change it and subsequent readers will get broken packet.
375 * --ANK (980803) 375 * --ANK (980803)
376 */ 376 */
377 377
378 static inline struct list_head *ptype_head(const struct packet_type *pt) 378 static inline struct list_head *ptype_head(const struct packet_type *pt)
379 { 379 {
380 if (pt->type == htons(ETH_P_ALL)) 380 if (pt->type == htons(ETH_P_ALL))
381 return &ptype_all; 381 return &ptype_all;
382 else 382 else
383 return &ptype_base[ntohs(pt->type) & PTYPE_HASH_MASK]; 383 return &ptype_base[ntohs(pt->type) & PTYPE_HASH_MASK];
384 } 384 }
385 385
386 /** 386 /**
387 * dev_add_pack - add packet handler 387 * dev_add_pack - add packet handler
388 * @pt: packet type declaration 388 * @pt: packet type declaration
389 * 389 *
390 * Add a protocol handler to the networking stack. The passed &packet_type 390 * Add a protocol handler to the networking stack. The passed &packet_type
391 * is linked into kernel lists and may not be freed until it has been 391 * is linked into kernel lists and may not be freed until it has been
392 * removed from the kernel lists. 392 * removed from the kernel lists.
393 * 393 *
394 * This call does not sleep therefore it can not 394 * This call does not sleep therefore it can not
395 * guarantee all CPU's that are in middle of receiving packets 395 * guarantee all CPU's that are in middle of receiving packets
396 * will see the new packet type (until the next received packet). 396 * will see the new packet type (until the next received packet).
397 */ 397 */
398 398
399 void dev_add_pack(struct packet_type *pt) 399 void dev_add_pack(struct packet_type *pt)
400 { 400 {
401 struct list_head *head = ptype_head(pt); 401 struct list_head *head = ptype_head(pt);
402 402
403 spin_lock(&ptype_lock); 403 spin_lock(&ptype_lock);
404 list_add_rcu(&pt->list, head); 404 list_add_rcu(&pt->list, head);
405 spin_unlock(&ptype_lock); 405 spin_unlock(&ptype_lock);
406 } 406 }
407 EXPORT_SYMBOL(dev_add_pack); 407 EXPORT_SYMBOL(dev_add_pack);
408 408
409 /** 409 /**
410 * __dev_remove_pack - remove packet handler 410 * __dev_remove_pack - remove packet handler
411 * @pt: packet type declaration 411 * @pt: packet type declaration
412 * 412 *
413 * Remove a protocol handler that was previously added to the kernel 413 * Remove a protocol handler that was previously added to the kernel
414 * protocol handlers by dev_add_pack(). The passed &packet_type is removed 414 * protocol handlers by dev_add_pack(). The passed &packet_type is removed
415 * from the kernel lists and can be freed or reused once this function 415 * from the kernel lists and can be freed or reused once this function
416 * returns. 416 * returns.
417 * 417 *
418 * The packet type might still be in use by receivers 418 * The packet type might still be in use by receivers
419 * and must not be freed until after all the CPU's have gone 419 * and must not be freed until after all the CPU's have gone
420 * through a quiescent state. 420 * through a quiescent state.
421 */ 421 */
422 void __dev_remove_pack(struct packet_type *pt) 422 void __dev_remove_pack(struct packet_type *pt)
423 { 423 {
424 struct list_head *head = ptype_head(pt); 424 struct list_head *head = ptype_head(pt);
425 struct packet_type *pt1; 425 struct packet_type *pt1;
426 426
427 spin_lock(&ptype_lock); 427 spin_lock(&ptype_lock);
428 428
429 list_for_each_entry(pt1, head, list) { 429 list_for_each_entry(pt1, head, list) {
430 if (pt == pt1) { 430 if (pt == pt1) {
431 list_del_rcu(&pt->list); 431 list_del_rcu(&pt->list);
432 goto out; 432 goto out;
433 } 433 }
434 } 434 }
435 435
436 printk(KERN_WARNING "dev_remove_pack: %p not found.\n", pt); 436 printk(KERN_WARNING "dev_remove_pack: %p not found.\n", pt);
437 out: 437 out:
438 spin_unlock(&ptype_lock); 438 spin_unlock(&ptype_lock);
439 } 439 }
440 EXPORT_SYMBOL(__dev_remove_pack); 440 EXPORT_SYMBOL(__dev_remove_pack);
441 441
442 /** 442 /**
443 * dev_remove_pack - remove packet handler 443 * dev_remove_pack - remove packet handler
444 * @pt: packet type declaration 444 * @pt: packet type declaration
445 * 445 *
446 * Remove a protocol handler that was previously added to the kernel 446 * Remove a protocol handler that was previously added to the kernel
447 * protocol handlers by dev_add_pack(). The passed &packet_type is removed 447 * protocol handlers by dev_add_pack(). The passed &packet_type is removed
448 * from the kernel lists and can be freed or reused once this function 448 * from the kernel lists and can be freed or reused once this function
449 * returns. 449 * returns.
450 * 450 *
451 * This call sleeps to guarantee that no CPU is looking at the packet 451 * This call sleeps to guarantee that no CPU is looking at the packet
452 * type after return. 452 * type after return.
453 */ 453 */
454 void dev_remove_pack(struct packet_type *pt) 454 void dev_remove_pack(struct packet_type *pt)
455 { 455 {
456 __dev_remove_pack(pt); 456 __dev_remove_pack(pt);
457 457
458 synchronize_net(); 458 synchronize_net();
459 } 459 }
460 EXPORT_SYMBOL(dev_remove_pack); 460 EXPORT_SYMBOL(dev_remove_pack);
461 461
462 /****************************************************************************** 462 /******************************************************************************
463 463
464 Device Boot-time Settings Routines 464 Device Boot-time Settings Routines
465 465
466 *******************************************************************************/ 466 *******************************************************************************/
467 467
468 /* Boot time configuration table */ 468 /* Boot time configuration table */
469 static struct netdev_boot_setup dev_boot_setup[NETDEV_BOOT_SETUP_MAX]; 469 static struct netdev_boot_setup dev_boot_setup[NETDEV_BOOT_SETUP_MAX];
470 470
471 /** 471 /**
472 * netdev_boot_setup_add - add new setup entry 472 * netdev_boot_setup_add - add new setup entry
473 * @name: name of the device 473 * @name: name of the device
474 * @map: configured settings for the device 474 * @map: configured settings for the device
475 * 475 *
476 * Adds new setup entry to the dev_boot_setup list. The function 476 * Adds new setup entry to the dev_boot_setup list. The function
477 * returns 0 on error and 1 on success. This is a generic routine to 477 * returns 0 on error and 1 on success. This is a generic routine to
478 * all netdevices. 478 * all netdevices.
479 */ 479 */
480 static int netdev_boot_setup_add(char *name, struct ifmap *map) 480 static int netdev_boot_setup_add(char *name, struct ifmap *map)
481 { 481 {
482 struct netdev_boot_setup *s; 482 struct netdev_boot_setup *s;
483 int i; 483 int i;
484 484
485 s = dev_boot_setup; 485 s = dev_boot_setup;
486 for (i = 0; i < NETDEV_BOOT_SETUP_MAX; i++) { 486 for (i = 0; i < NETDEV_BOOT_SETUP_MAX; i++) {
487 if (s[i].name[0] == '\0' || s[i].name[0] == ' ') { 487 if (s[i].name[0] == '\0' || s[i].name[0] == ' ') {
488 memset(s[i].name, 0, sizeof(s[i].name)); 488 memset(s[i].name, 0, sizeof(s[i].name));
489 strlcpy(s[i].name, name, IFNAMSIZ); 489 strlcpy(s[i].name, name, IFNAMSIZ);
490 memcpy(&s[i].map, map, sizeof(s[i].map)); 490 memcpy(&s[i].map, map, sizeof(s[i].map));
491 break; 491 break;
492 } 492 }
493 } 493 }
494 494
495 return i >= NETDEV_BOOT_SETUP_MAX ? 0 : 1; 495 return i >= NETDEV_BOOT_SETUP_MAX ? 0 : 1;
496 } 496 }
497 497
498 /** 498 /**
499 * netdev_boot_setup_check - check boot time settings 499 * netdev_boot_setup_check - check boot time settings
500 * @dev: the netdevice 500 * @dev: the netdevice
501 * 501 *
502 * Check boot time settings for the device. 502 * Check boot time settings for the device.
503 * The found settings are set for the device to be used 503 * The found settings are set for the device to be used
504 * later in the device probing. 504 * later in the device probing.
505 * Returns 0 if no settings found, 1 if they are. 505 * Returns 0 if no settings found, 1 if they are.
506 */ 506 */
507 int netdev_boot_setup_check(struct net_device *dev) 507 int netdev_boot_setup_check(struct net_device *dev)
508 { 508 {
509 struct netdev_boot_setup *s = dev_boot_setup; 509 struct netdev_boot_setup *s = dev_boot_setup;
510 int i; 510 int i;
511 511
512 for (i = 0; i < NETDEV_BOOT_SETUP_MAX; i++) { 512 for (i = 0; i < NETDEV_BOOT_SETUP_MAX; i++) {
513 if (s[i].name[0] != '\0' && s[i].name[0] != ' ' && 513 if (s[i].name[0] != '\0' && s[i].name[0] != ' ' &&
514 !strcmp(dev->name, s[i].name)) { 514 !strcmp(dev->name, s[i].name)) {
515 dev->irq = s[i].map.irq; 515 dev->irq = s[i].map.irq;
516 dev->base_addr = s[i].map.base_addr; 516 dev->base_addr = s[i].map.base_addr;
517 dev->mem_start = s[i].map.mem_start; 517 dev->mem_start = s[i].map.mem_start;
518 dev->mem_end = s[i].map.mem_end; 518 dev->mem_end = s[i].map.mem_end;
519 return 1; 519 return 1;
520 } 520 }
521 } 521 }
522 return 0; 522 return 0;
523 } 523 }
524 EXPORT_SYMBOL(netdev_boot_setup_check); 524 EXPORT_SYMBOL(netdev_boot_setup_check);
525 525
526 526
527 /** 527 /**
528 * netdev_boot_base - get address from boot time settings 528 * netdev_boot_base - get address from boot time settings
529 * @prefix: prefix for network device 529 * @prefix: prefix for network device
530 * @unit: id for network device 530 * @unit: id for network device
531 * 531 *
532 * Check boot time settings for the base address of device. 532 * Check boot time settings for the base address of device.
533 * The found settings are set for the device to be used 533 * The found settings are set for the device to be used
534 * later in the device probing. 534 * later in the device probing.
535 * Returns 0 if no settings found. 535 * Returns 0 if no settings found.
536 */ 536 */
537 unsigned long netdev_boot_base(const char *prefix, int unit) 537 unsigned long netdev_boot_base(const char *prefix, int unit)
538 { 538 {
539 const struct netdev_boot_setup *s = dev_boot_setup; 539 const struct netdev_boot_setup *s = dev_boot_setup;
540 char name[IFNAMSIZ]; 540 char name[IFNAMSIZ];
541 int i; 541 int i;
542 542
543 sprintf(name, "%s%d", prefix, unit); 543 sprintf(name, "%s%d", prefix, unit);
544 544
545 /* 545 /*
546 * If device already registered then return base of 1 546 * If device already registered then return base of 1
547 * to indicate not to probe for this interface 547 * to indicate not to probe for this interface
548 */ 548 */
549 if (__dev_get_by_name(&init_net, name)) 549 if (__dev_get_by_name(&init_net, name))
550 return 1; 550 return 1;
551 551
552 for (i = 0; i < NETDEV_BOOT_SETUP_MAX; i++) 552 for (i = 0; i < NETDEV_BOOT_SETUP_MAX; i++)
553 if (!strcmp(name, s[i].name)) 553 if (!strcmp(name, s[i].name))
554 return s[i].map.base_addr; 554 return s[i].map.base_addr;
555 return 0; 555 return 0;
556 } 556 }
557 557
558 /* 558 /*
559 * Saves at boot time configured settings for any netdevice. 559 * Saves at boot time configured settings for any netdevice.
560 */ 560 */
561 int __init netdev_boot_setup(char *str) 561 int __init netdev_boot_setup(char *str)
562 { 562 {
563 int ints[5]; 563 int ints[5];
564 struct ifmap map; 564 struct ifmap map;
565 565
566 str = get_options(str, ARRAY_SIZE(ints), ints); 566 str = get_options(str, ARRAY_SIZE(ints), ints);
567 if (!str || !*str) 567 if (!str || !*str)
568 return 0; 568 return 0;
569 569
570 /* Save settings */ 570 /* Save settings */
571 memset(&map, 0, sizeof(map)); 571 memset(&map, 0, sizeof(map));
572 if (ints[0] > 0) 572 if (ints[0] > 0)
573 map.irq = ints[1]; 573 map.irq = ints[1];
574 if (ints[0] > 1) 574 if (ints[0] > 1)
575 map.base_addr = ints[2]; 575 map.base_addr = ints[2];
576 if (ints[0] > 2) 576 if (ints[0] > 2)
577 map.mem_start = ints[3]; 577 map.mem_start = ints[3];
578 if (ints[0] > 3) 578 if (ints[0] > 3)
579 map.mem_end = ints[4]; 579 map.mem_end = ints[4];
580 580
581 /* Add new entry to the list */ 581 /* Add new entry to the list */
582 return netdev_boot_setup_add(str, &map); 582 return netdev_boot_setup_add(str, &map);
583 } 583 }
584 584
585 __setup("netdev=", netdev_boot_setup); 585 __setup("netdev=", netdev_boot_setup);
586 586
587 /******************************************************************************* 587 /*******************************************************************************
588 588
589 Device Interface Subroutines 589 Device Interface Subroutines
590 590
591 *******************************************************************************/ 591 *******************************************************************************/
592 592
593 /** 593 /**
594 * __dev_get_by_name - find a device by its name 594 * __dev_get_by_name - find a device by its name
595 * @net: the applicable net namespace 595 * @net: the applicable net namespace
596 * @name: name to find 596 * @name: name to find
597 * 597 *
598 * Find an interface by name. Must be called under RTNL semaphore 598 * Find an interface by name. Must be called under RTNL semaphore
599 * or @dev_base_lock. If the name is found a pointer to the device 599 * or @dev_base_lock. If the name is found a pointer to the device
600 * is returned. If the name is not found then %NULL is returned. The 600 * is returned. If the name is not found then %NULL is returned. The
601 * reference counters are not incremented so the caller must be 601 * reference counters are not incremented so the caller must be
602 * careful with locks. 602 * careful with locks.
603 */ 603 */
604 604
605 struct net_device *__dev_get_by_name(struct net *net, const char *name) 605 struct net_device *__dev_get_by_name(struct net *net, const char *name)
606 { 606 {
607 struct hlist_node *p; 607 struct hlist_node *p;
608 struct net_device *dev; 608 struct net_device *dev;
609 struct hlist_head *head = dev_name_hash(net, name); 609 struct hlist_head *head = dev_name_hash(net, name);
610 610
611 hlist_for_each_entry(dev, p, head, name_hlist) 611 hlist_for_each_entry(dev, p, head, name_hlist)
612 if (!strncmp(dev->name, name, IFNAMSIZ)) 612 if (!strncmp(dev->name, name, IFNAMSIZ))
613 return dev; 613 return dev;
614 614
615 return NULL; 615 return NULL;
616 } 616 }
617 EXPORT_SYMBOL(__dev_get_by_name); 617 EXPORT_SYMBOL(__dev_get_by_name);
618 618
619 /** 619 /**
620 * dev_get_by_name_rcu - find a device by its name 620 * dev_get_by_name_rcu - find a device by its name
621 * @net: the applicable net namespace 621 * @net: the applicable net namespace
622 * @name: name to find 622 * @name: name to find
623 * 623 *
624 * Find an interface by name. 624 * Find an interface by name.
625 * If the name is found a pointer to the device is returned. 625 * If the name is found a pointer to the device is returned.
626 * If the name is not found then %NULL is returned. 626 * If the name is not found then %NULL is returned.
627 * The reference counters are not incremented so the caller must be 627 * The reference counters are not incremented so the caller must be
628 * careful with locks. The caller must hold RCU lock. 628 * careful with locks. The caller must hold RCU lock.
629 */ 629 */
630 630
631 struct net_device *dev_get_by_name_rcu(struct net *net, const char *name) 631 struct net_device *dev_get_by_name_rcu(struct net *net, const char *name)
632 { 632 {
633 struct hlist_node *p; 633 struct hlist_node *p;
634 struct net_device *dev; 634 struct net_device *dev;
635 struct hlist_head *head = dev_name_hash(net, name); 635 struct hlist_head *head = dev_name_hash(net, name);
636 636
637 hlist_for_each_entry_rcu(dev, p, head, name_hlist) 637 hlist_for_each_entry_rcu(dev, p, head, name_hlist)
638 if (!strncmp(dev->name, name, IFNAMSIZ)) 638 if (!strncmp(dev->name, name, IFNAMSIZ))
639 return dev; 639 return dev;
640 640
641 return NULL; 641 return NULL;
642 } 642 }
643 EXPORT_SYMBOL(dev_get_by_name_rcu); 643 EXPORT_SYMBOL(dev_get_by_name_rcu);
644 644
645 /** 645 /**
646 * dev_get_by_name - find a device by its name 646 * dev_get_by_name - find a device by its name
647 * @net: the applicable net namespace 647 * @net: the applicable net namespace
648 * @name: name to find 648 * @name: name to find
649 * 649 *
650 * Find an interface by name. This can be called from any 650 * Find an interface by name. This can be called from any
651 * context and does its own locking. The returned handle has 651 * context and does its own locking. The returned handle has
652 * the usage count incremented and the caller must use dev_put() to 652 * the usage count incremented and the caller must use dev_put() to
653 * release it when it is no longer needed. %NULL is returned if no 653 * release it when it is no longer needed. %NULL is returned if no
654 * matching device is found. 654 * matching device is found.
655 */ 655 */
656 656
657 struct net_device *dev_get_by_name(struct net *net, const char *name) 657 struct net_device *dev_get_by_name(struct net *net, const char *name)
658 { 658 {
659 struct net_device *dev; 659 struct net_device *dev;
660 660
661 rcu_read_lock(); 661 rcu_read_lock();
662 dev = dev_get_by_name_rcu(net, name); 662 dev = dev_get_by_name_rcu(net, name);
663 if (dev) 663 if (dev)
664 dev_hold(dev); 664 dev_hold(dev);
665 rcu_read_unlock(); 665 rcu_read_unlock();
666 return dev; 666 return dev;
667 } 667 }
668 EXPORT_SYMBOL(dev_get_by_name); 668 EXPORT_SYMBOL(dev_get_by_name);
669 669
670 /** 670 /**
671 * __dev_get_by_index - find a device by its ifindex 671 * __dev_get_by_index - find a device by its ifindex
672 * @net: the applicable net namespace 672 * @net: the applicable net namespace
673 * @ifindex: index of device 673 * @ifindex: index of device
674 * 674 *
675 * Search for an interface by index. Returns %NULL if the device 675 * Search for an interface by index. Returns %NULL if the device
676 * is not found or a pointer to the device. The device has not 676 * is not found or a pointer to the device. The device has not
677 * had its reference counter increased so the caller must be careful 677 * had its reference counter increased so the caller must be careful
678 * about locking. The caller must hold either the RTNL semaphore 678 * about locking. The caller must hold either the RTNL semaphore
679 * or @dev_base_lock. 679 * or @dev_base_lock.
680 */ 680 */
681 681
682 struct net_device *__dev_get_by_index(struct net *net, int ifindex) 682 struct net_device *__dev_get_by_index(struct net *net, int ifindex)
683 { 683 {
684 struct hlist_node *p; 684 struct hlist_node *p;
685 struct net_device *dev; 685 struct net_device *dev;
686 struct hlist_head *head = dev_index_hash(net, ifindex); 686 struct hlist_head *head = dev_index_hash(net, ifindex);
687 687
688 hlist_for_each_entry(dev, p, head, index_hlist) 688 hlist_for_each_entry(dev, p, head, index_hlist)
689 if (dev->ifindex == ifindex) 689 if (dev->ifindex == ifindex)
690 return dev; 690 return dev;
691 691
692 return NULL; 692 return NULL;
693 } 693 }
694 EXPORT_SYMBOL(__dev_get_by_index); 694 EXPORT_SYMBOL(__dev_get_by_index);
695 695
696 /** 696 /**
697 * dev_get_by_index_rcu - find a device by its ifindex 697 * dev_get_by_index_rcu - find a device by its ifindex
698 * @net: the applicable net namespace 698 * @net: the applicable net namespace
699 * @ifindex: index of device 699 * @ifindex: index of device
700 * 700 *
701 * Search for an interface by index. Returns %NULL if the device 701 * Search for an interface by index. Returns %NULL if the device
702 * is not found or a pointer to the device. The device has not 702 * is not found or a pointer to the device. The device has not
703 * had its reference counter increased so the caller must be careful 703 * had its reference counter increased so the caller must be careful
704 * about locking. The caller must hold RCU lock. 704 * about locking. The caller must hold RCU lock.
705 */ 705 */
706 706
707 struct net_device *dev_get_by_index_rcu(struct net *net, int ifindex) 707 struct net_device *dev_get_by_index_rcu(struct net *net, int ifindex)
708 { 708 {
709 struct hlist_node *p; 709 struct hlist_node *p;
710 struct net_device *dev; 710 struct net_device *dev;
711 struct hlist_head *head = dev_index_hash(net, ifindex); 711 struct hlist_head *head = dev_index_hash(net, ifindex);
712 712
713 hlist_for_each_entry_rcu(dev, p, head, index_hlist) 713 hlist_for_each_entry_rcu(dev, p, head, index_hlist)
714 if (dev->ifindex == ifindex) 714 if (dev->ifindex == ifindex)
715 return dev; 715 return dev;
716 716
717 return NULL; 717 return NULL;
718 } 718 }
719 EXPORT_SYMBOL(dev_get_by_index_rcu); 719 EXPORT_SYMBOL(dev_get_by_index_rcu);
720 720
721 721
722 /** 722 /**
723 * dev_get_by_index - find a device by its ifindex 723 * dev_get_by_index - find a device by its ifindex
724 * @net: the applicable net namespace 724 * @net: the applicable net namespace
725 * @ifindex: index of device 725 * @ifindex: index of device
726 * 726 *
727 * Search for an interface by index. Returns NULL if the device 727 * Search for an interface by index. Returns NULL if the device
728 * is not found or a pointer to the device. The device returned has 728 * is not found or a pointer to the device. The device returned has
729 * had a reference added and the pointer is safe until the user calls 729 * had a reference added and the pointer is safe until the user calls
730 * dev_put to indicate they have finished with it. 730 * dev_put to indicate they have finished with it.
731 */ 731 */
732 732
733 struct net_device *dev_get_by_index(struct net *net, int ifindex) 733 struct net_device *dev_get_by_index(struct net *net, int ifindex)
734 { 734 {
735 struct net_device *dev; 735 struct net_device *dev;
736 736
737 rcu_read_lock(); 737 rcu_read_lock();
738 dev = dev_get_by_index_rcu(net, ifindex); 738 dev = dev_get_by_index_rcu(net, ifindex);
739 if (dev) 739 if (dev)
740 dev_hold(dev); 740 dev_hold(dev);
741 rcu_read_unlock(); 741 rcu_read_unlock();
742 return dev; 742 return dev;
743 } 743 }
744 EXPORT_SYMBOL(dev_get_by_index); 744 EXPORT_SYMBOL(dev_get_by_index);
745 745
746 /** 746 /**
747 * dev_getbyhwaddr_rcu - find a device by its hardware address 747 * dev_getbyhwaddr_rcu - find a device by its hardware address
748 * @net: the applicable net namespace 748 * @net: the applicable net namespace
749 * @type: media type of device 749 * @type: media type of device
750 * @ha: hardware address 750 * @ha: hardware address
751 * 751 *
752 * Search for an interface by MAC address. Returns NULL if the device 752 * Search for an interface by MAC address. Returns NULL if the device
753 * is not found or a pointer to the device. 753 * is not found or a pointer to the device.
754 * The caller must hold RCU or RTNL. 754 * The caller must hold RCU or RTNL.
755 * The returned device has not had its ref count increased 755 * The returned device has not had its ref count increased
756 * and the caller must therefore be careful about locking 756 * and the caller must therefore be careful about locking
757 * 757 *
758 */ 758 */
759 759
760 struct net_device *dev_getbyhwaddr_rcu(struct net *net, unsigned short type, 760 struct net_device *dev_getbyhwaddr_rcu(struct net *net, unsigned short type,
761 const char *ha) 761 const char *ha)
762 { 762 {
763 struct net_device *dev; 763 struct net_device *dev;
764 764
765 for_each_netdev_rcu(net, dev) 765 for_each_netdev_rcu(net, dev)
766 if (dev->type == type && 766 if (dev->type == type &&
767 !memcmp(dev->dev_addr, ha, dev->addr_len)) 767 !memcmp(dev->dev_addr, ha, dev->addr_len))
768 return dev; 768 return dev;
769 769
770 return NULL; 770 return NULL;
771 } 771 }
772 EXPORT_SYMBOL(dev_getbyhwaddr_rcu); 772 EXPORT_SYMBOL(dev_getbyhwaddr_rcu);
773 773
774 struct net_device *__dev_getfirstbyhwtype(struct net *net, unsigned short type) 774 struct net_device *__dev_getfirstbyhwtype(struct net *net, unsigned short type)
775 { 775 {
776 struct net_device *dev; 776 struct net_device *dev;
777 777
778 ASSERT_RTNL(); 778 ASSERT_RTNL();
779 for_each_netdev(net, dev) 779 for_each_netdev(net, dev)
780 if (dev->type == type) 780 if (dev->type == type)
781 return dev; 781 return dev;
782 782
783 return NULL; 783 return NULL;
784 } 784 }
785 EXPORT_SYMBOL(__dev_getfirstbyhwtype); 785 EXPORT_SYMBOL(__dev_getfirstbyhwtype);
786 786
787 struct net_device *dev_getfirstbyhwtype(struct net *net, unsigned short type) 787 struct net_device *dev_getfirstbyhwtype(struct net *net, unsigned short type)
788 { 788 {
789 struct net_device *dev, *ret = NULL; 789 struct net_device *dev, *ret = NULL;
790 790
791 rcu_read_lock(); 791 rcu_read_lock();
792 for_each_netdev_rcu(net, dev) 792 for_each_netdev_rcu(net, dev)
793 if (dev->type == type) { 793 if (dev->type == type) {
794 dev_hold(dev); 794 dev_hold(dev);
795 ret = dev; 795 ret = dev;
796 break; 796 break;
797 } 797 }
798 rcu_read_unlock(); 798 rcu_read_unlock();
799 return ret; 799 return ret;
800 } 800 }
801 EXPORT_SYMBOL(dev_getfirstbyhwtype); 801 EXPORT_SYMBOL(dev_getfirstbyhwtype);
802 802
803 /** 803 /**
804 * dev_get_by_flags_rcu - find any device with given flags 804 * dev_get_by_flags_rcu - find any device with given flags
805 * @net: the applicable net namespace 805 * @net: the applicable net namespace
806 * @if_flags: IFF_* values 806 * @if_flags: IFF_* values
807 * @mask: bitmask of bits in if_flags to check 807 * @mask: bitmask of bits in if_flags to check
808 * 808 *
809 * Search for any interface with the given flags. Returns NULL if a device 809 * Search for any interface with the given flags. Returns NULL if a device
810 * is not found or a pointer to the device. Must be called inside 810 * is not found or a pointer to the device. Must be called inside
811 * rcu_read_lock(), and result refcount is unchanged. 811 * rcu_read_lock(), and result refcount is unchanged.
812 */ 812 */
813 813
814 struct net_device *dev_get_by_flags_rcu(struct net *net, unsigned short if_flags, 814 struct net_device *dev_get_by_flags_rcu(struct net *net, unsigned short if_flags,
815 unsigned short mask) 815 unsigned short mask)
816 { 816 {
817 struct net_device *dev, *ret; 817 struct net_device *dev, *ret;
818 818
819 ret = NULL; 819 ret = NULL;
820 for_each_netdev_rcu(net, dev) { 820 for_each_netdev_rcu(net, dev) {
821 if (((dev->flags ^ if_flags) & mask) == 0) { 821 if (((dev->flags ^ if_flags) & mask) == 0) {
822 ret = dev; 822 ret = dev;
823 break; 823 break;
824 } 824 }
825 } 825 }
826 return ret; 826 return ret;
827 } 827 }
828 EXPORT_SYMBOL(dev_get_by_flags_rcu); 828 EXPORT_SYMBOL(dev_get_by_flags_rcu);
829 829
830 /** 830 /**
831 * dev_valid_name - check if name is okay for network device 831 * dev_valid_name - check if name is okay for network device
832 * @name: name string 832 * @name: name string
833 * 833 *
834 * Network device names need to be valid file names to 834 * Network device names need to be valid file names to
835 * to allow sysfs to work. We also disallow any kind of 835 * to allow sysfs to work. We also disallow any kind of
836 * whitespace. 836 * whitespace.
837 */ 837 */
838 int dev_valid_name(const char *name) 838 int dev_valid_name(const char *name)
839 { 839 {
840 if (*name == '\0') 840 if (*name == '\0')
841 return 0; 841 return 0;
842 if (strlen(name) >= IFNAMSIZ) 842 if (strlen(name) >= IFNAMSIZ)
843 return 0; 843 return 0;
844 if (!strcmp(name, ".") || !strcmp(name, "..")) 844 if (!strcmp(name, ".") || !strcmp(name, ".."))
845 return 0; 845 return 0;
846 846
847 while (*name) { 847 while (*name) {
848 if (*name == '/' || isspace(*name)) 848 if (*name == '/' || isspace(*name))
849 return 0; 849 return 0;
850 name++; 850 name++;
851 } 851 }
852 return 1; 852 return 1;
853 } 853 }
854 EXPORT_SYMBOL(dev_valid_name); 854 EXPORT_SYMBOL(dev_valid_name);
855 855
856 /** 856 /**
857 * __dev_alloc_name - allocate a name for a device 857 * __dev_alloc_name - allocate a name for a device
858 * @net: network namespace to allocate the device name in 858 * @net: network namespace to allocate the device name in
859 * @name: name format string 859 * @name: name format string
860 * @buf: scratch buffer and result name string 860 * @buf: scratch buffer and result name string
861 * 861 *
862 * Passed a format string - eg "lt%d" it will try and find a suitable 862 * Passed a format string - eg "lt%d" it will try and find a suitable
863 * id. It scans list of devices to build up a free map, then chooses 863 * id. It scans list of devices to build up a free map, then chooses
864 * the first empty slot. The caller must hold the dev_base or rtnl lock 864 * the first empty slot. The caller must hold the dev_base or rtnl lock
865 * while allocating the name and adding the device in order to avoid 865 * while allocating the name and adding the device in order to avoid
866 * duplicates. 866 * duplicates.
867 * Limited to bits_per_byte * page size devices (ie 32K on most platforms). 867 * Limited to bits_per_byte * page size devices (ie 32K on most platforms).
868 * Returns the number of the unit assigned or a negative errno code. 868 * Returns the number of the unit assigned or a negative errno code.
869 */ 869 */
870 870
871 static int __dev_alloc_name(struct net *net, const char *name, char *buf) 871 static int __dev_alloc_name(struct net *net, const char *name, char *buf)
872 { 872 {
873 int i = 0; 873 int i = 0;
874 const char *p; 874 const char *p;
875 const int max_netdevices = 8*PAGE_SIZE; 875 const int max_netdevices = 8*PAGE_SIZE;
876 unsigned long *inuse; 876 unsigned long *inuse;
877 struct net_device *d; 877 struct net_device *d;
878 878
879 p = strnchr(name, IFNAMSIZ-1, '%'); 879 p = strnchr(name, IFNAMSIZ-1, '%');
880 if (p) { 880 if (p) {
881 /* 881 /*
882 * Verify the string as this thing may have come from 882 * Verify the string as this thing may have come from
883 * the user. There must be either one "%d" and no other "%" 883 * the user. There must be either one "%d" and no other "%"
884 * characters. 884 * characters.
885 */ 885 */
886 if (p[1] != 'd' || strchr(p + 2, '%')) 886 if (p[1] != 'd' || strchr(p + 2, '%'))
887 return -EINVAL; 887 return -EINVAL;
888 888
889 /* Use one page as a bit array of possible slots */ 889 /* Use one page as a bit array of possible slots */
890 inuse = (unsigned long *) get_zeroed_page(GFP_ATOMIC); 890 inuse = (unsigned long *) get_zeroed_page(GFP_ATOMIC);
891 if (!inuse) 891 if (!inuse)
892 return -ENOMEM; 892 return -ENOMEM;
893 893
894 for_each_netdev(net, d) { 894 for_each_netdev(net, d) {
895 if (!sscanf(d->name, name, &i)) 895 if (!sscanf(d->name, name, &i))
896 continue; 896 continue;
897 if (i < 0 || i >= max_netdevices) 897 if (i < 0 || i >= max_netdevices)
898 continue; 898 continue;
899 899
900 /* avoid cases where sscanf is not exact inverse of printf */ 900 /* avoid cases where sscanf is not exact inverse of printf */
901 snprintf(buf, IFNAMSIZ, name, i); 901 snprintf(buf, IFNAMSIZ, name, i);
902 if (!strncmp(buf, d->name, IFNAMSIZ)) 902 if (!strncmp(buf, d->name, IFNAMSIZ))
903 set_bit(i, inuse); 903 set_bit(i, inuse);
904 } 904 }
905 905
906 i = find_first_zero_bit(inuse, max_netdevices); 906 i = find_first_zero_bit(inuse, max_netdevices);
907 free_page((unsigned long) inuse); 907 free_page((unsigned long) inuse);
908 } 908 }
909 909
910 if (buf != name) 910 if (buf != name)
911 snprintf(buf, IFNAMSIZ, name, i); 911 snprintf(buf, IFNAMSIZ, name, i);
912 if (!__dev_get_by_name(net, buf)) 912 if (!__dev_get_by_name(net, buf))
913 return i; 913 return i;
914 914
915 /* It is possible to run out of possible slots 915 /* It is possible to run out of possible slots
916 * when the name is long and there isn't enough space left 916 * when the name is long and there isn't enough space left
917 * for the digits, or if all bits are used. 917 * for the digits, or if all bits are used.
918 */ 918 */
919 return -ENFILE; 919 return -ENFILE;
920 } 920 }
921 921
922 /** 922 /**
923 * dev_alloc_name - allocate a name for a device 923 * dev_alloc_name - allocate a name for a device
924 * @dev: device 924 * @dev: device
925 * @name: name format string 925 * @name: name format string
926 * 926 *
927 * Passed a format string - eg "lt%d" it will try and find a suitable 927 * Passed a format string - eg "lt%d" it will try and find a suitable
928 * id. It scans list of devices to build up a free map, then chooses 928 * id. It scans list of devices to build up a free map, then chooses
929 * the first empty slot. The caller must hold the dev_base or rtnl lock 929 * the first empty slot. The caller must hold the dev_base or rtnl lock
930 * while allocating the name and adding the device in order to avoid 930 * while allocating the name and adding the device in order to avoid
931 * duplicates. 931 * duplicates.
932 * Limited to bits_per_byte * page size devices (ie 32K on most platforms). 932 * Limited to bits_per_byte * page size devices (ie 32K on most platforms).
933 * Returns the number of the unit assigned or a negative errno code. 933 * Returns the number of the unit assigned or a negative errno code.
934 */ 934 */
935 935
936 int dev_alloc_name(struct net_device *dev, const char *name) 936 int dev_alloc_name(struct net_device *dev, const char *name)
937 { 937 {
938 char buf[IFNAMSIZ]; 938 char buf[IFNAMSIZ];
939 struct net *net; 939 struct net *net;
940 int ret; 940 int ret;
941 941
942 BUG_ON(!dev_net(dev)); 942 BUG_ON(!dev_net(dev));
943 net = dev_net(dev); 943 net = dev_net(dev);
944 ret = __dev_alloc_name(net, name, buf); 944 ret = __dev_alloc_name(net, name, buf);
945 if (ret >= 0) 945 if (ret >= 0)
946 strlcpy(dev->name, buf, IFNAMSIZ); 946 strlcpy(dev->name, buf, IFNAMSIZ);
947 return ret; 947 return ret;
948 } 948 }
949 EXPORT_SYMBOL(dev_alloc_name); 949 EXPORT_SYMBOL(dev_alloc_name);
950 950
951 static int dev_get_valid_name(struct net_device *dev, const char *name) 951 static int dev_get_valid_name(struct net_device *dev, const char *name)
952 { 952 {
953 struct net *net; 953 struct net *net;
954 954
955 BUG_ON(!dev_net(dev)); 955 BUG_ON(!dev_net(dev));
956 net = dev_net(dev); 956 net = dev_net(dev);
957 957
958 if (!dev_valid_name(name)) 958 if (!dev_valid_name(name))
959 return -EINVAL; 959 return -EINVAL;
960 960
961 if (strchr(name, '%')) 961 if (strchr(name, '%'))
962 return dev_alloc_name(dev, name); 962 return dev_alloc_name(dev, name);
963 else if (__dev_get_by_name(net, name)) 963 else if (__dev_get_by_name(net, name))
964 return -EEXIST; 964 return -EEXIST;
965 else if (dev->name != name) 965 else if (dev->name != name)
966 strlcpy(dev->name, name, IFNAMSIZ); 966 strlcpy(dev->name, name, IFNAMSIZ);
967 967
968 return 0; 968 return 0;
969 } 969 }
970 970
971 /** 971 /**
972 * dev_change_name - change name of a device 972 * dev_change_name - change name of a device
973 * @dev: device 973 * @dev: device
974 * @newname: name (or format string) must be at least IFNAMSIZ 974 * @newname: name (or format string) must be at least IFNAMSIZ
975 * 975 *
976 * Change name of a device, can pass format strings "eth%d". 976 * Change name of a device, can pass format strings "eth%d".
977 * for wildcarding. 977 * for wildcarding.
978 */ 978 */
979 int dev_change_name(struct net_device *dev, const char *newname) 979 int dev_change_name(struct net_device *dev, const char *newname)
980 { 980 {
981 char oldname[IFNAMSIZ]; 981 char oldname[IFNAMSIZ];
982 int err = 0; 982 int err = 0;
983 int ret; 983 int ret;
984 struct net *net; 984 struct net *net;
985 985
986 ASSERT_RTNL(); 986 ASSERT_RTNL();
987 BUG_ON(!dev_net(dev)); 987 BUG_ON(!dev_net(dev));
988 988
989 net = dev_net(dev); 989 net = dev_net(dev);
990 if (dev->flags & IFF_UP) 990 if (dev->flags & IFF_UP)
991 return -EBUSY; 991 return -EBUSY;
992 992
993 if (strncmp(newname, dev->name, IFNAMSIZ) == 0) 993 if (strncmp(newname, dev->name, IFNAMSIZ) == 0)
994 return 0; 994 return 0;
995 995
996 memcpy(oldname, dev->name, IFNAMSIZ); 996 memcpy(oldname, dev->name, IFNAMSIZ);
997 997
998 err = dev_get_valid_name(dev, newname); 998 err = dev_get_valid_name(dev, newname);
999 if (err < 0) 999 if (err < 0)
1000 return err; 1000 return err;
1001 1001
1002 rollback: 1002 rollback:
1003 ret = device_rename(&dev->dev, dev->name); 1003 ret = device_rename(&dev->dev, dev->name);
1004 if (ret) { 1004 if (ret) {
1005 memcpy(dev->name, oldname, IFNAMSIZ); 1005 memcpy(dev->name, oldname, IFNAMSIZ);
1006 return ret; 1006 return ret;
1007 } 1007 }
1008 1008
1009 write_lock_bh(&dev_base_lock); 1009 write_lock_bh(&dev_base_lock);
1010 hlist_del_rcu(&dev->name_hlist); 1010 hlist_del_rcu(&dev->name_hlist);
1011 write_unlock_bh(&dev_base_lock); 1011 write_unlock_bh(&dev_base_lock);
1012 1012
1013 synchronize_rcu(); 1013 synchronize_rcu();
1014 1014
1015 write_lock_bh(&dev_base_lock); 1015 write_lock_bh(&dev_base_lock);
1016 hlist_add_head_rcu(&dev->name_hlist, dev_name_hash(net, dev->name)); 1016 hlist_add_head_rcu(&dev->name_hlist, dev_name_hash(net, dev->name));
1017 write_unlock_bh(&dev_base_lock); 1017 write_unlock_bh(&dev_base_lock);
1018 1018
1019 ret = call_netdevice_notifiers(NETDEV_CHANGENAME, dev); 1019 ret = call_netdevice_notifiers(NETDEV_CHANGENAME, dev);
1020 ret = notifier_to_errno(ret); 1020 ret = notifier_to_errno(ret);
1021 1021
1022 if (ret) { 1022 if (ret) {
1023 /* err >= 0 after dev_alloc_name() or stores the first errno */ 1023 /* err >= 0 after dev_alloc_name() or stores the first errno */
1024 if (err >= 0) { 1024 if (err >= 0) {
1025 err = ret; 1025 err = ret;
1026 memcpy(dev->name, oldname, IFNAMSIZ); 1026 memcpy(dev->name, oldname, IFNAMSIZ);
1027 goto rollback; 1027 goto rollback;
1028 } else { 1028 } else {
1029 printk(KERN_ERR 1029 printk(KERN_ERR
1030 "%s: name change rollback failed: %d.\n", 1030 "%s: name change rollback failed: %d.\n",
1031 dev->name, ret); 1031 dev->name, ret);
1032 } 1032 }
1033 } 1033 }
1034 1034
1035 return err; 1035 return err;
1036 } 1036 }
1037 1037
1038 /** 1038 /**
1039 * dev_set_alias - change ifalias of a device 1039 * dev_set_alias - change ifalias of a device
1040 * @dev: device 1040 * @dev: device
1041 * @alias: name up to IFALIASZ 1041 * @alias: name up to IFALIASZ
1042 * @len: limit of bytes to copy from info 1042 * @len: limit of bytes to copy from info
1043 * 1043 *
1044 * Set ifalias for a device, 1044 * Set ifalias for a device,
1045 */ 1045 */
1046 int dev_set_alias(struct net_device *dev, const char *alias, size_t len) 1046 int dev_set_alias(struct net_device *dev, const char *alias, size_t len)
1047 { 1047 {
1048 ASSERT_RTNL(); 1048 ASSERT_RTNL();
1049 1049
1050 if (len >= IFALIASZ) 1050 if (len >= IFALIASZ)
1051 return -EINVAL; 1051 return -EINVAL;
1052 1052
1053 if (!len) { 1053 if (!len) {
1054 if (dev->ifalias) { 1054 if (dev->ifalias) {
1055 kfree(dev->ifalias); 1055 kfree(dev->ifalias);
1056 dev->ifalias = NULL; 1056 dev->ifalias = NULL;
1057 } 1057 }
1058 return 0; 1058 return 0;
1059 } 1059 }
1060 1060
1061 dev->ifalias = krealloc(dev->ifalias, len + 1, GFP_KERNEL); 1061 dev->ifalias = krealloc(dev->ifalias, len + 1, GFP_KERNEL);
1062 if (!dev->ifalias) 1062 if (!dev->ifalias)
1063 return -ENOMEM; 1063 return -ENOMEM;
1064 1064
1065 strlcpy(dev->ifalias, alias, len+1); 1065 strlcpy(dev->ifalias, alias, len+1);
1066 return len; 1066 return len;
1067 } 1067 }
1068 1068
1069 1069
1070 /** 1070 /**
1071 * netdev_features_change - device changes features 1071 * netdev_features_change - device changes features
1072 * @dev: device to cause notification 1072 * @dev: device to cause notification
1073 * 1073 *
1074 * Called to indicate a device has changed features. 1074 * Called to indicate a device has changed features.
1075 */ 1075 */
1076 void netdev_features_change(struct net_device *dev) 1076 void netdev_features_change(struct net_device *dev)
1077 { 1077 {
1078 call_netdevice_notifiers(NETDEV_FEAT_CHANGE, dev); 1078 call_netdevice_notifiers(NETDEV_FEAT_CHANGE, dev);
1079 } 1079 }
1080 EXPORT_SYMBOL(netdev_features_change); 1080 EXPORT_SYMBOL(netdev_features_change);
1081 1081
1082 /** 1082 /**
1083 * netdev_state_change - device changes state 1083 * netdev_state_change - device changes state
1084 * @dev: device to cause notification 1084 * @dev: device to cause notification
1085 * 1085 *
1086 * Called to indicate a device has changed state. This function calls 1086 * Called to indicate a device has changed state. This function calls
1087 * the notifier chains for netdev_chain and sends a NEWLINK message 1087 * the notifier chains for netdev_chain and sends a NEWLINK message
1088 * to the routing socket. 1088 * to the routing socket.
1089 */ 1089 */
1090 void netdev_state_change(struct net_device *dev) 1090 void netdev_state_change(struct net_device *dev)
1091 { 1091 {
1092 if (dev->flags & IFF_UP) { 1092 if (dev->flags & IFF_UP) {
1093 call_netdevice_notifiers(NETDEV_CHANGE, dev); 1093 call_netdevice_notifiers(NETDEV_CHANGE, dev);
1094 rtmsg_ifinfo(RTM_NEWLINK, dev, 0); 1094 rtmsg_ifinfo(RTM_NEWLINK, dev, 0);
1095 } 1095 }
1096 } 1096 }
1097 EXPORT_SYMBOL(netdev_state_change); 1097 EXPORT_SYMBOL(netdev_state_change);
1098 1098
1099 int netdev_bonding_change(struct net_device *dev, unsigned long event) 1099 int netdev_bonding_change(struct net_device *dev, unsigned long event)
1100 { 1100 {
1101 return call_netdevice_notifiers(event, dev); 1101 return call_netdevice_notifiers(event, dev);
1102 } 1102 }
1103 EXPORT_SYMBOL(netdev_bonding_change); 1103 EXPORT_SYMBOL(netdev_bonding_change);
1104 1104
1105 /** 1105 /**
1106 * dev_load - load a network module 1106 * dev_load - load a network module
1107 * @net: the applicable net namespace 1107 * @net: the applicable net namespace
1108 * @name: name of interface 1108 * @name: name of interface
1109 * 1109 *
1110 * If a network interface is not present and the process has suitable 1110 * If a network interface is not present and the process has suitable
1111 * privileges this function loads the module. If module loading is not 1111 * privileges this function loads the module. If module loading is not
1112 * available in this kernel then it becomes a nop. 1112 * available in this kernel then it becomes a nop.
1113 */ 1113 */
1114 1114
1115 void dev_load(struct net *net, const char *name) 1115 void dev_load(struct net *net, const char *name)
1116 { 1116 {
1117 struct net_device *dev; 1117 struct net_device *dev;
1118 int no_module; 1118 int no_module;
1119 1119
1120 rcu_read_lock(); 1120 rcu_read_lock();
1121 dev = dev_get_by_name_rcu(net, name); 1121 dev = dev_get_by_name_rcu(net, name);
1122 rcu_read_unlock(); 1122 rcu_read_unlock();
1123 1123
1124 no_module = !dev; 1124 no_module = !dev;
1125 if (no_module && capable(CAP_NET_ADMIN)) 1125 if (no_module && capable(CAP_NET_ADMIN))
1126 no_module = request_module("netdev-%s", name); 1126 no_module = request_module("netdev-%s", name);
1127 if (no_module && capable(CAP_SYS_MODULE)) { 1127 if (no_module && capable(CAP_SYS_MODULE)) {
1128 if (!request_module("%s", name)) 1128 if (!request_module("%s", name))
1129 pr_err("Loading kernel module for a network device " 1129 pr_err("Loading kernel module for a network device "
1130 "with CAP_SYS_MODULE (deprecated). Use CAP_NET_ADMIN and alias netdev-%s " 1130 "with CAP_SYS_MODULE (deprecated). Use CAP_NET_ADMIN and alias netdev-%s "
1131 "instead\n", name); 1131 "instead\n", name);
1132 } 1132 }
1133 } 1133 }
1134 EXPORT_SYMBOL(dev_load); 1134 EXPORT_SYMBOL(dev_load);
1135 1135
1136 static int __dev_open(struct net_device *dev) 1136 static int __dev_open(struct net_device *dev)
1137 { 1137 {
1138 const struct net_device_ops *ops = dev->netdev_ops; 1138 const struct net_device_ops *ops = dev->netdev_ops;
1139 int ret; 1139 int ret;
1140 1140
1141 ASSERT_RTNL(); 1141 ASSERT_RTNL();
1142 1142
1143 if (!netif_device_present(dev)) 1143 if (!netif_device_present(dev))
1144 return -ENODEV; 1144 return -ENODEV;
1145 1145
1146 ret = call_netdevice_notifiers(NETDEV_PRE_UP, dev); 1146 ret = call_netdevice_notifiers(NETDEV_PRE_UP, dev);
1147 ret = notifier_to_errno(ret); 1147 ret = notifier_to_errno(ret);
1148 if (ret) 1148 if (ret)
1149 return ret; 1149 return ret;
1150 1150
1151 set_bit(__LINK_STATE_START, &dev->state); 1151 set_bit(__LINK_STATE_START, &dev->state);
1152 1152
1153 if (ops->ndo_validate_addr) 1153 if (ops->ndo_validate_addr)
1154 ret = ops->ndo_validate_addr(dev); 1154 ret = ops->ndo_validate_addr(dev);
1155 1155
1156 if (!ret && ops->ndo_open) 1156 if (!ret && ops->ndo_open)
1157 ret = ops->ndo_open(dev); 1157 ret = ops->ndo_open(dev);
1158 1158
1159 if (ret) 1159 if (ret)
1160 clear_bit(__LINK_STATE_START, &dev->state); 1160 clear_bit(__LINK_STATE_START, &dev->state);
1161 else { 1161 else {
1162 dev->flags |= IFF_UP; 1162 dev->flags |= IFF_UP;
1163 net_dmaengine_get(); 1163 net_dmaengine_get();
1164 dev_set_rx_mode(dev); 1164 dev_set_rx_mode(dev);
1165 dev_activate(dev); 1165 dev_activate(dev);
1166 } 1166 }
1167 1167
1168 return ret; 1168 return ret;
1169 } 1169 }
1170 1170
1171 /** 1171 /**
1172 * dev_open - prepare an interface for use. 1172 * dev_open - prepare an interface for use.
1173 * @dev: device to open 1173 * @dev: device to open
1174 * 1174 *
1175 * Takes a device from down to up state. The device's private open 1175 * Takes a device from down to up state. The device's private open
1176 * function is invoked and then the multicast lists are loaded. Finally 1176 * function is invoked and then the multicast lists are loaded. Finally
1177 * the device is moved into the up state and a %NETDEV_UP message is 1177 * the device is moved into the up state and a %NETDEV_UP message is
1178 * sent to the netdev notifier chain. 1178 * sent to the netdev notifier chain.
1179 * 1179 *
1180 * Calling this function on an active interface is a nop. On a failure 1180 * Calling this function on an active interface is a nop. On a failure
1181 * a negative errno code is returned. 1181 * a negative errno code is returned.
1182 */ 1182 */
1183 int dev_open(struct net_device *dev) 1183 int dev_open(struct net_device *dev)
1184 { 1184 {
1185 int ret; 1185 int ret;
1186 1186
1187 if (dev->flags & IFF_UP) 1187 if (dev->flags & IFF_UP)
1188 return 0; 1188 return 0;
1189 1189
1190 ret = __dev_open(dev); 1190 ret = __dev_open(dev);
1191 if (ret < 0) 1191 if (ret < 0)
1192 return ret; 1192 return ret;
1193 1193
1194 rtmsg_ifinfo(RTM_NEWLINK, dev, IFF_UP|IFF_RUNNING); 1194 rtmsg_ifinfo(RTM_NEWLINK, dev, IFF_UP|IFF_RUNNING);
1195 call_netdevice_notifiers(NETDEV_UP, dev); 1195 call_netdevice_notifiers(NETDEV_UP, dev);
1196 1196
1197 return ret; 1197 return ret;
1198 } 1198 }
1199 EXPORT_SYMBOL(dev_open); 1199 EXPORT_SYMBOL(dev_open);
1200 1200
1201 static int __dev_close_many(struct list_head *head) 1201 static int __dev_close_many(struct list_head *head)
1202 { 1202 {
1203 struct net_device *dev; 1203 struct net_device *dev;
1204 1204
1205 ASSERT_RTNL(); 1205 ASSERT_RTNL();
1206 might_sleep(); 1206 might_sleep();
1207 1207
1208 list_for_each_entry(dev, head, unreg_list) { 1208 list_for_each_entry(dev, head, unreg_list) {
1209 call_netdevice_notifiers(NETDEV_GOING_DOWN, dev); 1209 call_netdevice_notifiers(NETDEV_GOING_DOWN, dev);
1210 1210
1211 clear_bit(__LINK_STATE_START, &dev->state); 1211 clear_bit(__LINK_STATE_START, &dev->state);
1212 1212
1213 /* Synchronize to scheduled poll. We cannot touch poll list, it 1213 /* Synchronize to scheduled poll. We cannot touch poll list, it
1214 * can be even on different cpu. So just clear netif_running(). 1214 * can be even on different cpu. So just clear netif_running().
1215 * 1215 *
1216 * dev->stop() will invoke napi_disable() on all of it's 1216 * dev->stop() will invoke napi_disable() on all of it's
1217 * napi_struct instances on this device. 1217 * napi_struct instances on this device.
1218 */ 1218 */
1219 smp_mb__after_clear_bit(); /* Commit netif_running(). */ 1219 smp_mb__after_clear_bit(); /* Commit netif_running(). */
1220 } 1220 }
1221 1221
1222 dev_deactivate_many(head); 1222 dev_deactivate_many(head);
1223 1223
1224 list_for_each_entry(dev, head, unreg_list) { 1224 list_for_each_entry(dev, head, unreg_list) {
1225 const struct net_device_ops *ops = dev->netdev_ops; 1225 const struct net_device_ops *ops = dev->netdev_ops;
1226 1226
1227 /* 1227 /*
1228 * Call the device specific close. This cannot fail. 1228 * Call the device specific close. This cannot fail.
1229 * Only if device is UP 1229 * Only if device is UP
1230 * 1230 *
1231 * We allow it to be called even after a DETACH hot-plug 1231 * We allow it to be called even after a DETACH hot-plug
1232 * event. 1232 * event.
1233 */ 1233 */
1234 if (ops->ndo_stop) 1234 if (ops->ndo_stop)
1235 ops->ndo_stop(dev); 1235 ops->ndo_stop(dev);
1236 1236
1237 dev->flags &= ~IFF_UP; 1237 dev->flags &= ~IFF_UP;
1238 net_dmaengine_put(); 1238 net_dmaengine_put();
1239 } 1239 }
1240 1240
1241 return 0; 1241 return 0;
1242 } 1242 }
1243 1243
1244 static int __dev_close(struct net_device *dev) 1244 static int __dev_close(struct net_device *dev)
1245 { 1245 {
1246 int retval; 1246 int retval;
1247 LIST_HEAD(single); 1247 LIST_HEAD(single);
1248 1248
1249 list_add(&dev->unreg_list, &single); 1249 list_add(&dev->unreg_list, &single);
1250 retval = __dev_close_many(&single); 1250 retval = __dev_close_many(&single);
1251 list_del(&single); 1251 list_del(&single);
1252 return retval; 1252 return retval;
1253 } 1253 }
1254 1254
1255 static int dev_close_many(struct list_head *head) 1255 static int dev_close_many(struct list_head *head)
1256 { 1256 {
1257 struct net_device *dev, *tmp; 1257 struct net_device *dev, *tmp;
1258 LIST_HEAD(tmp_list); 1258 LIST_HEAD(tmp_list);
1259 1259
1260 list_for_each_entry_safe(dev, tmp, head, unreg_list) 1260 list_for_each_entry_safe(dev, tmp, head, unreg_list)
1261 if (!(dev->flags & IFF_UP)) 1261 if (!(dev->flags & IFF_UP))
1262 list_move(&dev->unreg_list, &tmp_list); 1262 list_move(&dev->unreg_list, &tmp_list);
1263 1263
1264 __dev_close_many(head); 1264 __dev_close_many(head);
1265 1265
1266 list_for_each_entry(dev, head, unreg_list) { 1266 list_for_each_entry(dev, head, unreg_list) {
1267 rtmsg_ifinfo(RTM_NEWLINK, dev, IFF_UP|IFF_RUNNING); 1267 rtmsg_ifinfo(RTM_NEWLINK, dev, IFF_UP|IFF_RUNNING);
1268 call_netdevice_notifiers(NETDEV_DOWN, dev); 1268 call_netdevice_notifiers(NETDEV_DOWN, dev);
1269 } 1269 }
1270 1270
1271 /* rollback_registered_many needs the complete original list */ 1271 /* rollback_registered_many needs the complete original list */
1272 list_splice(&tmp_list, head); 1272 list_splice(&tmp_list, head);
1273 return 0; 1273 return 0;
1274 } 1274 }
1275 1275
1276 /** 1276 /**
1277 * dev_close - shutdown an interface. 1277 * dev_close - shutdown an interface.
1278 * @dev: device to shutdown 1278 * @dev: device to shutdown
1279 * 1279 *
1280 * This function moves an active device into down state. A 1280 * This function moves an active device into down state. A
1281 * %NETDEV_GOING_DOWN is sent to the netdev notifier chain. The device 1281 * %NETDEV_GOING_DOWN is sent to the netdev notifier chain. The device
1282 * is then deactivated and finally a %NETDEV_DOWN is sent to the notifier 1282 * is then deactivated and finally a %NETDEV_DOWN is sent to the notifier
1283 * chain. 1283 * chain.
1284 */ 1284 */
1285 int dev_close(struct net_device *dev) 1285 int dev_close(struct net_device *dev)
1286 { 1286 {
1287 if (dev->flags & IFF_UP) { 1287 if (dev->flags & IFF_UP) {
1288 LIST_HEAD(single); 1288 LIST_HEAD(single);
1289 1289
1290 list_add(&dev->unreg_list, &single); 1290 list_add(&dev->unreg_list, &single);
1291 dev_close_many(&single); 1291 dev_close_many(&single);
1292 list_del(&single); 1292 list_del(&single);
1293 } 1293 }
1294 return 0; 1294 return 0;
1295 } 1295 }
1296 EXPORT_SYMBOL(dev_close); 1296 EXPORT_SYMBOL(dev_close);
1297 1297
1298 1298
1299 /** 1299 /**
1300 * dev_disable_lro - disable Large Receive Offload on a device 1300 * dev_disable_lro - disable Large Receive Offload on a device
1301 * @dev: device 1301 * @dev: device
1302 * 1302 *
1303 * Disable Large Receive Offload (LRO) on a net device. Must be 1303 * Disable Large Receive Offload (LRO) on a net device. Must be
1304 * called under RTNL. This is needed if received packets may be 1304 * called under RTNL. This is needed if received packets may be
1305 * forwarded to another interface. 1305 * forwarded to another interface.
1306 */ 1306 */
1307 void dev_disable_lro(struct net_device *dev) 1307 void dev_disable_lro(struct net_device *dev)
1308 { 1308 {
1309 u32 flags; 1309 u32 flags;
1310 1310
1311 /*
1312 * If we're trying to disable lro on a vlan device
1313 * use the underlying physical device instead
1314 */
1315 if (is_vlan_dev(dev))
1316 dev = vlan_dev_real_dev(dev);
1317
1311 if (dev->ethtool_ops && dev->ethtool_ops->get_flags) 1318 if (dev->ethtool_ops && dev->ethtool_ops->get_flags)
1312 flags = dev->ethtool_ops->get_flags(dev); 1319 flags = dev->ethtool_ops->get_flags(dev);
1313 else 1320 else
1314 flags = ethtool_op_get_flags(dev); 1321 flags = ethtool_op_get_flags(dev);
1315 1322
1316 if (!(flags & ETH_FLAG_LRO)) 1323 if (!(flags & ETH_FLAG_LRO))
1317 return; 1324 return;
1318 1325
1319 __ethtool_set_flags(dev, flags & ~ETH_FLAG_LRO); 1326 __ethtool_set_flags(dev, flags & ~ETH_FLAG_LRO);
1320 if (unlikely(dev->features & NETIF_F_LRO)) 1327 if (unlikely(dev->features & NETIF_F_LRO))
1321 netdev_WARN(dev, "failed to disable LRO!\n"); 1328 netdev_WARN(dev, "failed to disable LRO!\n");
1322 } 1329 }
1323 EXPORT_SYMBOL(dev_disable_lro); 1330 EXPORT_SYMBOL(dev_disable_lro);
1324 1331
1325 1332
1326 static int dev_boot_phase = 1; 1333 static int dev_boot_phase = 1;
1327 1334
1328 /** 1335 /**
1329 * register_netdevice_notifier - register a network notifier block 1336 * register_netdevice_notifier - register a network notifier block
1330 * @nb: notifier 1337 * @nb: notifier
1331 * 1338 *
1332 * Register a notifier to be called when network device events occur. 1339 * Register a notifier to be called when network device events occur.
1333 * The notifier passed is linked into the kernel structures and must 1340 * The notifier passed is linked into the kernel structures and must
1334 * not be reused until it has been unregistered. A negative errno code 1341 * not be reused until it has been unregistered. A negative errno code
1335 * is returned on a failure. 1342 * is returned on a failure.
1336 * 1343 *
1337 * When registered all registration and up events are replayed 1344 * When registered all registration and up events are replayed
1338 * to the new notifier to allow device to have a race free 1345 * to the new notifier to allow device to have a race free
1339 * view of the network device list. 1346 * view of the network device list.
1340 */ 1347 */
1341 1348
1342 int register_netdevice_notifier(struct notifier_block *nb) 1349 int register_netdevice_notifier(struct notifier_block *nb)
1343 { 1350 {
1344 struct net_device *dev; 1351 struct net_device *dev;
1345 struct net_device *last; 1352 struct net_device *last;
1346 struct net *net; 1353 struct net *net;
1347 int err; 1354 int err;
1348 1355
1349 rtnl_lock(); 1356 rtnl_lock();
1350 err = raw_notifier_chain_register(&netdev_chain, nb); 1357 err = raw_notifier_chain_register(&netdev_chain, nb);
1351 if (err) 1358 if (err)
1352 goto unlock; 1359 goto unlock;
1353 if (dev_boot_phase) 1360 if (dev_boot_phase)
1354 goto unlock; 1361 goto unlock;
1355 for_each_net(net) { 1362 for_each_net(net) {
1356 for_each_netdev(net, dev) { 1363 for_each_netdev(net, dev) {
1357 err = nb->notifier_call(nb, NETDEV_REGISTER, dev); 1364 err = nb->notifier_call(nb, NETDEV_REGISTER, dev);
1358 err = notifier_to_errno(err); 1365 err = notifier_to_errno(err);
1359 if (err) 1366 if (err)
1360 goto rollback; 1367 goto rollback;
1361 1368
1362 if (!(dev->flags & IFF_UP)) 1369 if (!(dev->flags & IFF_UP))
1363 continue; 1370 continue;
1364 1371
1365 nb->notifier_call(nb, NETDEV_UP, dev); 1372 nb->notifier_call(nb, NETDEV_UP, dev);
1366 } 1373 }
1367 } 1374 }
1368 1375
1369 unlock: 1376 unlock:
1370 rtnl_unlock(); 1377 rtnl_unlock();
1371 return err; 1378 return err;
1372 1379
1373 rollback: 1380 rollback:
1374 last = dev; 1381 last = dev;
1375 for_each_net(net) { 1382 for_each_net(net) {
1376 for_each_netdev(net, dev) { 1383 for_each_netdev(net, dev) {
1377 if (dev == last) 1384 if (dev == last)
1378 break; 1385 break;
1379 1386
1380 if (dev->flags & IFF_UP) { 1387 if (dev->flags & IFF_UP) {
1381 nb->notifier_call(nb, NETDEV_GOING_DOWN, dev); 1388 nb->notifier_call(nb, NETDEV_GOING_DOWN, dev);
1382 nb->notifier_call(nb, NETDEV_DOWN, dev); 1389 nb->notifier_call(nb, NETDEV_DOWN, dev);
1383 } 1390 }
1384 nb->notifier_call(nb, NETDEV_UNREGISTER, dev); 1391 nb->notifier_call(nb, NETDEV_UNREGISTER, dev);
1385 nb->notifier_call(nb, NETDEV_UNREGISTER_BATCH, dev); 1392 nb->notifier_call(nb, NETDEV_UNREGISTER_BATCH, dev);
1386 } 1393 }
1387 } 1394 }
1388 1395
1389 raw_notifier_chain_unregister(&netdev_chain, nb); 1396 raw_notifier_chain_unregister(&netdev_chain, nb);
1390 goto unlock; 1397 goto unlock;
1391 } 1398 }
1392 EXPORT_SYMBOL(register_netdevice_notifier); 1399 EXPORT_SYMBOL(register_netdevice_notifier);
1393 1400
1394 /** 1401 /**
1395 * unregister_netdevice_notifier - unregister a network notifier block 1402 * unregister_netdevice_notifier - unregister a network notifier block
1396 * @nb: notifier 1403 * @nb: notifier
1397 * 1404 *
1398 * Unregister a notifier previously registered by 1405 * Unregister a notifier previously registered by
1399 * register_netdevice_notifier(). The notifier is unlinked into the 1406 * register_netdevice_notifier(). The notifier is unlinked into the
1400 * kernel structures and may then be reused. A negative errno code 1407 * kernel structures and may then be reused. A negative errno code
1401 * is returned on a failure. 1408 * is returned on a failure.
1402 */ 1409 */
1403 1410
1404 int unregister_netdevice_notifier(struct notifier_block *nb) 1411 int unregister_netdevice_notifier(struct notifier_block *nb)
1405 { 1412 {
1406 int err; 1413 int err;
1407 1414
1408 rtnl_lock(); 1415 rtnl_lock();
1409 err = raw_notifier_chain_unregister(&netdev_chain, nb); 1416 err = raw_notifier_chain_unregister(&netdev_chain, nb);
1410 rtnl_unlock(); 1417 rtnl_unlock();
1411 return err; 1418 return err;
1412 } 1419 }
1413 EXPORT_SYMBOL(unregister_netdevice_notifier); 1420 EXPORT_SYMBOL(unregister_netdevice_notifier);
1414 1421
1415 /** 1422 /**
1416 * call_netdevice_notifiers - call all network notifier blocks 1423 * call_netdevice_notifiers - call all network notifier blocks
1417 * @val: value passed unmodified to notifier function 1424 * @val: value passed unmodified to notifier function
1418 * @dev: net_device pointer passed unmodified to notifier function 1425 * @dev: net_device pointer passed unmodified to notifier function
1419 * 1426 *
1420 * Call all network notifier blocks. Parameters and return value 1427 * Call all network notifier blocks. Parameters and return value
1421 * are as for raw_notifier_call_chain(). 1428 * are as for raw_notifier_call_chain().
1422 */ 1429 */
1423 1430
1424 int call_netdevice_notifiers(unsigned long val, struct net_device *dev) 1431 int call_netdevice_notifiers(unsigned long val, struct net_device *dev)
1425 { 1432 {
1426 ASSERT_RTNL(); 1433 ASSERT_RTNL();
1427 return raw_notifier_call_chain(&netdev_chain, val, dev); 1434 return raw_notifier_call_chain(&netdev_chain, val, dev);
1428 } 1435 }
1429 EXPORT_SYMBOL(call_netdevice_notifiers); 1436 EXPORT_SYMBOL(call_netdevice_notifiers);
1430 1437
1431 /* When > 0 there are consumers of rx skb time stamps */ 1438 /* When > 0 there are consumers of rx skb time stamps */
1432 static atomic_t netstamp_needed = ATOMIC_INIT(0); 1439 static atomic_t netstamp_needed = ATOMIC_INIT(0);
1433 1440
1434 void net_enable_timestamp(void) 1441 void net_enable_timestamp(void)
1435 { 1442 {
1436 atomic_inc(&netstamp_needed); 1443 atomic_inc(&netstamp_needed);
1437 } 1444 }
1438 EXPORT_SYMBOL(net_enable_timestamp); 1445 EXPORT_SYMBOL(net_enable_timestamp);
1439 1446
1440 void net_disable_timestamp(void) 1447 void net_disable_timestamp(void)
1441 { 1448 {
1442 atomic_dec(&netstamp_needed); 1449 atomic_dec(&netstamp_needed);
1443 } 1450 }
1444 EXPORT_SYMBOL(net_disable_timestamp); 1451 EXPORT_SYMBOL(net_disable_timestamp);
1445 1452
1446 static inline void net_timestamp_set(struct sk_buff *skb) 1453 static inline void net_timestamp_set(struct sk_buff *skb)
1447 { 1454 {
1448 if (atomic_read(&netstamp_needed)) 1455 if (atomic_read(&netstamp_needed))
1449 __net_timestamp(skb); 1456 __net_timestamp(skb);
1450 else 1457 else
1451 skb->tstamp.tv64 = 0; 1458 skb->tstamp.tv64 = 0;
1452 } 1459 }
1453 1460
1454 static inline void net_timestamp_check(struct sk_buff *skb) 1461 static inline void net_timestamp_check(struct sk_buff *skb)
1455 { 1462 {
1456 if (!skb->tstamp.tv64 && atomic_read(&netstamp_needed)) 1463 if (!skb->tstamp.tv64 && atomic_read(&netstamp_needed))
1457 __net_timestamp(skb); 1464 __net_timestamp(skb);
1458 } 1465 }
1459 1466
1460 static inline bool is_skb_forwardable(struct net_device *dev, 1467 static inline bool is_skb_forwardable(struct net_device *dev,
1461 struct sk_buff *skb) 1468 struct sk_buff *skb)
1462 { 1469 {
1463 unsigned int len; 1470 unsigned int len;
1464 1471
1465 if (!(dev->flags & IFF_UP)) 1472 if (!(dev->flags & IFF_UP))
1466 return false; 1473 return false;
1467 1474
1468 len = dev->mtu + dev->hard_header_len + VLAN_HLEN; 1475 len = dev->mtu + dev->hard_header_len + VLAN_HLEN;
1469 if (skb->len <= len) 1476 if (skb->len <= len)
1470 return true; 1477 return true;
1471 1478
1472 /* if TSO is enabled, we don't care about the length as the packet 1479 /* if TSO is enabled, we don't care about the length as the packet
1473 * could be forwarded without being segmented before 1480 * could be forwarded without being segmented before
1474 */ 1481 */
1475 if (skb_is_gso(skb)) 1482 if (skb_is_gso(skb))
1476 return true; 1483 return true;
1477 1484
1478 return false; 1485 return false;
1479 } 1486 }
1480 1487
1481 /** 1488 /**
1482 * dev_forward_skb - loopback an skb to another netif 1489 * dev_forward_skb - loopback an skb to another netif
1483 * 1490 *
1484 * @dev: destination network device 1491 * @dev: destination network device
1485 * @skb: buffer to forward 1492 * @skb: buffer to forward
1486 * 1493 *
1487 * return values: 1494 * return values:
1488 * NET_RX_SUCCESS (no congestion) 1495 * NET_RX_SUCCESS (no congestion)
1489 * NET_RX_DROP (packet was dropped, but freed) 1496 * NET_RX_DROP (packet was dropped, but freed)
1490 * 1497 *
1491 * dev_forward_skb can be used for injecting an skb from the 1498 * dev_forward_skb can be used for injecting an skb from the
1492 * start_xmit function of one device into the receive queue 1499 * start_xmit function of one device into the receive queue
1493 * of another device. 1500 * of another device.
1494 * 1501 *
1495 * The receiving device may be in another namespace, so 1502 * The receiving device may be in another namespace, so
1496 * we have to clear all information in the skb that could 1503 * we have to clear all information in the skb that could
1497 * impact namespace isolation. 1504 * impact namespace isolation.
1498 */ 1505 */
1499 int dev_forward_skb(struct net_device *dev, struct sk_buff *skb) 1506 int dev_forward_skb(struct net_device *dev, struct sk_buff *skb)
1500 { 1507 {
1501 skb_orphan(skb); 1508 skb_orphan(skb);
1502 nf_reset(skb); 1509 nf_reset(skb);
1503 1510
1504 if (unlikely(!is_skb_forwardable(dev, skb))) { 1511 if (unlikely(!is_skb_forwardable(dev, skb))) {
1505 atomic_long_inc(&dev->rx_dropped); 1512 atomic_long_inc(&dev->rx_dropped);
1506 kfree_skb(skb); 1513 kfree_skb(skb);
1507 return NET_RX_DROP; 1514 return NET_RX_DROP;
1508 } 1515 }
1509 skb_set_dev(skb, dev); 1516 skb_set_dev(skb, dev);
1510 skb->tstamp.tv64 = 0; 1517 skb->tstamp.tv64 = 0;
1511 skb->pkt_type = PACKET_HOST; 1518 skb->pkt_type = PACKET_HOST;
1512 skb->protocol = eth_type_trans(skb, dev); 1519 skb->protocol = eth_type_trans(skb, dev);
1513 return netif_rx(skb); 1520 return netif_rx(skb);
1514 } 1521 }
1515 EXPORT_SYMBOL_GPL(dev_forward_skb); 1522 EXPORT_SYMBOL_GPL(dev_forward_skb);
1516 1523
1517 static inline int deliver_skb(struct sk_buff *skb, 1524 static inline int deliver_skb(struct sk_buff *skb,
1518 struct packet_type *pt_prev, 1525 struct packet_type *pt_prev,
1519 struct net_device *orig_dev) 1526 struct net_device *orig_dev)
1520 { 1527 {
1521 atomic_inc(&skb->users); 1528 atomic_inc(&skb->users);
1522 return pt_prev->func(skb, skb->dev, pt_prev, orig_dev); 1529 return pt_prev->func(skb, skb->dev, pt_prev, orig_dev);
1523 } 1530 }
1524 1531
1525 /* 1532 /*
1526 * Support routine. Sends outgoing frames to any network 1533 * Support routine. Sends outgoing frames to any network
1527 * taps currently in use. 1534 * taps currently in use.
1528 */ 1535 */
1529 1536
1530 static void dev_queue_xmit_nit(struct sk_buff *skb, struct net_device *dev) 1537 static void dev_queue_xmit_nit(struct sk_buff *skb, struct net_device *dev)
1531 { 1538 {
1532 struct packet_type *ptype; 1539 struct packet_type *ptype;
1533 struct sk_buff *skb2 = NULL; 1540 struct sk_buff *skb2 = NULL;
1534 struct packet_type *pt_prev = NULL; 1541 struct packet_type *pt_prev = NULL;
1535 1542
1536 rcu_read_lock(); 1543 rcu_read_lock();
1537 list_for_each_entry_rcu(ptype, &ptype_all, list) { 1544 list_for_each_entry_rcu(ptype, &ptype_all, list) {
1538 /* Never send packets back to the socket 1545 /* Never send packets back to the socket
1539 * they originated from - MvS (miquels@drinkel.ow.org) 1546 * they originated from - MvS (miquels@drinkel.ow.org)
1540 */ 1547 */
1541 if ((ptype->dev == dev || !ptype->dev) && 1548 if ((ptype->dev == dev || !ptype->dev) &&
1542 (ptype->af_packet_priv == NULL || 1549 (ptype->af_packet_priv == NULL ||
1543 (struct sock *)ptype->af_packet_priv != skb->sk)) { 1550 (struct sock *)ptype->af_packet_priv != skb->sk)) {
1544 if (pt_prev) { 1551 if (pt_prev) {
1545 deliver_skb(skb2, pt_prev, skb->dev); 1552 deliver_skb(skb2, pt_prev, skb->dev);
1546 pt_prev = ptype; 1553 pt_prev = ptype;
1547 continue; 1554 continue;
1548 } 1555 }
1549 1556
1550 skb2 = skb_clone(skb, GFP_ATOMIC); 1557 skb2 = skb_clone(skb, GFP_ATOMIC);
1551 if (!skb2) 1558 if (!skb2)
1552 break; 1559 break;
1553 1560
1554 net_timestamp_set(skb2); 1561 net_timestamp_set(skb2);
1555 1562
1556 /* skb->nh should be correctly 1563 /* skb->nh should be correctly
1557 set by sender, so that the second statement is 1564 set by sender, so that the second statement is
1558 just protection against buggy protocols. 1565 just protection against buggy protocols.
1559 */ 1566 */
1560 skb_reset_mac_header(skb2); 1567 skb_reset_mac_header(skb2);
1561 1568
1562 if (skb_network_header(skb2) < skb2->data || 1569 if (skb_network_header(skb2) < skb2->data ||
1563 skb2->network_header > skb2->tail) { 1570 skb2->network_header > skb2->tail) {
1564 if (net_ratelimit()) 1571 if (net_ratelimit())
1565 printk(KERN_CRIT "protocol %04x is " 1572 printk(KERN_CRIT "protocol %04x is "
1566 "buggy, dev %s\n", 1573 "buggy, dev %s\n",
1567 ntohs(skb2->protocol), 1574 ntohs(skb2->protocol),
1568 dev->name); 1575 dev->name);
1569 skb_reset_network_header(skb2); 1576 skb_reset_network_header(skb2);
1570 } 1577 }
1571 1578
1572 skb2->transport_header = skb2->network_header; 1579 skb2->transport_header = skb2->network_header;
1573 skb2->pkt_type = PACKET_OUTGOING; 1580 skb2->pkt_type = PACKET_OUTGOING;
1574 pt_prev = ptype; 1581 pt_prev = ptype;
1575 } 1582 }
1576 } 1583 }
1577 if (pt_prev) 1584 if (pt_prev)
1578 pt_prev->func(skb2, skb->dev, pt_prev, skb->dev); 1585 pt_prev->func(skb2, skb->dev, pt_prev, skb->dev);
1579 rcu_read_unlock(); 1586 rcu_read_unlock();
1580 } 1587 }
1581 1588
1582 /* netif_setup_tc - Handle tc mappings on real_num_tx_queues change 1589 /* netif_setup_tc - Handle tc mappings on real_num_tx_queues change
1583 * @dev: Network device 1590 * @dev: Network device
1584 * @txq: number of queues available 1591 * @txq: number of queues available
1585 * 1592 *
1586 * If real_num_tx_queues is changed the tc mappings may no longer be 1593 * If real_num_tx_queues is changed the tc mappings may no longer be
1587 * valid. To resolve this verify the tc mapping remains valid and if 1594 * valid. To resolve this verify the tc mapping remains valid and if
1588 * not NULL the mapping. With no priorities mapping to this 1595 * not NULL the mapping. With no priorities mapping to this
1589 * offset/count pair it will no longer be used. In the worst case TC0 1596 * offset/count pair it will no longer be used. In the worst case TC0
1590 * is invalid nothing can be done so disable priority mappings. If is 1597 * is invalid nothing can be done so disable priority mappings. If is
1591 * expected that drivers will fix this mapping if they can before 1598 * expected that drivers will fix this mapping if they can before
1592 * calling netif_set_real_num_tx_queues. 1599 * calling netif_set_real_num_tx_queues.
1593 */ 1600 */
1594 static void netif_setup_tc(struct net_device *dev, unsigned int txq) 1601 static void netif_setup_tc(struct net_device *dev, unsigned int txq)
1595 { 1602 {
1596 int i; 1603 int i;
1597 struct netdev_tc_txq *tc = &dev->tc_to_txq[0]; 1604 struct netdev_tc_txq *tc = &dev->tc_to_txq[0];
1598 1605
1599 /* If TC0 is invalidated disable TC mapping */ 1606 /* If TC0 is invalidated disable TC mapping */
1600 if (tc->offset + tc->count > txq) { 1607 if (tc->offset + tc->count > txq) {
1601 pr_warning("Number of in use tx queues changed " 1608 pr_warning("Number of in use tx queues changed "
1602 "invalidating tc mappings. Priority " 1609 "invalidating tc mappings. Priority "
1603 "traffic classification disabled!\n"); 1610 "traffic classification disabled!\n");
1604 dev->num_tc = 0; 1611 dev->num_tc = 0;
1605 return; 1612 return;
1606 } 1613 }
1607 1614
1608 /* Invalidated prio to tc mappings set to TC0 */ 1615 /* Invalidated prio to tc mappings set to TC0 */
1609 for (i = 1; i < TC_BITMASK + 1; i++) { 1616 for (i = 1; i < TC_BITMASK + 1; i++) {
1610 int q = netdev_get_prio_tc_map(dev, i); 1617 int q = netdev_get_prio_tc_map(dev, i);
1611 1618
1612 tc = &dev->tc_to_txq[q]; 1619 tc = &dev->tc_to_txq[q];
1613 if (tc->offset + tc->count > txq) { 1620 if (tc->offset + tc->count > txq) {
1614 pr_warning("Number of in use tx queues " 1621 pr_warning("Number of in use tx queues "
1615 "changed. Priority %i to tc " 1622 "changed. Priority %i to tc "
1616 "mapping %i is no longer valid " 1623 "mapping %i is no longer valid "
1617 "setting map to 0\n", 1624 "setting map to 0\n",
1618 i, q); 1625 i, q);
1619 netdev_set_prio_tc_map(dev, i, 0); 1626 netdev_set_prio_tc_map(dev, i, 0);
1620 } 1627 }
1621 } 1628 }
1622 } 1629 }
1623 1630
1624 /* 1631 /*
1625 * Routine to help set real_num_tx_queues. To avoid skbs mapped to queues 1632 * Routine to help set real_num_tx_queues. To avoid skbs mapped to queues
1626 * greater then real_num_tx_queues stale skbs on the qdisc must be flushed. 1633 * greater then real_num_tx_queues stale skbs on the qdisc must be flushed.
1627 */ 1634 */
1628 int netif_set_real_num_tx_queues(struct net_device *dev, unsigned int txq) 1635 int netif_set_real_num_tx_queues(struct net_device *dev, unsigned int txq)
1629 { 1636 {
1630 int rc; 1637 int rc;
1631 1638
1632 if (txq < 1 || txq > dev->num_tx_queues) 1639 if (txq < 1 || txq > dev->num_tx_queues)
1633 return -EINVAL; 1640 return -EINVAL;
1634 1641
1635 if (dev->reg_state == NETREG_REGISTERED || 1642 if (dev->reg_state == NETREG_REGISTERED ||
1636 dev->reg_state == NETREG_UNREGISTERING) { 1643 dev->reg_state == NETREG_UNREGISTERING) {
1637 ASSERT_RTNL(); 1644 ASSERT_RTNL();
1638 1645
1639 rc = netdev_queue_update_kobjects(dev, dev->real_num_tx_queues, 1646 rc = netdev_queue_update_kobjects(dev, dev->real_num_tx_queues,
1640 txq); 1647 txq);
1641 if (rc) 1648 if (rc)
1642 return rc; 1649 return rc;
1643 1650
1644 if (dev->num_tc) 1651 if (dev->num_tc)
1645 netif_setup_tc(dev, txq); 1652 netif_setup_tc(dev, txq);
1646 1653
1647 if (txq < dev->real_num_tx_queues) 1654 if (txq < dev->real_num_tx_queues)
1648 qdisc_reset_all_tx_gt(dev, txq); 1655 qdisc_reset_all_tx_gt(dev, txq);
1649 } 1656 }
1650 1657
1651 dev->real_num_tx_queues = txq; 1658 dev->real_num_tx_queues = txq;
1652 return 0; 1659 return 0;
1653 } 1660 }
1654 EXPORT_SYMBOL(netif_set_real_num_tx_queues); 1661 EXPORT_SYMBOL(netif_set_real_num_tx_queues);
1655 1662
1656 #ifdef CONFIG_RPS 1663 #ifdef CONFIG_RPS
1657 /** 1664 /**
1658 * netif_set_real_num_rx_queues - set actual number of RX queues used 1665 * netif_set_real_num_rx_queues - set actual number of RX queues used
1659 * @dev: Network device 1666 * @dev: Network device
1660 * @rxq: Actual number of RX queues 1667 * @rxq: Actual number of RX queues
1661 * 1668 *
1662 * This must be called either with the rtnl_lock held or before 1669 * This must be called either with the rtnl_lock held or before
1663 * registration of the net device. Returns 0 on success, or a 1670 * registration of the net device. Returns 0 on success, or a
1664 * negative error code. If called before registration, it always 1671 * negative error code. If called before registration, it always
1665 * succeeds. 1672 * succeeds.
1666 */ 1673 */
1667 int netif_set_real_num_rx_queues(struct net_device *dev, unsigned int rxq) 1674 int netif_set_real_num_rx_queues(struct net_device *dev, unsigned int rxq)
1668 { 1675 {
1669 int rc; 1676 int rc;
1670 1677
1671 if (rxq < 1 || rxq > dev->num_rx_queues) 1678 if (rxq < 1 || rxq > dev->num_rx_queues)
1672 return -EINVAL; 1679 return -EINVAL;
1673 1680
1674 if (dev->reg_state == NETREG_REGISTERED) { 1681 if (dev->reg_state == NETREG_REGISTERED) {
1675 ASSERT_RTNL(); 1682 ASSERT_RTNL();
1676 1683
1677 rc = net_rx_queue_update_kobjects(dev, dev->real_num_rx_queues, 1684 rc = net_rx_queue_update_kobjects(dev, dev->real_num_rx_queues,
1678 rxq); 1685 rxq);
1679 if (rc) 1686 if (rc)
1680 return rc; 1687 return rc;
1681 } 1688 }
1682 1689
1683 dev->real_num_rx_queues = rxq; 1690 dev->real_num_rx_queues = rxq;
1684 return 0; 1691 return 0;
1685 } 1692 }
1686 EXPORT_SYMBOL(netif_set_real_num_rx_queues); 1693 EXPORT_SYMBOL(netif_set_real_num_rx_queues);
1687 #endif 1694 #endif
1688 1695
1689 static inline void __netif_reschedule(struct Qdisc *q) 1696 static inline void __netif_reschedule(struct Qdisc *q)
1690 { 1697 {
1691 struct softnet_data *sd; 1698 struct softnet_data *sd;
1692 unsigned long flags; 1699 unsigned long flags;
1693 1700
1694 local_irq_save(flags); 1701 local_irq_save(flags);
1695 sd = &__get_cpu_var(softnet_data); 1702 sd = &__get_cpu_var(softnet_data);
1696 q->next_sched = NULL; 1703 q->next_sched = NULL;
1697 *sd->output_queue_tailp = q; 1704 *sd->output_queue_tailp = q;
1698 sd->output_queue_tailp = &q->next_sched; 1705 sd->output_queue_tailp = &q->next_sched;
1699 raise_softirq_irqoff(NET_TX_SOFTIRQ); 1706 raise_softirq_irqoff(NET_TX_SOFTIRQ);
1700 local_irq_restore(flags); 1707 local_irq_restore(flags);
1701 } 1708 }
1702 1709
1703 void __netif_schedule(struct Qdisc *q) 1710 void __netif_schedule(struct Qdisc *q)
1704 { 1711 {
1705 if (!test_and_set_bit(__QDISC_STATE_SCHED, &q->state)) 1712 if (!test_and_set_bit(__QDISC_STATE_SCHED, &q->state))
1706 __netif_reschedule(q); 1713 __netif_reschedule(q);
1707 } 1714 }
1708 EXPORT_SYMBOL(__netif_schedule); 1715 EXPORT_SYMBOL(__netif_schedule);
1709 1716
1710 void dev_kfree_skb_irq(struct sk_buff *skb) 1717 void dev_kfree_skb_irq(struct sk_buff *skb)
1711 { 1718 {
1712 if (atomic_dec_and_test(&skb->users)) { 1719 if (atomic_dec_and_test(&skb->users)) {
1713 struct softnet_data *sd; 1720 struct softnet_data *sd;
1714 unsigned long flags; 1721 unsigned long flags;
1715 1722
1716 local_irq_save(flags); 1723 local_irq_save(flags);
1717 sd = &__get_cpu_var(softnet_data); 1724 sd = &__get_cpu_var(softnet_data);
1718 skb->next = sd->completion_queue; 1725 skb->next = sd->completion_queue;
1719 sd->completion_queue = skb; 1726 sd->completion_queue = skb;
1720 raise_softirq_irqoff(NET_TX_SOFTIRQ); 1727 raise_softirq_irqoff(NET_TX_SOFTIRQ);
1721 local_irq_restore(flags); 1728 local_irq_restore(flags);
1722 } 1729 }
1723 } 1730 }
1724 EXPORT_SYMBOL(dev_kfree_skb_irq); 1731 EXPORT_SYMBOL(dev_kfree_skb_irq);
1725 1732
1726 void dev_kfree_skb_any(struct sk_buff *skb) 1733 void dev_kfree_skb_any(struct sk_buff *skb)
1727 { 1734 {
1728 if (in_irq() || irqs_disabled()) 1735 if (in_irq() || irqs_disabled())
1729 dev_kfree_skb_irq(skb); 1736 dev_kfree_skb_irq(skb);
1730 else 1737 else
1731 dev_kfree_skb(skb); 1738 dev_kfree_skb(skb);
1732 } 1739 }
1733 EXPORT_SYMBOL(dev_kfree_skb_any); 1740 EXPORT_SYMBOL(dev_kfree_skb_any);
1734 1741
1735 1742
1736 /** 1743 /**
1737 * netif_device_detach - mark device as removed 1744 * netif_device_detach - mark device as removed
1738 * @dev: network device 1745 * @dev: network device
1739 * 1746 *
1740 * Mark device as removed from system and therefore no longer available. 1747 * Mark device as removed from system and therefore no longer available.
1741 */ 1748 */
1742 void netif_device_detach(struct net_device *dev) 1749 void netif_device_detach(struct net_device *dev)
1743 { 1750 {
1744 if (test_and_clear_bit(__LINK_STATE_PRESENT, &dev->state) && 1751 if (test_and_clear_bit(__LINK_STATE_PRESENT, &dev->state) &&
1745 netif_running(dev)) { 1752 netif_running(dev)) {
1746 netif_tx_stop_all_queues(dev); 1753 netif_tx_stop_all_queues(dev);
1747 } 1754 }
1748 } 1755 }
1749 EXPORT_SYMBOL(netif_device_detach); 1756 EXPORT_SYMBOL(netif_device_detach);
1750 1757
1751 /** 1758 /**
1752 * netif_device_attach - mark device as attached 1759 * netif_device_attach - mark device as attached
1753 * @dev: network device 1760 * @dev: network device
1754 * 1761 *
1755 * Mark device as attached from system and restart if needed. 1762 * Mark device as attached from system and restart if needed.
1756 */ 1763 */
1757 void netif_device_attach(struct net_device *dev) 1764 void netif_device_attach(struct net_device *dev)
1758 { 1765 {
1759 if (!test_and_set_bit(__LINK_STATE_PRESENT, &dev->state) && 1766 if (!test_and_set_bit(__LINK_STATE_PRESENT, &dev->state) &&
1760 netif_running(dev)) { 1767 netif_running(dev)) {
1761 netif_tx_wake_all_queues(dev); 1768 netif_tx_wake_all_queues(dev);
1762 __netdev_watchdog_up(dev); 1769 __netdev_watchdog_up(dev);
1763 } 1770 }
1764 } 1771 }
1765 EXPORT_SYMBOL(netif_device_attach); 1772 EXPORT_SYMBOL(netif_device_attach);
1766 1773
1767 /** 1774 /**
1768 * skb_dev_set -- assign a new device to a buffer 1775 * skb_dev_set -- assign a new device to a buffer
1769 * @skb: buffer for the new device 1776 * @skb: buffer for the new device
1770 * @dev: network device 1777 * @dev: network device
1771 * 1778 *
1772 * If an skb is owned by a device already, we have to reset 1779 * If an skb is owned by a device already, we have to reset
1773 * all data private to the namespace a device belongs to 1780 * all data private to the namespace a device belongs to
1774 * before assigning it a new device. 1781 * before assigning it a new device.
1775 */ 1782 */
1776 #ifdef CONFIG_NET_NS 1783 #ifdef CONFIG_NET_NS
1777 void skb_set_dev(struct sk_buff *skb, struct net_device *dev) 1784 void skb_set_dev(struct sk_buff *skb, struct net_device *dev)
1778 { 1785 {
1779 skb_dst_drop(skb); 1786 skb_dst_drop(skb);
1780 if (skb->dev && !net_eq(dev_net(skb->dev), dev_net(dev))) { 1787 if (skb->dev && !net_eq(dev_net(skb->dev), dev_net(dev))) {
1781 secpath_reset(skb); 1788 secpath_reset(skb);
1782 nf_reset(skb); 1789 nf_reset(skb);
1783 skb_init_secmark(skb); 1790 skb_init_secmark(skb);
1784 skb->mark = 0; 1791 skb->mark = 0;
1785 skb->priority = 0; 1792 skb->priority = 0;
1786 skb->nf_trace = 0; 1793 skb->nf_trace = 0;
1787 skb->ipvs_property = 0; 1794 skb->ipvs_property = 0;
1788 #ifdef CONFIG_NET_SCHED 1795 #ifdef CONFIG_NET_SCHED
1789 skb->tc_index = 0; 1796 skb->tc_index = 0;
1790 #endif 1797 #endif
1791 } 1798 }
1792 skb->dev = dev; 1799 skb->dev = dev;
1793 } 1800 }
1794 EXPORT_SYMBOL(skb_set_dev); 1801 EXPORT_SYMBOL(skb_set_dev);
1795 #endif /* CONFIG_NET_NS */ 1802 #endif /* CONFIG_NET_NS */
1796 1803
1797 /* 1804 /*
1798 * Invalidate hardware checksum when packet is to be mangled, and 1805 * Invalidate hardware checksum when packet is to be mangled, and
1799 * complete checksum manually on outgoing path. 1806 * complete checksum manually on outgoing path.
1800 */ 1807 */
1801 int skb_checksum_help(struct sk_buff *skb) 1808 int skb_checksum_help(struct sk_buff *skb)
1802 { 1809 {
1803 __wsum csum; 1810 __wsum csum;
1804 int ret = 0, offset; 1811 int ret = 0, offset;
1805 1812
1806 if (skb->ip_summed == CHECKSUM_COMPLETE) 1813 if (skb->ip_summed == CHECKSUM_COMPLETE)
1807 goto out_set_summed; 1814 goto out_set_summed;
1808 1815
1809 if (unlikely(skb_shinfo(skb)->gso_size)) { 1816 if (unlikely(skb_shinfo(skb)->gso_size)) {
1810 /* Let GSO fix up the checksum. */ 1817 /* Let GSO fix up the checksum. */
1811 goto out_set_summed; 1818 goto out_set_summed;
1812 } 1819 }
1813 1820
1814 offset = skb_checksum_start_offset(skb); 1821 offset = skb_checksum_start_offset(skb);
1815 BUG_ON(offset >= skb_headlen(skb)); 1822 BUG_ON(offset >= skb_headlen(skb));
1816 csum = skb_checksum(skb, offset, skb->len - offset, 0); 1823 csum = skb_checksum(skb, offset, skb->len - offset, 0);
1817 1824
1818 offset += skb->csum_offset; 1825 offset += skb->csum_offset;
1819 BUG_ON(offset + sizeof(__sum16) > skb_headlen(skb)); 1826 BUG_ON(offset + sizeof(__sum16) > skb_headlen(skb));
1820 1827
1821 if (skb_cloned(skb) && 1828 if (skb_cloned(skb) &&
1822 !skb_clone_writable(skb, offset + sizeof(__sum16))) { 1829 !skb_clone_writable(skb, offset + sizeof(__sum16))) {
1823 ret = pskb_expand_head(skb, 0, 0, GFP_ATOMIC); 1830 ret = pskb_expand_head(skb, 0, 0, GFP_ATOMIC);
1824 if (ret) 1831 if (ret)
1825 goto out; 1832 goto out;
1826 } 1833 }
1827 1834
1828 *(__sum16 *)(skb->data + offset) = csum_fold(csum); 1835 *(__sum16 *)(skb->data + offset) = csum_fold(csum);
1829 out_set_summed: 1836 out_set_summed:
1830 skb->ip_summed = CHECKSUM_NONE; 1837 skb->ip_summed = CHECKSUM_NONE;
1831 out: 1838 out:
1832 return ret; 1839 return ret;
1833 } 1840 }
1834 EXPORT_SYMBOL(skb_checksum_help); 1841 EXPORT_SYMBOL(skb_checksum_help);
1835 1842
1836 /** 1843 /**
1837 * skb_gso_segment - Perform segmentation on skb. 1844 * skb_gso_segment - Perform segmentation on skb.
1838 * @skb: buffer to segment 1845 * @skb: buffer to segment
1839 * @features: features for the output path (see dev->features) 1846 * @features: features for the output path (see dev->features)
1840 * 1847 *
1841 * This function segments the given skb and returns a list of segments. 1848 * This function segments the given skb and returns a list of segments.
1842 * 1849 *
1843 * It may return NULL if the skb requires no segmentation. This is 1850 * It may return NULL if the skb requires no segmentation. This is
1844 * only possible when GSO is used for verifying header integrity. 1851 * only possible when GSO is used for verifying header integrity.
1845 */ 1852 */
1846 struct sk_buff *skb_gso_segment(struct sk_buff *skb, u32 features) 1853 struct sk_buff *skb_gso_segment(struct sk_buff *skb, u32 features)
1847 { 1854 {
1848 struct sk_buff *segs = ERR_PTR(-EPROTONOSUPPORT); 1855 struct sk_buff *segs = ERR_PTR(-EPROTONOSUPPORT);
1849 struct packet_type *ptype; 1856 struct packet_type *ptype;
1850 __be16 type = skb->protocol; 1857 __be16 type = skb->protocol;
1851 int vlan_depth = ETH_HLEN; 1858 int vlan_depth = ETH_HLEN;
1852 int err; 1859 int err;
1853 1860
1854 while (type == htons(ETH_P_8021Q)) { 1861 while (type == htons(ETH_P_8021Q)) {
1855 struct vlan_hdr *vh; 1862 struct vlan_hdr *vh;
1856 1863
1857 if (unlikely(!pskb_may_pull(skb, vlan_depth + VLAN_HLEN))) 1864 if (unlikely(!pskb_may_pull(skb, vlan_depth + VLAN_HLEN)))
1858 return ERR_PTR(-EINVAL); 1865 return ERR_PTR(-EINVAL);
1859 1866
1860 vh = (struct vlan_hdr *)(skb->data + vlan_depth); 1867 vh = (struct vlan_hdr *)(skb->data + vlan_depth);
1861 type = vh->h_vlan_encapsulated_proto; 1868 type = vh->h_vlan_encapsulated_proto;
1862 vlan_depth += VLAN_HLEN; 1869 vlan_depth += VLAN_HLEN;
1863 } 1870 }
1864 1871
1865 skb_reset_mac_header(skb); 1872 skb_reset_mac_header(skb);
1866 skb->mac_len = skb->network_header - skb->mac_header; 1873 skb->mac_len = skb->network_header - skb->mac_header;
1867 __skb_pull(skb, skb->mac_len); 1874 __skb_pull(skb, skb->mac_len);
1868 1875
1869 if (unlikely(skb->ip_summed != CHECKSUM_PARTIAL)) { 1876 if (unlikely(skb->ip_summed != CHECKSUM_PARTIAL)) {
1870 struct net_device *dev = skb->dev; 1877 struct net_device *dev = skb->dev;
1871 struct ethtool_drvinfo info = {}; 1878 struct ethtool_drvinfo info = {};
1872 1879
1873 if (dev && dev->ethtool_ops && dev->ethtool_ops->get_drvinfo) 1880 if (dev && dev->ethtool_ops && dev->ethtool_ops->get_drvinfo)
1874 dev->ethtool_ops->get_drvinfo(dev, &info); 1881 dev->ethtool_ops->get_drvinfo(dev, &info);
1875 1882
1876 WARN(1, "%s: caps=(0x%lx, 0x%lx) len=%d data_len=%d ip_summed=%d\n", 1883 WARN(1, "%s: caps=(0x%lx, 0x%lx) len=%d data_len=%d ip_summed=%d\n",
1877 info.driver, dev ? dev->features : 0L, 1884 info.driver, dev ? dev->features : 0L,
1878 skb->sk ? skb->sk->sk_route_caps : 0L, 1885 skb->sk ? skb->sk->sk_route_caps : 0L,
1879 skb->len, skb->data_len, skb->ip_summed); 1886 skb->len, skb->data_len, skb->ip_summed);
1880 1887
1881 if (skb_header_cloned(skb) && 1888 if (skb_header_cloned(skb) &&
1882 (err = pskb_expand_head(skb, 0, 0, GFP_ATOMIC))) 1889 (err = pskb_expand_head(skb, 0, 0, GFP_ATOMIC)))
1883 return ERR_PTR(err); 1890 return ERR_PTR(err);
1884 } 1891 }
1885 1892
1886 rcu_read_lock(); 1893 rcu_read_lock();
1887 list_for_each_entry_rcu(ptype, 1894 list_for_each_entry_rcu(ptype,
1888 &ptype_base[ntohs(type) & PTYPE_HASH_MASK], list) { 1895 &ptype_base[ntohs(type) & PTYPE_HASH_MASK], list) {
1889 if (ptype->type == type && !ptype->dev && ptype->gso_segment) { 1896 if (ptype->type == type && !ptype->dev && ptype->gso_segment) {
1890 if (unlikely(skb->ip_summed != CHECKSUM_PARTIAL)) { 1897 if (unlikely(skb->ip_summed != CHECKSUM_PARTIAL)) {
1891 err = ptype->gso_send_check(skb); 1898 err = ptype->gso_send_check(skb);
1892 segs = ERR_PTR(err); 1899 segs = ERR_PTR(err);
1893 if (err || skb_gso_ok(skb, features)) 1900 if (err || skb_gso_ok(skb, features))
1894 break; 1901 break;
1895 __skb_push(skb, (skb->data - 1902 __skb_push(skb, (skb->data -
1896 skb_network_header(skb))); 1903 skb_network_header(skb)));
1897 } 1904 }
1898 segs = ptype->gso_segment(skb, features); 1905 segs = ptype->gso_segment(skb, features);
1899 break; 1906 break;
1900 } 1907 }
1901 } 1908 }
1902 rcu_read_unlock(); 1909 rcu_read_unlock();
1903 1910
1904 __skb_push(skb, skb->data - skb_mac_header(skb)); 1911 __skb_push(skb, skb->data - skb_mac_header(skb));
1905 1912
1906 return segs; 1913 return segs;
1907 } 1914 }
1908 EXPORT_SYMBOL(skb_gso_segment); 1915 EXPORT_SYMBOL(skb_gso_segment);
1909 1916
1910 /* Take action when hardware reception checksum errors are detected. */ 1917 /* Take action when hardware reception checksum errors are detected. */
1911 #ifdef CONFIG_BUG 1918 #ifdef CONFIG_BUG
1912 void netdev_rx_csum_fault(struct net_device *dev) 1919 void netdev_rx_csum_fault(struct net_device *dev)
1913 { 1920 {
1914 if (net_ratelimit()) { 1921 if (net_ratelimit()) {
1915 printk(KERN_ERR "%s: hw csum failure.\n", 1922 printk(KERN_ERR "%s: hw csum failure.\n",
1916 dev ? dev->name : "<unknown>"); 1923 dev ? dev->name : "<unknown>");
1917 dump_stack(); 1924 dump_stack();
1918 } 1925 }
1919 } 1926 }
1920 EXPORT_SYMBOL(netdev_rx_csum_fault); 1927 EXPORT_SYMBOL(netdev_rx_csum_fault);
1921 #endif 1928 #endif
1922 1929
1923 /* Actually, we should eliminate this check as soon as we know, that: 1930 /* Actually, we should eliminate this check as soon as we know, that:
1924 * 1. IOMMU is present and allows to map all the memory. 1931 * 1. IOMMU is present and allows to map all the memory.
1925 * 2. No high memory really exists on this machine. 1932 * 2. No high memory really exists on this machine.
1926 */ 1933 */
1927 1934
1928 static int illegal_highdma(struct net_device *dev, struct sk_buff *skb) 1935 static int illegal_highdma(struct net_device *dev, struct sk_buff *skb)
1929 { 1936 {
1930 #ifdef CONFIG_HIGHMEM 1937 #ifdef CONFIG_HIGHMEM
1931 int i; 1938 int i;
1932 if (!(dev->features & NETIF_F_HIGHDMA)) { 1939 if (!(dev->features & NETIF_F_HIGHDMA)) {
1933 for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) 1940 for (i = 0; i < skb_shinfo(skb)->nr_frags; i++)
1934 if (PageHighMem(skb_shinfo(skb)->frags[i].page)) 1941 if (PageHighMem(skb_shinfo(skb)->frags[i].page))
1935 return 1; 1942 return 1;
1936 } 1943 }
1937 1944
1938 if (PCI_DMA_BUS_IS_PHYS) { 1945 if (PCI_DMA_BUS_IS_PHYS) {
1939 struct device *pdev = dev->dev.parent; 1946 struct device *pdev = dev->dev.parent;
1940 1947
1941 if (!pdev) 1948 if (!pdev)
1942 return 0; 1949 return 0;
1943 for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) { 1950 for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) {
1944 dma_addr_t addr = page_to_phys(skb_shinfo(skb)->frags[i].page); 1951 dma_addr_t addr = page_to_phys(skb_shinfo(skb)->frags[i].page);
1945 if (!pdev->dma_mask || addr + PAGE_SIZE - 1 > *pdev->dma_mask) 1952 if (!pdev->dma_mask || addr + PAGE_SIZE - 1 > *pdev->dma_mask)
1946 return 1; 1953 return 1;
1947 } 1954 }
1948 } 1955 }
1949 #endif 1956 #endif
1950 return 0; 1957 return 0;
1951 } 1958 }
1952 1959
1953 struct dev_gso_cb { 1960 struct dev_gso_cb {
1954 void (*destructor)(struct sk_buff *skb); 1961 void (*destructor)(struct sk_buff *skb);
1955 }; 1962 };
1956 1963
1957 #define DEV_GSO_CB(skb) ((struct dev_gso_cb *)(skb)->cb) 1964 #define DEV_GSO_CB(skb) ((struct dev_gso_cb *)(skb)->cb)
1958 1965
1959 static void dev_gso_skb_destructor(struct sk_buff *skb) 1966 static void dev_gso_skb_destructor(struct sk_buff *skb)
1960 { 1967 {
1961 struct dev_gso_cb *cb; 1968 struct dev_gso_cb *cb;
1962 1969
1963 do { 1970 do {
1964 struct sk_buff *nskb = skb->next; 1971 struct sk_buff *nskb = skb->next;
1965 1972
1966 skb->next = nskb->next; 1973 skb->next = nskb->next;
1967 nskb->next = NULL; 1974 nskb->next = NULL;
1968 kfree_skb(nskb); 1975 kfree_skb(nskb);
1969 } while (skb->next); 1976 } while (skb->next);
1970 1977
1971 cb = DEV_GSO_CB(skb); 1978 cb = DEV_GSO_CB(skb);
1972 if (cb->destructor) 1979 if (cb->destructor)
1973 cb->destructor(skb); 1980 cb->destructor(skb);
1974 } 1981 }
1975 1982
1976 /** 1983 /**
1977 * dev_gso_segment - Perform emulated hardware segmentation on skb. 1984 * dev_gso_segment - Perform emulated hardware segmentation on skb.
1978 * @skb: buffer to segment 1985 * @skb: buffer to segment
1979 * @features: device features as applicable to this skb 1986 * @features: device features as applicable to this skb
1980 * 1987 *
1981 * This function segments the given skb and stores the list of segments 1988 * This function segments the given skb and stores the list of segments
1982 * in skb->next. 1989 * in skb->next.
1983 */ 1990 */
1984 static int dev_gso_segment(struct sk_buff *skb, int features) 1991 static int dev_gso_segment(struct sk_buff *skb, int features)
1985 { 1992 {
1986 struct sk_buff *segs; 1993 struct sk_buff *segs;
1987 1994
1988 segs = skb_gso_segment(skb, features); 1995 segs = skb_gso_segment(skb, features);
1989 1996
1990 /* Verifying header integrity only. */ 1997 /* Verifying header integrity only. */
1991 if (!segs) 1998 if (!segs)
1992 return 0; 1999 return 0;
1993 2000
1994 if (IS_ERR(segs)) 2001 if (IS_ERR(segs))
1995 return PTR_ERR(segs); 2002 return PTR_ERR(segs);
1996 2003
1997 skb->next = segs; 2004 skb->next = segs;
1998 DEV_GSO_CB(skb)->destructor = skb->destructor; 2005 DEV_GSO_CB(skb)->destructor = skb->destructor;
1999 skb->destructor = dev_gso_skb_destructor; 2006 skb->destructor = dev_gso_skb_destructor;
2000 2007
2001 return 0; 2008 return 0;
2002 } 2009 }
2003 2010
2004 /* 2011 /*
2005 * Try to orphan skb early, right before transmission by the device. 2012 * Try to orphan skb early, right before transmission by the device.
2006 * We cannot orphan skb if tx timestamp is requested or the sk-reference 2013 * We cannot orphan skb if tx timestamp is requested or the sk-reference
2007 * is needed on driver level for other reasons, e.g. see net/can/raw.c 2014 * is needed on driver level for other reasons, e.g. see net/can/raw.c
2008 */ 2015 */
2009 static inline void skb_orphan_try(struct sk_buff *skb) 2016 static inline void skb_orphan_try(struct sk_buff *skb)
2010 { 2017 {
2011 struct sock *sk = skb->sk; 2018 struct sock *sk = skb->sk;
2012 2019
2013 if (sk && !skb_shinfo(skb)->tx_flags) { 2020 if (sk && !skb_shinfo(skb)->tx_flags) {
2014 /* skb_tx_hash() wont be able to get sk. 2021 /* skb_tx_hash() wont be able to get sk.
2015 * We copy sk_hash into skb->rxhash 2022 * We copy sk_hash into skb->rxhash
2016 */ 2023 */
2017 if (!skb->rxhash) 2024 if (!skb->rxhash)
2018 skb->rxhash = sk->sk_hash; 2025 skb->rxhash = sk->sk_hash;
2019 skb_orphan(skb); 2026 skb_orphan(skb);
2020 } 2027 }
2021 } 2028 }
2022 2029
2023 static bool can_checksum_protocol(unsigned long features, __be16 protocol) 2030 static bool can_checksum_protocol(unsigned long features, __be16 protocol)
2024 { 2031 {
2025 return ((features & NETIF_F_GEN_CSUM) || 2032 return ((features & NETIF_F_GEN_CSUM) ||
2026 ((features & NETIF_F_V4_CSUM) && 2033 ((features & NETIF_F_V4_CSUM) &&
2027 protocol == htons(ETH_P_IP)) || 2034 protocol == htons(ETH_P_IP)) ||
2028 ((features & NETIF_F_V6_CSUM) && 2035 ((features & NETIF_F_V6_CSUM) &&
2029 protocol == htons(ETH_P_IPV6)) || 2036 protocol == htons(ETH_P_IPV6)) ||
2030 ((features & NETIF_F_FCOE_CRC) && 2037 ((features & NETIF_F_FCOE_CRC) &&
2031 protocol == htons(ETH_P_FCOE))); 2038 protocol == htons(ETH_P_FCOE)));
2032 } 2039 }
2033 2040
2034 static u32 harmonize_features(struct sk_buff *skb, __be16 protocol, u32 features) 2041 static u32 harmonize_features(struct sk_buff *skb, __be16 protocol, u32 features)
2035 { 2042 {
2036 if (!can_checksum_protocol(features, protocol)) { 2043 if (!can_checksum_protocol(features, protocol)) {
2037 features &= ~NETIF_F_ALL_CSUM; 2044 features &= ~NETIF_F_ALL_CSUM;
2038 features &= ~NETIF_F_SG; 2045 features &= ~NETIF_F_SG;
2039 } else if (illegal_highdma(skb->dev, skb)) { 2046 } else if (illegal_highdma(skb->dev, skb)) {
2040 features &= ~NETIF_F_SG; 2047 features &= ~NETIF_F_SG;
2041 } 2048 }
2042 2049
2043 return features; 2050 return features;
2044 } 2051 }
2045 2052
2046 u32 netif_skb_features(struct sk_buff *skb) 2053 u32 netif_skb_features(struct sk_buff *skb)
2047 { 2054 {
2048 __be16 protocol = skb->protocol; 2055 __be16 protocol = skb->protocol;
2049 u32 features = skb->dev->features; 2056 u32 features = skb->dev->features;
2050 2057
2051 if (protocol == htons(ETH_P_8021Q)) { 2058 if (protocol == htons(ETH_P_8021Q)) {
2052 struct vlan_ethhdr *veh = (struct vlan_ethhdr *)skb->data; 2059 struct vlan_ethhdr *veh = (struct vlan_ethhdr *)skb->data;
2053 protocol = veh->h_vlan_encapsulated_proto; 2060 protocol = veh->h_vlan_encapsulated_proto;
2054 } else if (!vlan_tx_tag_present(skb)) { 2061 } else if (!vlan_tx_tag_present(skb)) {
2055 return harmonize_features(skb, protocol, features); 2062 return harmonize_features(skb, protocol, features);
2056 } 2063 }
2057 2064
2058 features &= (skb->dev->vlan_features | NETIF_F_HW_VLAN_TX); 2065 features &= (skb->dev->vlan_features | NETIF_F_HW_VLAN_TX);
2059 2066
2060 if (protocol != htons(ETH_P_8021Q)) { 2067 if (protocol != htons(ETH_P_8021Q)) {
2061 return harmonize_features(skb, protocol, features); 2068 return harmonize_features(skb, protocol, features);
2062 } else { 2069 } else {
2063 features &= NETIF_F_SG | NETIF_F_HIGHDMA | NETIF_F_FRAGLIST | 2070 features &= NETIF_F_SG | NETIF_F_HIGHDMA | NETIF_F_FRAGLIST |
2064 NETIF_F_GEN_CSUM | NETIF_F_HW_VLAN_TX; 2071 NETIF_F_GEN_CSUM | NETIF_F_HW_VLAN_TX;
2065 return harmonize_features(skb, protocol, features); 2072 return harmonize_features(skb, protocol, features);
2066 } 2073 }
2067 } 2074 }
2068 EXPORT_SYMBOL(netif_skb_features); 2075 EXPORT_SYMBOL(netif_skb_features);
2069 2076
2070 /* 2077 /*
2071 * Returns true if either: 2078 * Returns true if either:
2072 * 1. skb has frag_list and the device doesn't support FRAGLIST, or 2079 * 1. skb has frag_list and the device doesn't support FRAGLIST, or
2073 * 2. skb is fragmented and the device does not support SG, or if 2080 * 2. skb is fragmented and the device does not support SG, or if
2074 * at least one of fragments is in highmem and device does not 2081 * at least one of fragments is in highmem and device does not
2075 * support DMA from it. 2082 * support DMA from it.
2076 */ 2083 */
2077 static inline int skb_needs_linearize(struct sk_buff *skb, 2084 static inline int skb_needs_linearize(struct sk_buff *skb,
2078 int features) 2085 int features)
2079 { 2086 {
2080 return skb_is_nonlinear(skb) && 2087 return skb_is_nonlinear(skb) &&
2081 ((skb_has_frag_list(skb) && 2088 ((skb_has_frag_list(skb) &&
2082 !(features & NETIF_F_FRAGLIST)) || 2089 !(features & NETIF_F_FRAGLIST)) ||
2083 (skb_shinfo(skb)->nr_frags && 2090 (skb_shinfo(skb)->nr_frags &&
2084 !(features & NETIF_F_SG))); 2091 !(features & NETIF_F_SG)));
2085 } 2092 }
2086 2093
2087 int dev_hard_start_xmit(struct sk_buff *skb, struct net_device *dev, 2094 int dev_hard_start_xmit(struct sk_buff *skb, struct net_device *dev,
2088 struct netdev_queue *txq) 2095 struct netdev_queue *txq)
2089 { 2096 {
2090 const struct net_device_ops *ops = dev->netdev_ops; 2097 const struct net_device_ops *ops = dev->netdev_ops;
2091 int rc = NETDEV_TX_OK; 2098 int rc = NETDEV_TX_OK;
2092 2099
2093 if (likely(!skb->next)) { 2100 if (likely(!skb->next)) {
2094 u32 features; 2101 u32 features;
2095 2102
2096 /* 2103 /*
2097 * If device doesn't need skb->dst, release it right now while 2104 * If device doesn't need skb->dst, release it right now while
2098 * its hot in this cpu cache 2105 * its hot in this cpu cache
2099 */ 2106 */
2100 if (dev->priv_flags & IFF_XMIT_DST_RELEASE) 2107 if (dev->priv_flags & IFF_XMIT_DST_RELEASE)
2101 skb_dst_drop(skb); 2108 skb_dst_drop(skb);
2102 2109
2103 if (!list_empty(&ptype_all)) 2110 if (!list_empty(&ptype_all))
2104 dev_queue_xmit_nit(skb, dev); 2111 dev_queue_xmit_nit(skb, dev);
2105 2112
2106 skb_orphan_try(skb); 2113 skb_orphan_try(skb);
2107 2114
2108 features = netif_skb_features(skb); 2115 features = netif_skb_features(skb);
2109 2116
2110 if (vlan_tx_tag_present(skb) && 2117 if (vlan_tx_tag_present(skb) &&
2111 !(features & NETIF_F_HW_VLAN_TX)) { 2118 !(features & NETIF_F_HW_VLAN_TX)) {
2112 skb = __vlan_put_tag(skb, vlan_tx_tag_get(skb)); 2119 skb = __vlan_put_tag(skb, vlan_tx_tag_get(skb));
2113 if (unlikely(!skb)) 2120 if (unlikely(!skb))
2114 goto out; 2121 goto out;
2115 2122
2116 skb->vlan_tci = 0; 2123 skb->vlan_tci = 0;
2117 } 2124 }
2118 2125
2119 if (netif_needs_gso(skb, features)) { 2126 if (netif_needs_gso(skb, features)) {
2120 if (unlikely(dev_gso_segment(skb, features))) 2127 if (unlikely(dev_gso_segment(skb, features)))
2121 goto out_kfree_skb; 2128 goto out_kfree_skb;
2122 if (skb->next) 2129 if (skb->next)
2123 goto gso; 2130 goto gso;
2124 } else { 2131 } else {
2125 if (skb_needs_linearize(skb, features) && 2132 if (skb_needs_linearize(skb, features) &&
2126 __skb_linearize(skb)) 2133 __skb_linearize(skb))
2127 goto out_kfree_skb; 2134 goto out_kfree_skb;
2128 2135
2129 /* If packet is not checksummed and device does not 2136 /* If packet is not checksummed and device does not
2130 * support checksumming for this protocol, complete 2137 * support checksumming for this protocol, complete
2131 * checksumming here. 2138 * checksumming here.
2132 */ 2139 */
2133 if (skb->ip_summed == CHECKSUM_PARTIAL) { 2140 if (skb->ip_summed == CHECKSUM_PARTIAL) {
2134 skb_set_transport_header(skb, 2141 skb_set_transport_header(skb,
2135 skb_checksum_start_offset(skb)); 2142 skb_checksum_start_offset(skb));
2136 if (!(features & NETIF_F_ALL_CSUM) && 2143 if (!(features & NETIF_F_ALL_CSUM) &&
2137 skb_checksum_help(skb)) 2144 skb_checksum_help(skb))
2138 goto out_kfree_skb; 2145 goto out_kfree_skb;
2139 } 2146 }
2140 } 2147 }
2141 2148
2142 rc = ops->ndo_start_xmit(skb, dev); 2149 rc = ops->ndo_start_xmit(skb, dev);
2143 trace_net_dev_xmit(skb, rc); 2150 trace_net_dev_xmit(skb, rc);
2144 if (rc == NETDEV_TX_OK) 2151 if (rc == NETDEV_TX_OK)
2145 txq_trans_update(txq); 2152 txq_trans_update(txq);
2146 return rc; 2153 return rc;
2147 } 2154 }
2148 2155
2149 gso: 2156 gso:
2150 do { 2157 do {
2151 struct sk_buff *nskb = skb->next; 2158 struct sk_buff *nskb = skb->next;
2152 2159
2153 skb->next = nskb->next; 2160 skb->next = nskb->next;
2154 nskb->next = NULL; 2161 nskb->next = NULL;
2155 2162
2156 /* 2163 /*
2157 * If device doesn't need nskb->dst, release it right now while 2164 * If device doesn't need nskb->dst, release it right now while
2158 * its hot in this cpu cache 2165 * its hot in this cpu cache
2159 */ 2166 */
2160 if (dev->priv_flags & IFF_XMIT_DST_RELEASE) 2167 if (dev->priv_flags & IFF_XMIT_DST_RELEASE)
2161 skb_dst_drop(nskb); 2168 skb_dst_drop(nskb);
2162 2169
2163 rc = ops->ndo_start_xmit(nskb, dev); 2170 rc = ops->ndo_start_xmit(nskb, dev);
2164 trace_net_dev_xmit(nskb, rc); 2171 trace_net_dev_xmit(nskb, rc);
2165 if (unlikely(rc != NETDEV_TX_OK)) { 2172 if (unlikely(rc != NETDEV_TX_OK)) {
2166 if (rc & ~NETDEV_TX_MASK) 2173 if (rc & ~NETDEV_TX_MASK)
2167 goto out_kfree_gso_skb; 2174 goto out_kfree_gso_skb;
2168 nskb->next = skb->next; 2175 nskb->next = skb->next;
2169 skb->next = nskb; 2176 skb->next = nskb;
2170 return rc; 2177 return rc;
2171 } 2178 }
2172 txq_trans_update(txq); 2179 txq_trans_update(txq);
2173 if (unlikely(netif_tx_queue_stopped(txq) && skb->next)) 2180 if (unlikely(netif_tx_queue_stopped(txq) && skb->next))
2174 return NETDEV_TX_BUSY; 2181 return NETDEV_TX_BUSY;
2175 } while (skb->next); 2182 } while (skb->next);
2176 2183
2177 out_kfree_gso_skb: 2184 out_kfree_gso_skb:
2178 if (likely(skb->next == NULL)) 2185 if (likely(skb->next == NULL))
2179 skb->destructor = DEV_GSO_CB(skb)->destructor; 2186 skb->destructor = DEV_GSO_CB(skb)->destructor;
2180 out_kfree_skb: 2187 out_kfree_skb:
2181 kfree_skb(skb); 2188 kfree_skb(skb);
2182 out: 2189 out:
2183 return rc; 2190 return rc;
2184 } 2191 }
2185 2192
2186 static u32 hashrnd __read_mostly; 2193 static u32 hashrnd __read_mostly;
2187 2194
2188 /* 2195 /*
2189 * Returns a Tx hash based on the given packet descriptor a Tx queues' number 2196 * Returns a Tx hash based on the given packet descriptor a Tx queues' number
2190 * to be used as a distribution range. 2197 * to be used as a distribution range.
2191 */ 2198 */
2192 u16 __skb_tx_hash(const struct net_device *dev, const struct sk_buff *skb, 2199 u16 __skb_tx_hash(const struct net_device *dev, const struct sk_buff *skb,
2193 unsigned int num_tx_queues) 2200 unsigned int num_tx_queues)
2194 { 2201 {
2195 u32 hash; 2202 u32 hash;
2196 u16 qoffset = 0; 2203 u16 qoffset = 0;
2197 u16 qcount = num_tx_queues; 2204 u16 qcount = num_tx_queues;
2198 2205
2199 if (skb_rx_queue_recorded(skb)) { 2206 if (skb_rx_queue_recorded(skb)) {
2200 hash = skb_get_rx_queue(skb); 2207 hash = skb_get_rx_queue(skb);
2201 while (unlikely(hash >= num_tx_queues)) 2208 while (unlikely(hash >= num_tx_queues))
2202 hash -= num_tx_queues; 2209 hash -= num_tx_queues;
2203 return hash; 2210 return hash;
2204 } 2211 }
2205 2212
2206 if (dev->num_tc) { 2213 if (dev->num_tc) {
2207 u8 tc = netdev_get_prio_tc_map(dev, skb->priority); 2214 u8 tc = netdev_get_prio_tc_map(dev, skb->priority);
2208 qoffset = dev->tc_to_txq[tc].offset; 2215 qoffset = dev->tc_to_txq[tc].offset;
2209 qcount = dev->tc_to_txq[tc].count; 2216 qcount = dev->tc_to_txq[tc].count;
2210 } 2217 }
2211 2218
2212 if (skb->sk && skb->sk->sk_hash) 2219 if (skb->sk && skb->sk->sk_hash)
2213 hash = skb->sk->sk_hash; 2220 hash = skb->sk->sk_hash;
2214 else 2221 else
2215 hash = (__force u16) skb->protocol ^ skb->rxhash; 2222 hash = (__force u16) skb->protocol ^ skb->rxhash;
2216 hash = jhash_1word(hash, hashrnd); 2223 hash = jhash_1word(hash, hashrnd);
2217 2224
2218 return (u16) (((u64) hash * qcount) >> 32) + qoffset; 2225 return (u16) (((u64) hash * qcount) >> 32) + qoffset;
2219 } 2226 }
2220 EXPORT_SYMBOL(__skb_tx_hash); 2227 EXPORT_SYMBOL(__skb_tx_hash);
2221 2228
2222 static inline u16 dev_cap_txqueue(struct net_device *dev, u16 queue_index) 2229 static inline u16 dev_cap_txqueue(struct net_device *dev, u16 queue_index)
2223 { 2230 {
2224 if (unlikely(queue_index >= dev->real_num_tx_queues)) { 2231 if (unlikely(queue_index >= dev->real_num_tx_queues)) {
2225 if (net_ratelimit()) { 2232 if (net_ratelimit()) {
2226 pr_warning("%s selects TX queue %d, but " 2233 pr_warning("%s selects TX queue %d, but "
2227 "real number of TX queues is %d\n", 2234 "real number of TX queues is %d\n",
2228 dev->name, queue_index, dev->real_num_tx_queues); 2235 dev->name, queue_index, dev->real_num_tx_queues);
2229 } 2236 }
2230 return 0; 2237 return 0;
2231 } 2238 }
2232 return queue_index; 2239 return queue_index;
2233 } 2240 }
2234 2241
2235 static inline int get_xps_queue(struct net_device *dev, struct sk_buff *skb) 2242 static inline int get_xps_queue(struct net_device *dev, struct sk_buff *skb)
2236 { 2243 {
2237 #ifdef CONFIG_XPS 2244 #ifdef CONFIG_XPS
2238 struct xps_dev_maps *dev_maps; 2245 struct xps_dev_maps *dev_maps;
2239 struct xps_map *map; 2246 struct xps_map *map;
2240 int queue_index = -1; 2247 int queue_index = -1;
2241 2248
2242 rcu_read_lock(); 2249 rcu_read_lock();
2243 dev_maps = rcu_dereference(dev->xps_maps); 2250 dev_maps = rcu_dereference(dev->xps_maps);
2244 if (dev_maps) { 2251 if (dev_maps) {
2245 map = rcu_dereference( 2252 map = rcu_dereference(
2246 dev_maps->cpu_map[raw_smp_processor_id()]); 2253 dev_maps->cpu_map[raw_smp_processor_id()]);
2247 if (map) { 2254 if (map) {
2248 if (map->len == 1) 2255 if (map->len == 1)
2249 queue_index = map->queues[0]; 2256 queue_index = map->queues[0];
2250 else { 2257 else {
2251 u32 hash; 2258 u32 hash;
2252 if (skb->sk && skb->sk->sk_hash) 2259 if (skb->sk && skb->sk->sk_hash)
2253 hash = skb->sk->sk_hash; 2260 hash = skb->sk->sk_hash;
2254 else 2261 else
2255 hash = (__force u16) skb->protocol ^ 2262 hash = (__force u16) skb->protocol ^
2256 skb->rxhash; 2263 skb->rxhash;
2257 hash = jhash_1word(hash, hashrnd); 2264 hash = jhash_1word(hash, hashrnd);
2258 queue_index = map->queues[ 2265 queue_index = map->queues[
2259 ((u64)hash * map->len) >> 32]; 2266 ((u64)hash * map->len) >> 32];
2260 } 2267 }
2261 if (unlikely(queue_index >= dev->real_num_tx_queues)) 2268 if (unlikely(queue_index >= dev->real_num_tx_queues))
2262 queue_index = -1; 2269 queue_index = -1;
2263 } 2270 }
2264 } 2271 }
2265 rcu_read_unlock(); 2272 rcu_read_unlock();
2266 2273
2267 return queue_index; 2274 return queue_index;
2268 #else 2275 #else
2269 return -1; 2276 return -1;
2270 #endif 2277 #endif
2271 } 2278 }
2272 2279
2273 static struct netdev_queue *dev_pick_tx(struct net_device *dev, 2280 static struct netdev_queue *dev_pick_tx(struct net_device *dev,
2274 struct sk_buff *skb) 2281 struct sk_buff *skb)
2275 { 2282 {
2276 int queue_index; 2283 int queue_index;
2277 const struct net_device_ops *ops = dev->netdev_ops; 2284 const struct net_device_ops *ops = dev->netdev_ops;
2278 2285
2279 if (dev->real_num_tx_queues == 1) 2286 if (dev->real_num_tx_queues == 1)
2280 queue_index = 0; 2287 queue_index = 0;
2281 else if (ops->ndo_select_queue) { 2288 else if (ops->ndo_select_queue) {
2282 queue_index = ops->ndo_select_queue(dev, skb); 2289 queue_index = ops->ndo_select_queue(dev, skb);
2283 queue_index = dev_cap_txqueue(dev, queue_index); 2290 queue_index = dev_cap_txqueue(dev, queue_index);
2284 } else { 2291 } else {
2285 struct sock *sk = skb->sk; 2292 struct sock *sk = skb->sk;
2286 queue_index = sk_tx_queue_get(sk); 2293 queue_index = sk_tx_queue_get(sk);
2287 2294
2288 if (queue_index < 0 || skb->ooo_okay || 2295 if (queue_index < 0 || skb->ooo_okay ||
2289 queue_index >= dev->real_num_tx_queues) { 2296 queue_index >= dev->real_num_tx_queues) {
2290 int old_index = queue_index; 2297 int old_index = queue_index;
2291 2298
2292 queue_index = get_xps_queue(dev, skb); 2299 queue_index = get_xps_queue(dev, skb);
2293 if (queue_index < 0) 2300 if (queue_index < 0)
2294 queue_index = skb_tx_hash(dev, skb); 2301 queue_index = skb_tx_hash(dev, skb);
2295 2302
2296 if (queue_index != old_index && sk) { 2303 if (queue_index != old_index && sk) {
2297 struct dst_entry *dst = 2304 struct dst_entry *dst =
2298 rcu_dereference_check(sk->sk_dst_cache, 1); 2305 rcu_dereference_check(sk->sk_dst_cache, 1);
2299 2306
2300 if (dst && skb_dst(skb) == dst) 2307 if (dst && skb_dst(skb) == dst)
2301 sk_tx_queue_set(sk, queue_index); 2308 sk_tx_queue_set(sk, queue_index);
2302 } 2309 }
2303 } 2310 }
2304 } 2311 }
2305 2312
2306 skb_set_queue_mapping(skb, queue_index); 2313 skb_set_queue_mapping(skb, queue_index);
2307 return netdev_get_tx_queue(dev, queue_index); 2314 return netdev_get_tx_queue(dev, queue_index);
2308 } 2315 }
2309 2316
2310 static inline int __dev_xmit_skb(struct sk_buff *skb, struct Qdisc *q, 2317 static inline int __dev_xmit_skb(struct sk_buff *skb, struct Qdisc *q,
2311 struct net_device *dev, 2318 struct net_device *dev,
2312 struct netdev_queue *txq) 2319 struct netdev_queue *txq)
2313 { 2320 {
2314 spinlock_t *root_lock = qdisc_lock(q); 2321 spinlock_t *root_lock = qdisc_lock(q);
2315 bool contended; 2322 bool contended;
2316 int rc; 2323 int rc;
2317 2324
2318 qdisc_skb_cb(skb)->pkt_len = skb->len; 2325 qdisc_skb_cb(skb)->pkt_len = skb->len;
2319 qdisc_calculate_pkt_len(skb, q); 2326 qdisc_calculate_pkt_len(skb, q);
2320 /* 2327 /*
2321 * Heuristic to force contended enqueues to serialize on a 2328 * Heuristic to force contended enqueues to serialize on a
2322 * separate lock before trying to get qdisc main lock. 2329 * separate lock before trying to get qdisc main lock.
2323 * This permits __QDISC_STATE_RUNNING owner to get the lock more often 2330 * This permits __QDISC_STATE_RUNNING owner to get the lock more often
2324 * and dequeue packets faster. 2331 * and dequeue packets faster.
2325 */ 2332 */
2326 contended = qdisc_is_running(q); 2333 contended = qdisc_is_running(q);
2327 if (unlikely(contended)) 2334 if (unlikely(contended))
2328 spin_lock(&q->busylock); 2335 spin_lock(&q->busylock);
2329 2336
2330 spin_lock(root_lock); 2337 spin_lock(root_lock);
2331 if (unlikely(test_bit(__QDISC_STATE_DEACTIVATED, &q->state))) { 2338 if (unlikely(test_bit(__QDISC_STATE_DEACTIVATED, &q->state))) {
2332 kfree_skb(skb); 2339 kfree_skb(skb);
2333 rc = NET_XMIT_DROP; 2340 rc = NET_XMIT_DROP;
2334 } else if ((q->flags & TCQ_F_CAN_BYPASS) && !qdisc_qlen(q) && 2341 } else if ((q->flags & TCQ_F_CAN_BYPASS) && !qdisc_qlen(q) &&
2335 qdisc_run_begin(q)) { 2342 qdisc_run_begin(q)) {
2336 /* 2343 /*
2337 * This is a work-conserving queue; there are no old skbs 2344 * This is a work-conserving queue; there are no old skbs
2338 * waiting to be sent out; and the qdisc is not running - 2345 * waiting to be sent out; and the qdisc is not running -
2339 * xmit the skb directly. 2346 * xmit the skb directly.
2340 */ 2347 */
2341 if (!(dev->priv_flags & IFF_XMIT_DST_RELEASE)) 2348 if (!(dev->priv_flags & IFF_XMIT_DST_RELEASE))
2342 skb_dst_force(skb); 2349 skb_dst_force(skb);
2343 2350
2344 qdisc_bstats_update(q, skb); 2351 qdisc_bstats_update(q, skb);
2345 2352
2346 if (sch_direct_xmit(skb, q, dev, txq, root_lock)) { 2353 if (sch_direct_xmit(skb, q, dev, txq, root_lock)) {
2347 if (unlikely(contended)) { 2354 if (unlikely(contended)) {
2348 spin_unlock(&q->busylock); 2355 spin_unlock(&q->busylock);
2349 contended = false; 2356 contended = false;
2350 } 2357 }
2351 __qdisc_run(q); 2358 __qdisc_run(q);
2352 } else 2359 } else
2353 qdisc_run_end(q); 2360 qdisc_run_end(q);
2354 2361
2355 rc = NET_XMIT_SUCCESS; 2362 rc = NET_XMIT_SUCCESS;
2356 } else { 2363 } else {
2357 skb_dst_force(skb); 2364 skb_dst_force(skb);
2358 rc = q->enqueue(skb, q) & NET_XMIT_MASK; 2365 rc = q->enqueue(skb, q) & NET_XMIT_MASK;
2359 if (qdisc_run_begin(q)) { 2366 if (qdisc_run_begin(q)) {
2360 if (unlikely(contended)) { 2367 if (unlikely(contended)) {
2361 spin_unlock(&q->busylock); 2368 spin_unlock(&q->busylock);
2362 contended = false; 2369 contended = false;
2363 } 2370 }
2364 __qdisc_run(q); 2371 __qdisc_run(q);
2365 } 2372 }
2366 } 2373 }
2367 spin_unlock(root_lock); 2374 spin_unlock(root_lock);
2368 if (unlikely(contended)) 2375 if (unlikely(contended))
2369 spin_unlock(&q->busylock); 2376 spin_unlock(&q->busylock);
2370 return rc; 2377 return rc;
2371 } 2378 }
2372 2379
2373 static DEFINE_PER_CPU(int, xmit_recursion); 2380 static DEFINE_PER_CPU(int, xmit_recursion);
2374 #define RECURSION_LIMIT 10 2381 #define RECURSION_LIMIT 10
2375 2382
2376 /** 2383 /**
2377 * dev_queue_xmit - transmit a buffer 2384 * dev_queue_xmit - transmit a buffer
2378 * @skb: buffer to transmit 2385 * @skb: buffer to transmit
2379 * 2386 *
2380 * Queue a buffer for transmission to a network device. The caller must 2387 * Queue a buffer for transmission to a network device. The caller must
2381 * have set the device and priority and built the buffer before calling 2388 * have set the device and priority and built the buffer before calling
2382 * this function. The function can be called from an interrupt. 2389 * this function. The function can be called from an interrupt.
2383 * 2390 *
2384 * A negative errno code is returned on a failure. A success does not 2391 * A negative errno code is returned on a failure. A success does not
2385 * guarantee the frame will be transmitted as it may be dropped due 2392 * guarantee the frame will be transmitted as it may be dropped due
2386 * to congestion or traffic shaping. 2393 * to congestion or traffic shaping.
2387 * 2394 *
2388 * ----------------------------------------------------------------------------------- 2395 * -----------------------------------------------------------------------------------
2389 * I notice this method can also return errors from the queue disciplines, 2396 * I notice this method can also return errors from the queue disciplines,
2390 * including NET_XMIT_DROP, which is a positive value. So, errors can also 2397 * including NET_XMIT_DROP, which is a positive value. So, errors can also
2391 * be positive. 2398 * be positive.
2392 * 2399 *
2393 * Regardless of the return value, the skb is consumed, so it is currently 2400 * Regardless of the return value, the skb is consumed, so it is currently
2394 * difficult to retry a send to this method. (You can bump the ref count 2401 * difficult to retry a send to this method. (You can bump the ref count
2395 * before sending to hold a reference for retry if you are careful.) 2402 * before sending to hold a reference for retry if you are careful.)
2396 * 2403 *
2397 * When calling this method, interrupts MUST be enabled. This is because 2404 * When calling this method, interrupts MUST be enabled. This is because
2398 * the BH enable code must have IRQs enabled so that it will not deadlock. 2405 * the BH enable code must have IRQs enabled so that it will not deadlock.
2399 * --BLG 2406 * --BLG
2400 */ 2407 */
2401 int dev_queue_xmit(struct sk_buff *skb) 2408 int dev_queue_xmit(struct sk_buff *skb)
2402 { 2409 {
2403 struct net_device *dev = skb->dev; 2410 struct net_device *dev = skb->dev;
2404 struct netdev_queue *txq; 2411 struct netdev_queue *txq;
2405 struct Qdisc *q; 2412 struct Qdisc *q;
2406 int rc = -ENOMEM; 2413 int rc = -ENOMEM;
2407 2414
2408 /* Disable soft irqs for various locks below. Also 2415 /* Disable soft irqs for various locks below. Also
2409 * stops preemption for RCU. 2416 * stops preemption for RCU.
2410 */ 2417 */
2411 rcu_read_lock_bh(); 2418 rcu_read_lock_bh();
2412 2419
2413 txq = dev_pick_tx(dev, skb); 2420 txq = dev_pick_tx(dev, skb);
2414 q = rcu_dereference_bh(txq->qdisc); 2421 q = rcu_dereference_bh(txq->qdisc);
2415 2422
2416 #ifdef CONFIG_NET_CLS_ACT 2423 #ifdef CONFIG_NET_CLS_ACT
2417 skb->tc_verd = SET_TC_AT(skb->tc_verd, AT_EGRESS); 2424 skb->tc_verd = SET_TC_AT(skb->tc_verd, AT_EGRESS);
2418 #endif 2425 #endif
2419 trace_net_dev_queue(skb); 2426 trace_net_dev_queue(skb);
2420 if (q->enqueue) { 2427 if (q->enqueue) {
2421 rc = __dev_xmit_skb(skb, q, dev, txq); 2428 rc = __dev_xmit_skb(skb, q, dev, txq);
2422 goto out; 2429 goto out;
2423 } 2430 }
2424 2431
2425 /* The device has no queue. Common case for software devices: 2432 /* The device has no queue. Common case for software devices:
2426 loopback, all the sorts of tunnels... 2433 loopback, all the sorts of tunnels...
2427 2434
2428 Really, it is unlikely that netif_tx_lock protection is necessary 2435 Really, it is unlikely that netif_tx_lock protection is necessary
2429 here. (f.e. loopback and IP tunnels are clean ignoring statistics 2436 here. (f.e. loopback and IP tunnels are clean ignoring statistics
2430 counters.) 2437 counters.)
2431 However, it is possible, that they rely on protection 2438 However, it is possible, that they rely on protection
2432 made by us here. 2439 made by us here.
2433 2440
2434 Check this and shot the lock. It is not prone from deadlocks. 2441 Check this and shot the lock. It is not prone from deadlocks.
2435 Either shot noqueue qdisc, it is even simpler 8) 2442 Either shot noqueue qdisc, it is even simpler 8)
2436 */ 2443 */
2437 if (dev->flags & IFF_UP) { 2444 if (dev->flags & IFF_UP) {
2438 int cpu = smp_processor_id(); /* ok because BHs are off */ 2445 int cpu = smp_processor_id(); /* ok because BHs are off */
2439 2446
2440 if (txq->xmit_lock_owner != cpu) { 2447 if (txq->xmit_lock_owner != cpu) {
2441 2448
2442 if (__this_cpu_read(xmit_recursion) > RECURSION_LIMIT) 2449 if (__this_cpu_read(xmit_recursion) > RECURSION_LIMIT)
2443 goto recursion_alert; 2450 goto recursion_alert;
2444 2451
2445 HARD_TX_LOCK(dev, txq, cpu); 2452 HARD_TX_LOCK(dev, txq, cpu);
2446 2453
2447 if (!netif_tx_queue_stopped(txq)) { 2454 if (!netif_tx_queue_stopped(txq)) {
2448 __this_cpu_inc(xmit_recursion); 2455 __this_cpu_inc(xmit_recursion);
2449 rc = dev_hard_start_xmit(skb, dev, txq); 2456 rc = dev_hard_start_xmit(skb, dev, txq);
2450 __this_cpu_dec(xmit_recursion); 2457 __this_cpu_dec(xmit_recursion);
2451 if (dev_xmit_complete(rc)) { 2458 if (dev_xmit_complete(rc)) {
2452 HARD_TX_UNLOCK(dev, txq); 2459 HARD_TX_UNLOCK(dev, txq);
2453 goto out; 2460 goto out;
2454 } 2461 }
2455 } 2462 }
2456 HARD_TX_UNLOCK(dev, txq); 2463 HARD_TX_UNLOCK(dev, txq);
2457 if (net_ratelimit()) 2464 if (net_ratelimit())
2458 printk(KERN_CRIT "Virtual device %s asks to " 2465 printk(KERN_CRIT "Virtual device %s asks to "
2459 "queue packet!\n", dev->name); 2466 "queue packet!\n", dev->name);
2460 } else { 2467 } else {
2461 /* Recursion is detected! It is possible, 2468 /* Recursion is detected! It is possible,
2462 * unfortunately 2469 * unfortunately
2463 */ 2470 */
2464 recursion_alert: 2471 recursion_alert:
2465 if (net_ratelimit()) 2472 if (net_ratelimit())
2466 printk(KERN_CRIT "Dead loop on virtual device " 2473 printk(KERN_CRIT "Dead loop on virtual device "
2467 "%s, fix it urgently!\n", dev->name); 2474 "%s, fix it urgently!\n", dev->name);
2468 } 2475 }
2469 } 2476 }
2470 2477
2471 rc = -ENETDOWN; 2478 rc = -ENETDOWN;
2472 rcu_read_unlock_bh(); 2479 rcu_read_unlock_bh();
2473 2480
2474 kfree_skb(skb); 2481 kfree_skb(skb);
2475 return rc; 2482 return rc;
2476 out: 2483 out:
2477 rcu_read_unlock_bh(); 2484 rcu_read_unlock_bh();
2478 return rc; 2485 return rc;
2479 } 2486 }
2480 EXPORT_SYMBOL(dev_queue_xmit); 2487 EXPORT_SYMBOL(dev_queue_xmit);
2481 2488
2482 2489
2483 /*======================================================================= 2490 /*=======================================================================
2484 Receiver routines 2491 Receiver routines
2485 =======================================================================*/ 2492 =======================================================================*/
2486 2493
2487 int netdev_max_backlog __read_mostly = 1000; 2494 int netdev_max_backlog __read_mostly = 1000;
2488 int netdev_tstamp_prequeue __read_mostly = 1; 2495 int netdev_tstamp_prequeue __read_mostly = 1;
2489 int netdev_budget __read_mostly = 300; 2496 int netdev_budget __read_mostly = 300;
2490 int weight_p __read_mostly = 64; /* old backlog weight */ 2497 int weight_p __read_mostly = 64; /* old backlog weight */
2491 2498
2492 /* Called with irq disabled */ 2499 /* Called with irq disabled */
2493 static inline void ____napi_schedule(struct softnet_data *sd, 2500 static inline void ____napi_schedule(struct softnet_data *sd,
2494 struct napi_struct *napi) 2501 struct napi_struct *napi)
2495 { 2502 {
2496 list_add_tail(&napi->poll_list, &sd->poll_list); 2503 list_add_tail(&napi->poll_list, &sd->poll_list);
2497 __raise_softirq_irqoff(NET_RX_SOFTIRQ); 2504 __raise_softirq_irqoff(NET_RX_SOFTIRQ);
2498 } 2505 }
2499 2506
2500 /* 2507 /*
2501 * __skb_get_rxhash: calculate a flow hash based on src/dst addresses 2508 * __skb_get_rxhash: calculate a flow hash based on src/dst addresses
2502 * and src/dst port numbers. Returns a non-zero hash number on success 2509 * and src/dst port numbers. Returns a non-zero hash number on success
2503 * and 0 on failure. 2510 * and 0 on failure.
2504 */ 2511 */
2505 __u32 __skb_get_rxhash(struct sk_buff *skb) 2512 __u32 __skb_get_rxhash(struct sk_buff *skb)
2506 { 2513 {
2507 int nhoff, hash = 0, poff; 2514 int nhoff, hash = 0, poff;
2508 const struct ipv6hdr *ip6; 2515 const struct ipv6hdr *ip6;
2509 const struct iphdr *ip; 2516 const struct iphdr *ip;
2510 u8 ip_proto; 2517 u8 ip_proto;
2511 u32 addr1, addr2, ihl; 2518 u32 addr1, addr2, ihl;
2512 union { 2519 union {
2513 u32 v32; 2520 u32 v32;
2514 u16 v16[2]; 2521 u16 v16[2];
2515 } ports; 2522 } ports;
2516 2523
2517 nhoff = skb_network_offset(skb); 2524 nhoff = skb_network_offset(skb);
2518 2525
2519 switch (skb->protocol) { 2526 switch (skb->protocol) {
2520 case __constant_htons(ETH_P_IP): 2527 case __constant_htons(ETH_P_IP):
2521 if (!pskb_may_pull(skb, sizeof(*ip) + nhoff)) 2528 if (!pskb_may_pull(skb, sizeof(*ip) + nhoff))
2522 goto done; 2529 goto done;
2523 2530
2524 ip = (const struct iphdr *) (skb->data + nhoff); 2531 ip = (const struct iphdr *) (skb->data + nhoff);
2525 if (ip->frag_off & htons(IP_MF | IP_OFFSET)) 2532 if (ip->frag_off & htons(IP_MF | IP_OFFSET))
2526 ip_proto = 0; 2533 ip_proto = 0;
2527 else 2534 else
2528 ip_proto = ip->protocol; 2535 ip_proto = ip->protocol;
2529 addr1 = (__force u32) ip->saddr; 2536 addr1 = (__force u32) ip->saddr;
2530 addr2 = (__force u32) ip->daddr; 2537 addr2 = (__force u32) ip->daddr;
2531 ihl = ip->ihl; 2538 ihl = ip->ihl;
2532 break; 2539 break;
2533 case __constant_htons(ETH_P_IPV6): 2540 case __constant_htons(ETH_P_IPV6):
2534 if (!pskb_may_pull(skb, sizeof(*ip6) + nhoff)) 2541 if (!pskb_may_pull(skb, sizeof(*ip6) + nhoff))
2535 goto done; 2542 goto done;
2536 2543
2537 ip6 = (const struct ipv6hdr *) (skb->data + nhoff); 2544 ip6 = (const struct ipv6hdr *) (skb->data + nhoff);
2538 ip_proto = ip6->nexthdr; 2545 ip_proto = ip6->nexthdr;
2539 addr1 = (__force u32) ip6->saddr.s6_addr32[3]; 2546 addr1 = (__force u32) ip6->saddr.s6_addr32[3];
2540 addr2 = (__force u32) ip6->daddr.s6_addr32[3]; 2547 addr2 = (__force u32) ip6->daddr.s6_addr32[3];
2541 ihl = (40 >> 2); 2548 ihl = (40 >> 2);
2542 break; 2549 break;
2543 default: 2550 default:
2544 goto done; 2551 goto done;
2545 } 2552 }
2546 2553
2547 ports.v32 = 0; 2554 ports.v32 = 0;
2548 poff = proto_ports_offset(ip_proto); 2555 poff = proto_ports_offset(ip_proto);
2549 if (poff >= 0) { 2556 if (poff >= 0) {
2550 nhoff += ihl * 4 + poff; 2557 nhoff += ihl * 4 + poff;
2551 if (pskb_may_pull(skb, nhoff + 4)) { 2558 if (pskb_may_pull(skb, nhoff + 4)) {
2552 ports.v32 = * (__force u32 *) (skb->data + nhoff); 2559 ports.v32 = * (__force u32 *) (skb->data + nhoff);
2553 if (ports.v16[1] < ports.v16[0]) 2560 if (ports.v16[1] < ports.v16[0])
2554 swap(ports.v16[0], ports.v16[1]); 2561 swap(ports.v16[0], ports.v16[1]);
2555 } 2562 }
2556 } 2563 }
2557 2564
2558 /* get a consistent hash (same value on both flow directions) */ 2565 /* get a consistent hash (same value on both flow directions) */
2559 if (addr2 < addr1) 2566 if (addr2 < addr1)
2560 swap(addr1, addr2); 2567 swap(addr1, addr2);
2561 2568
2562 hash = jhash_3words(addr1, addr2, ports.v32, hashrnd); 2569 hash = jhash_3words(addr1, addr2, ports.v32, hashrnd);
2563 if (!hash) 2570 if (!hash)
2564 hash = 1; 2571 hash = 1;
2565 2572
2566 done: 2573 done:
2567 return hash; 2574 return hash;
2568 } 2575 }
2569 EXPORT_SYMBOL(__skb_get_rxhash); 2576 EXPORT_SYMBOL(__skb_get_rxhash);
2570 2577
2571 #ifdef CONFIG_RPS 2578 #ifdef CONFIG_RPS
2572 2579
2573 /* One global table that all flow-based protocols share. */ 2580 /* One global table that all flow-based protocols share. */
2574 struct rps_sock_flow_table __rcu *rps_sock_flow_table __read_mostly; 2581 struct rps_sock_flow_table __rcu *rps_sock_flow_table __read_mostly;
2575 EXPORT_SYMBOL(rps_sock_flow_table); 2582 EXPORT_SYMBOL(rps_sock_flow_table);
2576 2583
2577 static struct rps_dev_flow * 2584 static struct rps_dev_flow *
2578 set_rps_cpu(struct net_device *dev, struct sk_buff *skb, 2585 set_rps_cpu(struct net_device *dev, struct sk_buff *skb,
2579 struct rps_dev_flow *rflow, u16 next_cpu) 2586 struct rps_dev_flow *rflow, u16 next_cpu)
2580 { 2587 {
2581 u16 tcpu; 2588 u16 tcpu;
2582 2589
2583 tcpu = rflow->cpu = next_cpu; 2590 tcpu = rflow->cpu = next_cpu;
2584 if (tcpu != RPS_NO_CPU) { 2591 if (tcpu != RPS_NO_CPU) {
2585 #ifdef CONFIG_RFS_ACCEL 2592 #ifdef CONFIG_RFS_ACCEL
2586 struct netdev_rx_queue *rxqueue; 2593 struct netdev_rx_queue *rxqueue;
2587 struct rps_dev_flow_table *flow_table; 2594 struct rps_dev_flow_table *flow_table;
2588 struct rps_dev_flow *old_rflow; 2595 struct rps_dev_flow *old_rflow;
2589 u32 flow_id; 2596 u32 flow_id;
2590 u16 rxq_index; 2597 u16 rxq_index;
2591 int rc; 2598 int rc;
2592 2599
2593 /* Should we steer this flow to a different hardware queue? */ 2600 /* Should we steer this flow to a different hardware queue? */
2594 if (!skb_rx_queue_recorded(skb) || !dev->rx_cpu_rmap || 2601 if (!skb_rx_queue_recorded(skb) || !dev->rx_cpu_rmap ||
2595 !(dev->features & NETIF_F_NTUPLE)) 2602 !(dev->features & NETIF_F_NTUPLE))
2596 goto out; 2603 goto out;
2597 rxq_index = cpu_rmap_lookup_index(dev->rx_cpu_rmap, next_cpu); 2604 rxq_index = cpu_rmap_lookup_index(dev->rx_cpu_rmap, next_cpu);
2598 if (rxq_index == skb_get_rx_queue(skb)) 2605 if (rxq_index == skb_get_rx_queue(skb))
2599 goto out; 2606 goto out;
2600 2607
2601 rxqueue = dev->_rx + rxq_index; 2608 rxqueue = dev->_rx + rxq_index;
2602 flow_table = rcu_dereference(rxqueue->rps_flow_table); 2609 flow_table = rcu_dereference(rxqueue->rps_flow_table);
2603 if (!flow_table) 2610 if (!flow_table)
2604 goto out; 2611 goto out;
2605 flow_id = skb->rxhash & flow_table->mask; 2612 flow_id = skb->rxhash & flow_table->mask;
2606 rc = dev->netdev_ops->ndo_rx_flow_steer(dev, skb, 2613 rc = dev->netdev_ops->ndo_rx_flow_steer(dev, skb,
2607 rxq_index, flow_id); 2614 rxq_index, flow_id);
2608 if (rc < 0) 2615 if (rc < 0)
2609 goto out; 2616 goto out;
2610 old_rflow = rflow; 2617 old_rflow = rflow;
2611 rflow = &flow_table->flows[flow_id]; 2618 rflow = &flow_table->flows[flow_id];
2612 rflow->cpu = next_cpu; 2619 rflow->cpu = next_cpu;
2613 rflow->filter = rc; 2620 rflow->filter = rc;
2614 if (old_rflow->filter == rflow->filter) 2621 if (old_rflow->filter == rflow->filter)
2615 old_rflow->filter = RPS_NO_FILTER; 2622 old_rflow->filter = RPS_NO_FILTER;
2616 out: 2623 out:
2617 #endif 2624 #endif
2618 rflow->last_qtail = 2625 rflow->last_qtail =
2619 per_cpu(softnet_data, tcpu).input_queue_head; 2626 per_cpu(softnet_data, tcpu).input_queue_head;
2620 } 2627 }
2621 2628
2622 return rflow; 2629 return rflow;
2623 } 2630 }
2624 2631
2625 /* 2632 /*
2626 * get_rps_cpu is called from netif_receive_skb and returns the target 2633 * get_rps_cpu is called from netif_receive_skb and returns the target
2627 * CPU from the RPS map of the receiving queue for a given skb. 2634 * CPU from the RPS map of the receiving queue for a given skb.
2628 * rcu_read_lock must be held on entry. 2635 * rcu_read_lock must be held on entry.
2629 */ 2636 */
2630 static int get_rps_cpu(struct net_device *dev, struct sk_buff *skb, 2637 static int get_rps_cpu(struct net_device *dev, struct sk_buff *skb,
2631 struct rps_dev_flow **rflowp) 2638 struct rps_dev_flow **rflowp)
2632 { 2639 {
2633 struct netdev_rx_queue *rxqueue; 2640 struct netdev_rx_queue *rxqueue;
2634 struct rps_map *map; 2641 struct rps_map *map;
2635 struct rps_dev_flow_table *flow_table; 2642 struct rps_dev_flow_table *flow_table;
2636 struct rps_sock_flow_table *sock_flow_table; 2643 struct rps_sock_flow_table *sock_flow_table;
2637 int cpu = -1; 2644 int cpu = -1;
2638 u16 tcpu; 2645 u16 tcpu;
2639 2646
2640 if (skb_rx_queue_recorded(skb)) { 2647 if (skb_rx_queue_recorded(skb)) {
2641 u16 index = skb_get_rx_queue(skb); 2648 u16 index = skb_get_rx_queue(skb);
2642 if (unlikely(index >= dev->real_num_rx_queues)) { 2649 if (unlikely(index >= dev->real_num_rx_queues)) {
2643 WARN_ONCE(dev->real_num_rx_queues > 1, 2650 WARN_ONCE(dev->real_num_rx_queues > 1,
2644 "%s received packet on queue %u, but number " 2651 "%s received packet on queue %u, but number "
2645 "of RX queues is %u\n", 2652 "of RX queues is %u\n",
2646 dev->name, index, dev->real_num_rx_queues); 2653 dev->name, index, dev->real_num_rx_queues);
2647 goto done; 2654 goto done;
2648 } 2655 }
2649 rxqueue = dev->_rx + index; 2656 rxqueue = dev->_rx + index;
2650 } else 2657 } else
2651 rxqueue = dev->_rx; 2658 rxqueue = dev->_rx;
2652 2659
2653 map = rcu_dereference(rxqueue->rps_map); 2660 map = rcu_dereference(rxqueue->rps_map);
2654 if (map) { 2661 if (map) {
2655 if (map->len == 1 && 2662 if (map->len == 1 &&
2656 !rcu_dereference_raw(rxqueue->rps_flow_table)) { 2663 !rcu_dereference_raw(rxqueue->rps_flow_table)) {
2657 tcpu = map->cpus[0]; 2664 tcpu = map->cpus[0];
2658 if (cpu_online(tcpu)) 2665 if (cpu_online(tcpu))
2659 cpu = tcpu; 2666 cpu = tcpu;
2660 goto done; 2667 goto done;
2661 } 2668 }
2662 } else if (!rcu_dereference_raw(rxqueue->rps_flow_table)) { 2669 } else if (!rcu_dereference_raw(rxqueue->rps_flow_table)) {
2663 goto done; 2670 goto done;
2664 } 2671 }
2665 2672
2666 skb_reset_network_header(skb); 2673 skb_reset_network_header(skb);
2667 if (!skb_get_rxhash(skb)) 2674 if (!skb_get_rxhash(skb))
2668 goto done; 2675 goto done;
2669 2676
2670 flow_table = rcu_dereference(rxqueue->rps_flow_table); 2677 flow_table = rcu_dereference(rxqueue->rps_flow_table);
2671 sock_flow_table = rcu_dereference(rps_sock_flow_table); 2678 sock_flow_table = rcu_dereference(rps_sock_flow_table);
2672 if (flow_table && sock_flow_table) { 2679 if (flow_table && sock_flow_table) {
2673 u16 next_cpu; 2680 u16 next_cpu;
2674 struct rps_dev_flow *rflow; 2681 struct rps_dev_flow *rflow;
2675 2682
2676 rflow = &flow_table->flows[skb->rxhash & flow_table->mask]; 2683 rflow = &flow_table->flows[skb->rxhash & flow_table->mask];
2677 tcpu = rflow->cpu; 2684 tcpu = rflow->cpu;
2678 2685
2679 next_cpu = sock_flow_table->ents[skb->rxhash & 2686 next_cpu = sock_flow_table->ents[skb->rxhash &
2680 sock_flow_table->mask]; 2687 sock_flow_table->mask];
2681 2688
2682 /* 2689 /*
2683 * If the desired CPU (where last recvmsg was done) is 2690 * If the desired CPU (where last recvmsg was done) is
2684 * different from current CPU (one in the rx-queue flow 2691 * different from current CPU (one in the rx-queue flow
2685 * table entry), switch if one of the following holds: 2692 * table entry), switch if one of the following holds:
2686 * - Current CPU is unset (equal to RPS_NO_CPU). 2693 * - Current CPU is unset (equal to RPS_NO_CPU).
2687 * - Current CPU is offline. 2694 * - Current CPU is offline.
2688 * - The current CPU's queue tail has advanced beyond the 2695 * - The current CPU's queue tail has advanced beyond the
2689 * last packet that was enqueued using this table entry. 2696 * last packet that was enqueued using this table entry.
2690 * This guarantees that all previous packets for the flow 2697 * This guarantees that all previous packets for the flow
2691 * have been dequeued, thus preserving in order delivery. 2698 * have been dequeued, thus preserving in order delivery.
2692 */ 2699 */
2693 if (unlikely(tcpu != next_cpu) && 2700 if (unlikely(tcpu != next_cpu) &&
2694 (tcpu == RPS_NO_CPU || !cpu_online(tcpu) || 2701 (tcpu == RPS_NO_CPU || !cpu_online(tcpu) ||
2695 ((int)(per_cpu(softnet_data, tcpu).input_queue_head - 2702 ((int)(per_cpu(softnet_data, tcpu).input_queue_head -
2696 rflow->last_qtail)) >= 0)) 2703 rflow->last_qtail)) >= 0))
2697 rflow = set_rps_cpu(dev, skb, rflow, next_cpu); 2704 rflow = set_rps_cpu(dev, skb, rflow, next_cpu);
2698 2705
2699 if (tcpu != RPS_NO_CPU && cpu_online(tcpu)) { 2706 if (tcpu != RPS_NO_CPU && cpu_online(tcpu)) {
2700 *rflowp = rflow; 2707 *rflowp = rflow;
2701 cpu = tcpu; 2708 cpu = tcpu;
2702 goto done; 2709 goto done;
2703 } 2710 }
2704 } 2711 }
2705 2712
2706 if (map) { 2713 if (map) {
2707 tcpu = map->cpus[((u64) skb->rxhash * map->len) >> 32]; 2714 tcpu = map->cpus[((u64) skb->rxhash * map->len) >> 32];
2708 2715
2709 if (cpu_online(tcpu)) { 2716 if (cpu_online(tcpu)) {
2710 cpu = tcpu; 2717 cpu = tcpu;
2711 goto done; 2718 goto done;
2712 } 2719 }
2713 } 2720 }
2714 2721
2715 done: 2722 done:
2716 return cpu; 2723 return cpu;
2717 } 2724 }
2718 2725
2719 #ifdef CONFIG_RFS_ACCEL 2726 #ifdef CONFIG_RFS_ACCEL
2720 2727
2721 /** 2728 /**
2722 * rps_may_expire_flow - check whether an RFS hardware filter may be removed 2729 * rps_may_expire_flow - check whether an RFS hardware filter may be removed
2723 * @dev: Device on which the filter was set 2730 * @dev: Device on which the filter was set
2724 * @rxq_index: RX queue index 2731 * @rxq_index: RX queue index
2725 * @flow_id: Flow ID passed to ndo_rx_flow_steer() 2732 * @flow_id: Flow ID passed to ndo_rx_flow_steer()
2726 * @filter_id: Filter ID returned by ndo_rx_flow_steer() 2733 * @filter_id: Filter ID returned by ndo_rx_flow_steer()
2727 * 2734 *
2728 * Drivers that implement ndo_rx_flow_steer() should periodically call 2735 * Drivers that implement ndo_rx_flow_steer() should periodically call
2729 * this function for each installed filter and remove the filters for 2736 * this function for each installed filter and remove the filters for
2730 * which it returns %true. 2737 * which it returns %true.
2731 */ 2738 */
2732 bool rps_may_expire_flow(struct net_device *dev, u16 rxq_index, 2739 bool rps_may_expire_flow(struct net_device *dev, u16 rxq_index,
2733 u32 flow_id, u16 filter_id) 2740 u32 flow_id, u16 filter_id)
2734 { 2741 {
2735 struct netdev_rx_queue *rxqueue = dev->_rx + rxq_index; 2742 struct netdev_rx_queue *rxqueue = dev->_rx + rxq_index;
2736 struct rps_dev_flow_table *flow_table; 2743 struct rps_dev_flow_table *flow_table;
2737 struct rps_dev_flow *rflow; 2744 struct rps_dev_flow *rflow;
2738 bool expire = true; 2745 bool expire = true;
2739 int cpu; 2746 int cpu;
2740 2747
2741 rcu_read_lock(); 2748 rcu_read_lock();
2742 flow_table = rcu_dereference(rxqueue->rps_flow_table); 2749 flow_table = rcu_dereference(rxqueue->rps_flow_table);
2743 if (flow_table && flow_id <= flow_table->mask) { 2750 if (flow_table && flow_id <= flow_table->mask) {
2744 rflow = &flow_table->flows[flow_id]; 2751 rflow = &flow_table->flows[flow_id];
2745 cpu = ACCESS_ONCE(rflow->cpu); 2752 cpu = ACCESS_ONCE(rflow->cpu);
2746 if (rflow->filter == filter_id && cpu != RPS_NO_CPU && 2753 if (rflow->filter == filter_id && cpu != RPS_NO_CPU &&
2747 ((int)(per_cpu(softnet_data, cpu).input_queue_head - 2754 ((int)(per_cpu(softnet_data, cpu).input_queue_head -
2748 rflow->last_qtail) < 2755 rflow->last_qtail) <
2749 (int)(10 * flow_table->mask))) 2756 (int)(10 * flow_table->mask)))
2750 expire = false; 2757 expire = false;
2751 } 2758 }
2752 rcu_read_unlock(); 2759 rcu_read_unlock();
2753 return expire; 2760 return expire;
2754 } 2761 }
2755 EXPORT_SYMBOL(rps_may_expire_flow); 2762 EXPORT_SYMBOL(rps_may_expire_flow);
2756 2763
2757 #endif /* CONFIG_RFS_ACCEL */ 2764 #endif /* CONFIG_RFS_ACCEL */
2758 2765
2759 /* Called from hardirq (IPI) context */ 2766 /* Called from hardirq (IPI) context */
2760 static void rps_trigger_softirq(void *data) 2767 static void rps_trigger_softirq(void *data)
2761 { 2768 {
2762 struct softnet_data *sd = data; 2769 struct softnet_data *sd = data;
2763 2770
2764 ____napi_schedule(sd, &sd->backlog); 2771 ____napi_schedule(sd, &sd->backlog);
2765 sd->received_rps++; 2772 sd->received_rps++;
2766 } 2773 }
2767 2774
2768 #endif /* CONFIG_RPS */ 2775 #endif /* CONFIG_RPS */
2769 2776
2770 /* 2777 /*
2771 * Check if this softnet_data structure is another cpu one 2778 * Check if this softnet_data structure is another cpu one
2772 * If yes, queue it to our IPI list and return 1 2779 * If yes, queue it to our IPI list and return 1
2773 * If no, return 0 2780 * If no, return 0
2774 */ 2781 */
2775 static int rps_ipi_queued(struct softnet_data *sd) 2782 static int rps_ipi_queued(struct softnet_data *sd)
2776 { 2783 {
2777 #ifdef CONFIG_RPS 2784 #ifdef CONFIG_RPS
2778 struct softnet_data *mysd = &__get_cpu_var(softnet_data); 2785 struct softnet_data *mysd = &__get_cpu_var(softnet_data);
2779 2786
2780 if (sd != mysd) { 2787 if (sd != mysd) {
2781 sd->rps_ipi_next = mysd->rps_ipi_list; 2788 sd->rps_ipi_next = mysd->rps_ipi_list;
2782 mysd->rps_ipi_list = sd; 2789 mysd->rps_ipi_list = sd;
2783 2790
2784 __raise_softirq_irqoff(NET_RX_SOFTIRQ); 2791 __raise_softirq_irqoff(NET_RX_SOFTIRQ);
2785 return 1; 2792 return 1;
2786 } 2793 }
2787 #endif /* CONFIG_RPS */ 2794 #endif /* CONFIG_RPS */
2788 return 0; 2795 return 0;
2789 } 2796 }
2790 2797
2791 /* 2798 /*
2792 * enqueue_to_backlog is called to queue an skb to a per CPU backlog 2799 * enqueue_to_backlog is called to queue an skb to a per CPU backlog
2793 * queue (may be a remote CPU queue). 2800 * queue (may be a remote CPU queue).
2794 */ 2801 */
2795 static int enqueue_to_backlog(struct sk_buff *skb, int cpu, 2802 static int enqueue_to_backlog(struct sk_buff *skb, int cpu,
2796 unsigned int *qtail) 2803 unsigned int *qtail)
2797 { 2804 {
2798 struct softnet_data *sd; 2805 struct softnet_data *sd;
2799 unsigned long flags; 2806 unsigned long flags;
2800 2807
2801 sd = &per_cpu(softnet_data, cpu); 2808 sd = &per_cpu(softnet_data, cpu);
2802 2809
2803 local_irq_save(flags); 2810 local_irq_save(flags);
2804 2811
2805 rps_lock(sd); 2812 rps_lock(sd);
2806 if (skb_queue_len(&sd->input_pkt_queue) <= netdev_max_backlog) { 2813 if (skb_queue_len(&sd->input_pkt_queue) <= netdev_max_backlog) {
2807 if (skb_queue_len(&sd->input_pkt_queue)) { 2814 if (skb_queue_len(&sd->input_pkt_queue)) {
2808 enqueue: 2815 enqueue:
2809 __skb_queue_tail(&sd->input_pkt_queue, skb); 2816 __skb_queue_tail(&sd->input_pkt_queue, skb);
2810 input_queue_tail_incr_save(sd, qtail); 2817 input_queue_tail_incr_save(sd, qtail);
2811 rps_unlock(sd); 2818 rps_unlock(sd);
2812 local_irq_restore(flags); 2819 local_irq_restore(flags);
2813 return NET_RX_SUCCESS; 2820 return NET_RX_SUCCESS;
2814 } 2821 }
2815 2822
2816 /* Schedule NAPI for backlog device 2823 /* Schedule NAPI for backlog device
2817 * We can use non atomic operation since we own the queue lock 2824 * We can use non atomic operation since we own the queue lock
2818 */ 2825 */
2819 if (!__test_and_set_bit(NAPI_STATE_SCHED, &sd->backlog.state)) { 2826 if (!__test_and_set_bit(NAPI_STATE_SCHED, &sd->backlog.state)) {
2820 if (!rps_ipi_queued(sd)) 2827 if (!rps_ipi_queued(sd))
2821 ____napi_schedule(sd, &sd->backlog); 2828 ____napi_schedule(sd, &sd->backlog);
2822 } 2829 }
2823 goto enqueue; 2830 goto enqueue;
2824 } 2831 }
2825 2832
2826 sd->dropped++; 2833 sd->dropped++;
2827 rps_unlock(sd); 2834 rps_unlock(sd);
2828 2835
2829 local_irq_restore(flags); 2836 local_irq_restore(flags);
2830 2837
2831 atomic_long_inc(&skb->dev->rx_dropped); 2838 atomic_long_inc(&skb->dev->rx_dropped);
2832 kfree_skb(skb); 2839 kfree_skb(skb);
2833 return NET_RX_DROP; 2840 return NET_RX_DROP;
2834 } 2841 }
2835 2842
2836 /** 2843 /**
2837 * netif_rx - post buffer to the network code 2844 * netif_rx - post buffer to the network code
2838 * @skb: buffer to post 2845 * @skb: buffer to post
2839 * 2846 *
2840 * This function receives a packet from a device driver and queues it for 2847 * This function receives a packet from a device driver and queues it for
2841 * the upper (protocol) levels to process. It always succeeds. The buffer 2848 * the upper (protocol) levels to process. It always succeeds. The buffer
2842 * may be dropped during processing for congestion control or by the 2849 * may be dropped during processing for congestion control or by the
2843 * protocol layers. 2850 * protocol layers.
2844 * 2851 *
2845 * return values: 2852 * return values:
2846 * NET_RX_SUCCESS (no congestion) 2853 * NET_RX_SUCCESS (no congestion)
2847 * NET_RX_DROP (packet was dropped) 2854 * NET_RX_DROP (packet was dropped)
2848 * 2855 *
2849 */ 2856 */
2850 2857
2851 int netif_rx(struct sk_buff *skb) 2858 int netif_rx(struct sk_buff *skb)
2852 { 2859 {
2853 int ret; 2860 int ret;
2854 2861
2855 /* if netpoll wants it, pretend we never saw it */ 2862 /* if netpoll wants it, pretend we never saw it */
2856 if (netpoll_rx(skb)) 2863 if (netpoll_rx(skb))
2857 return NET_RX_DROP; 2864 return NET_RX_DROP;
2858 2865
2859 if (netdev_tstamp_prequeue) 2866 if (netdev_tstamp_prequeue)
2860 net_timestamp_check(skb); 2867 net_timestamp_check(skb);
2861 2868
2862 trace_netif_rx(skb); 2869 trace_netif_rx(skb);
2863 #ifdef CONFIG_RPS 2870 #ifdef CONFIG_RPS
2864 { 2871 {
2865 struct rps_dev_flow voidflow, *rflow = &voidflow; 2872 struct rps_dev_flow voidflow, *rflow = &voidflow;
2866 int cpu; 2873 int cpu;
2867 2874
2868 preempt_disable(); 2875 preempt_disable();
2869 rcu_read_lock(); 2876 rcu_read_lock();
2870 2877
2871 cpu = get_rps_cpu(skb->dev, skb, &rflow); 2878 cpu = get_rps_cpu(skb->dev, skb, &rflow);
2872 if (cpu < 0) 2879 if (cpu < 0)
2873 cpu = smp_processor_id(); 2880 cpu = smp_processor_id();
2874 2881
2875 ret = enqueue_to_backlog(skb, cpu, &rflow->last_qtail); 2882 ret = enqueue_to_backlog(skb, cpu, &rflow->last_qtail);
2876 2883
2877 rcu_read_unlock(); 2884 rcu_read_unlock();
2878 preempt_enable(); 2885 preempt_enable();
2879 } 2886 }
2880 #else 2887 #else
2881 { 2888 {
2882 unsigned int qtail; 2889 unsigned int qtail;
2883 ret = enqueue_to_backlog(skb, get_cpu(), &qtail); 2890 ret = enqueue_to_backlog(skb, get_cpu(), &qtail);
2884 put_cpu(); 2891 put_cpu();
2885 } 2892 }
2886 #endif 2893 #endif
2887 return ret; 2894 return ret;
2888 } 2895 }
2889 EXPORT_SYMBOL(netif_rx); 2896 EXPORT_SYMBOL(netif_rx);
2890 2897
2891 int netif_rx_ni(struct sk_buff *skb) 2898 int netif_rx_ni(struct sk_buff *skb)
2892 { 2899 {
2893 int err; 2900 int err;
2894 2901
2895 preempt_disable(); 2902 preempt_disable();
2896 err = netif_rx(skb); 2903 err = netif_rx(skb);
2897 if (local_softirq_pending()) 2904 if (local_softirq_pending())
2898 do_softirq(); 2905 do_softirq();
2899 preempt_enable(); 2906 preempt_enable();
2900 2907
2901 return err; 2908 return err;
2902 } 2909 }
2903 EXPORT_SYMBOL(netif_rx_ni); 2910 EXPORT_SYMBOL(netif_rx_ni);
2904 2911
2905 static void net_tx_action(struct softirq_action *h) 2912 static void net_tx_action(struct softirq_action *h)
2906 { 2913 {
2907 struct softnet_data *sd = &__get_cpu_var(softnet_data); 2914 struct softnet_data *sd = &__get_cpu_var(softnet_data);
2908 2915
2909 if (sd->completion_queue) { 2916 if (sd->completion_queue) {
2910 struct sk_buff *clist; 2917 struct sk_buff *clist;
2911 2918
2912 local_irq_disable(); 2919 local_irq_disable();
2913 clist = sd->completion_queue; 2920 clist = sd->completion_queue;
2914 sd->completion_queue = NULL; 2921 sd->completion_queue = NULL;
2915 local_irq_enable(); 2922 local_irq_enable();
2916 2923
2917 while (clist) { 2924 while (clist) {
2918 struct sk_buff *skb = clist; 2925 struct sk_buff *skb = clist;
2919 clist = clist->next; 2926 clist = clist->next;
2920 2927
2921 WARN_ON(atomic_read(&skb->users)); 2928 WARN_ON(atomic_read(&skb->users));
2922 trace_kfree_skb(skb, net_tx_action); 2929 trace_kfree_skb(skb, net_tx_action);
2923 __kfree_skb(skb); 2930 __kfree_skb(skb);
2924 } 2931 }
2925 } 2932 }
2926 2933
2927 if (sd->output_queue) { 2934 if (sd->output_queue) {
2928 struct Qdisc *head; 2935 struct Qdisc *head;
2929 2936
2930 local_irq_disable(); 2937 local_irq_disable();
2931 head = sd->output_queue; 2938 head = sd->output_queue;
2932 sd->output_queue = NULL; 2939 sd->output_queue = NULL;
2933 sd->output_queue_tailp = &sd->output_queue; 2940 sd->output_queue_tailp = &sd->output_queue;
2934 local_irq_enable(); 2941 local_irq_enable();
2935 2942
2936 while (head) { 2943 while (head) {
2937 struct Qdisc *q = head; 2944 struct Qdisc *q = head;
2938 spinlock_t *root_lock; 2945 spinlock_t *root_lock;
2939 2946
2940 head = head->next_sched; 2947 head = head->next_sched;
2941 2948
2942 root_lock = qdisc_lock(q); 2949 root_lock = qdisc_lock(q);
2943 if (spin_trylock(root_lock)) { 2950 if (spin_trylock(root_lock)) {
2944 smp_mb__before_clear_bit(); 2951 smp_mb__before_clear_bit();
2945 clear_bit(__QDISC_STATE_SCHED, 2952 clear_bit(__QDISC_STATE_SCHED,
2946 &q->state); 2953 &q->state);
2947 qdisc_run(q); 2954 qdisc_run(q);
2948 spin_unlock(root_lock); 2955 spin_unlock(root_lock);
2949 } else { 2956 } else {
2950 if (!test_bit(__QDISC_STATE_DEACTIVATED, 2957 if (!test_bit(__QDISC_STATE_DEACTIVATED,
2951 &q->state)) { 2958 &q->state)) {
2952 __netif_reschedule(q); 2959 __netif_reschedule(q);
2953 } else { 2960 } else {
2954 smp_mb__before_clear_bit(); 2961 smp_mb__before_clear_bit();
2955 clear_bit(__QDISC_STATE_SCHED, 2962 clear_bit(__QDISC_STATE_SCHED,
2956 &q->state); 2963 &q->state);
2957 } 2964 }
2958 } 2965 }
2959 } 2966 }
2960 } 2967 }
2961 } 2968 }
2962 2969
2963 #if (defined(CONFIG_BRIDGE) || defined(CONFIG_BRIDGE_MODULE)) && \ 2970 #if (defined(CONFIG_BRIDGE) || defined(CONFIG_BRIDGE_MODULE)) && \
2964 (defined(CONFIG_ATM_LANE) || defined(CONFIG_ATM_LANE_MODULE)) 2971 (defined(CONFIG_ATM_LANE) || defined(CONFIG_ATM_LANE_MODULE))
2965 /* This hook is defined here for ATM LANE */ 2972 /* This hook is defined here for ATM LANE */
2966 int (*br_fdb_test_addr_hook)(struct net_device *dev, 2973 int (*br_fdb_test_addr_hook)(struct net_device *dev,
2967 unsigned char *addr) __read_mostly; 2974 unsigned char *addr) __read_mostly;
2968 EXPORT_SYMBOL_GPL(br_fdb_test_addr_hook); 2975 EXPORT_SYMBOL_GPL(br_fdb_test_addr_hook);
2969 #endif 2976 #endif
2970 2977
2971 #ifdef CONFIG_NET_CLS_ACT 2978 #ifdef CONFIG_NET_CLS_ACT
2972 /* TODO: Maybe we should just force sch_ingress to be compiled in 2979 /* TODO: Maybe we should just force sch_ingress to be compiled in
2973 * when CONFIG_NET_CLS_ACT is? otherwise some useless instructions 2980 * when CONFIG_NET_CLS_ACT is? otherwise some useless instructions
2974 * a compare and 2 stores extra right now if we dont have it on 2981 * a compare and 2 stores extra right now if we dont have it on
2975 * but have CONFIG_NET_CLS_ACT 2982 * but have CONFIG_NET_CLS_ACT
2976 * NOTE: This doesn't stop any functionality; if you dont have 2983 * NOTE: This doesn't stop any functionality; if you dont have
2977 * the ingress scheduler, you just can't add policies on ingress. 2984 * the ingress scheduler, you just can't add policies on ingress.
2978 * 2985 *
2979 */ 2986 */
2980 static int ing_filter(struct sk_buff *skb, struct netdev_queue *rxq) 2987 static int ing_filter(struct sk_buff *skb, struct netdev_queue *rxq)
2981 { 2988 {
2982 struct net_device *dev = skb->dev; 2989 struct net_device *dev = skb->dev;
2983 u32 ttl = G_TC_RTTL(skb->tc_verd); 2990 u32 ttl = G_TC_RTTL(skb->tc_verd);
2984 int result = TC_ACT_OK; 2991 int result = TC_ACT_OK;
2985 struct Qdisc *q; 2992 struct Qdisc *q;
2986 2993
2987 if (unlikely(MAX_RED_LOOP < ttl++)) { 2994 if (unlikely(MAX_RED_LOOP < ttl++)) {
2988 if (net_ratelimit()) 2995 if (net_ratelimit())
2989 pr_warning( "Redir loop detected Dropping packet (%d->%d)\n", 2996 pr_warning( "Redir loop detected Dropping packet (%d->%d)\n",
2990 skb->skb_iif, dev->ifindex); 2997 skb->skb_iif, dev->ifindex);
2991 return TC_ACT_SHOT; 2998 return TC_ACT_SHOT;
2992 } 2999 }
2993 3000
2994 skb->tc_verd = SET_TC_RTTL(skb->tc_verd, ttl); 3001 skb->tc_verd = SET_TC_RTTL(skb->tc_verd, ttl);
2995 skb->tc_verd = SET_TC_AT(skb->tc_verd, AT_INGRESS); 3002 skb->tc_verd = SET_TC_AT(skb->tc_verd, AT_INGRESS);
2996 3003
2997 q = rxq->qdisc; 3004 q = rxq->qdisc;
2998 if (q != &noop_qdisc) { 3005 if (q != &noop_qdisc) {
2999 spin_lock(qdisc_lock(q)); 3006 spin_lock(qdisc_lock(q));
3000 if (likely(!test_bit(__QDISC_STATE_DEACTIVATED, &q->state))) 3007 if (likely(!test_bit(__QDISC_STATE_DEACTIVATED, &q->state)))
3001 result = qdisc_enqueue_root(skb, q); 3008 result = qdisc_enqueue_root(skb, q);
3002 spin_unlock(qdisc_lock(q)); 3009 spin_unlock(qdisc_lock(q));
3003 } 3010 }
3004 3011
3005 return result; 3012 return result;
3006 } 3013 }
3007 3014
3008 static inline struct sk_buff *handle_ing(struct sk_buff *skb, 3015 static inline struct sk_buff *handle_ing(struct sk_buff *skb,
3009 struct packet_type **pt_prev, 3016 struct packet_type **pt_prev,
3010 int *ret, struct net_device *orig_dev) 3017 int *ret, struct net_device *orig_dev)
3011 { 3018 {
3012 struct netdev_queue *rxq = rcu_dereference(skb->dev->ingress_queue); 3019 struct netdev_queue *rxq = rcu_dereference(skb->dev->ingress_queue);
3013 3020
3014 if (!rxq || rxq->qdisc == &noop_qdisc) 3021 if (!rxq || rxq->qdisc == &noop_qdisc)
3015 goto out; 3022 goto out;
3016 3023
3017 if (*pt_prev) { 3024 if (*pt_prev) {
3018 *ret = deliver_skb(skb, *pt_prev, orig_dev); 3025 *ret = deliver_skb(skb, *pt_prev, orig_dev);
3019 *pt_prev = NULL; 3026 *pt_prev = NULL;
3020 } 3027 }
3021 3028
3022 switch (ing_filter(skb, rxq)) { 3029 switch (ing_filter(skb, rxq)) {
3023 case TC_ACT_SHOT: 3030 case TC_ACT_SHOT:
3024 case TC_ACT_STOLEN: 3031 case TC_ACT_STOLEN:
3025 kfree_skb(skb); 3032 kfree_skb(skb);
3026 return NULL; 3033 return NULL;
3027 } 3034 }
3028 3035
3029 out: 3036 out:
3030 skb->tc_verd = 0; 3037 skb->tc_verd = 0;
3031 return skb; 3038 return skb;
3032 } 3039 }
3033 #endif 3040 #endif
3034 3041
3035 /** 3042 /**
3036 * netdev_rx_handler_register - register receive handler 3043 * netdev_rx_handler_register - register receive handler
3037 * @dev: device to register a handler for 3044 * @dev: device to register a handler for
3038 * @rx_handler: receive handler to register 3045 * @rx_handler: receive handler to register
3039 * @rx_handler_data: data pointer that is used by rx handler 3046 * @rx_handler_data: data pointer that is used by rx handler
3040 * 3047 *
3041 * Register a receive hander for a device. This handler will then be 3048 * Register a receive hander for a device. This handler will then be
3042 * called from __netif_receive_skb. A negative errno code is returned 3049 * called from __netif_receive_skb. A negative errno code is returned
3043 * on a failure. 3050 * on a failure.
3044 * 3051 *
3045 * The caller must hold the rtnl_mutex. 3052 * The caller must hold the rtnl_mutex.
3046 * 3053 *
3047 * For a general description of rx_handler, see enum rx_handler_result. 3054 * For a general description of rx_handler, see enum rx_handler_result.
3048 */ 3055 */
3049 int netdev_rx_handler_register(struct net_device *dev, 3056 int netdev_rx_handler_register(struct net_device *dev,
3050 rx_handler_func_t *rx_handler, 3057 rx_handler_func_t *rx_handler,
3051 void *rx_handler_data) 3058 void *rx_handler_data)
3052 { 3059 {
3053 ASSERT_RTNL(); 3060 ASSERT_RTNL();
3054 3061
3055 if (dev->rx_handler) 3062 if (dev->rx_handler)
3056 return -EBUSY; 3063 return -EBUSY;
3057 3064
3058 rcu_assign_pointer(dev->rx_handler_data, rx_handler_data); 3065 rcu_assign_pointer(dev->rx_handler_data, rx_handler_data);
3059 rcu_assign_pointer(dev->rx_handler, rx_handler); 3066 rcu_assign_pointer(dev->rx_handler, rx_handler);
3060 3067
3061 return 0; 3068 return 0;
3062 } 3069 }
3063 EXPORT_SYMBOL_GPL(netdev_rx_handler_register); 3070 EXPORT_SYMBOL_GPL(netdev_rx_handler_register);
3064 3071
3065 /** 3072 /**
3066 * netdev_rx_handler_unregister - unregister receive handler 3073 * netdev_rx_handler_unregister - unregister receive handler
3067 * @dev: device to unregister a handler from 3074 * @dev: device to unregister a handler from
3068 * 3075 *
3069 * Unregister a receive hander from a device. 3076 * Unregister a receive hander from a device.
3070 * 3077 *
3071 * The caller must hold the rtnl_mutex. 3078 * The caller must hold the rtnl_mutex.
3072 */ 3079 */
3073 void netdev_rx_handler_unregister(struct net_device *dev) 3080 void netdev_rx_handler_unregister(struct net_device *dev)
3074 { 3081 {
3075 3082
3076 ASSERT_RTNL(); 3083 ASSERT_RTNL();
3077 rcu_assign_pointer(dev->rx_handler, NULL); 3084 rcu_assign_pointer(dev->rx_handler, NULL);
3078 rcu_assign_pointer(dev->rx_handler_data, NULL); 3085 rcu_assign_pointer(dev->rx_handler_data, NULL);
3079 } 3086 }
3080 EXPORT_SYMBOL_GPL(netdev_rx_handler_unregister); 3087 EXPORT_SYMBOL_GPL(netdev_rx_handler_unregister);
3081 3088
3082 static int __netif_receive_skb(struct sk_buff *skb) 3089 static int __netif_receive_skb(struct sk_buff *skb)
3083 { 3090 {
3084 struct packet_type *ptype, *pt_prev; 3091 struct packet_type *ptype, *pt_prev;
3085 rx_handler_func_t *rx_handler; 3092 rx_handler_func_t *rx_handler;
3086 struct net_device *orig_dev; 3093 struct net_device *orig_dev;
3087 struct net_device *null_or_dev; 3094 struct net_device *null_or_dev;
3088 bool deliver_exact = false; 3095 bool deliver_exact = false;
3089 int ret = NET_RX_DROP; 3096 int ret = NET_RX_DROP;
3090 __be16 type; 3097 __be16 type;
3091 3098
3092 if (!netdev_tstamp_prequeue) 3099 if (!netdev_tstamp_prequeue)
3093 net_timestamp_check(skb); 3100 net_timestamp_check(skb);
3094 3101
3095 trace_netif_receive_skb(skb); 3102 trace_netif_receive_skb(skb);
3096 3103
3097 /* if we've gotten here through NAPI, check netpoll */ 3104 /* if we've gotten here through NAPI, check netpoll */
3098 if (netpoll_receive_skb(skb)) 3105 if (netpoll_receive_skb(skb))
3099 return NET_RX_DROP; 3106 return NET_RX_DROP;
3100 3107
3101 if (!skb->skb_iif) 3108 if (!skb->skb_iif)
3102 skb->skb_iif = skb->dev->ifindex; 3109 skb->skb_iif = skb->dev->ifindex;
3103 orig_dev = skb->dev; 3110 orig_dev = skb->dev;
3104 3111
3105 skb_reset_network_header(skb); 3112 skb_reset_network_header(skb);
3106 skb_reset_transport_header(skb); 3113 skb_reset_transport_header(skb);
3107 skb->mac_len = skb->network_header - skb->mac_header; 3114 skb->mac_len = skb->network_header - skb->mac_header;
3108 3115
3109 pt_prev = NULL; 3116 pt_prev = NULL;
3110 3117
3111 rcu_read_lock(); 3118 rcu_read_lock();
3112 3119
3113 another_round: 3120 another_round:
3114 3121
3115 __this_cpu_inc(softnet_data.processed); 3122 __this_cpu_inc(softnet_data.processed);
3116 3123
3117 if (skb->protocol == cpu_to_be16(ETH_P_8021Q)) { 3124 if (skb->protocol == cpu_to_be16(ETH_P_8021Q)) {
3118 skb = vlan_untag(skb); 3125 skb = vlan_untag(skb);
3119 if (unlikely(!skb)) 3126 if (unlikely(!skb))
3120 goto out; 3127 goto out;
3121 } 3128 }
3122 3129
3123 #ifdef CONFIG_NET_CLS_ACT 3130 #ifdef CONFIG_NET_CLS_ACT
3124 if (skb->tc_verd & TC_NCLS) { 3131 if (skb->tc_verd & TC_NCLS) {
3125 skb->tc_verd = CLR_TC_NCLS(skb->tc_verd); 3132 skb->tc_verd = CLR_TC_NCLS(skb->tc_verd);
3126 goto ncls; 3133 goto ncls;
3127 } 3134 }
3128 #endif 3135 #endif
3129 3136
3130 list_for_each_entry_rcu(ptype, &ptype_all, list) { 3137 list_for_each_entry_rcu(ptype, &ptype_all, list) {
3131 if (!ptype->dev || ptype->dev == skb->dev) { 3138 if (!ptype->dev || ptype->dev == skb->dev) {
3132 if (pt_prev) 3139 if (pt_prev)
3133 ret = deliver_skb(skb, pt_prev, orig_dev); 3140 ret = deliver_skb(skb, pt_prev, orig_dev);
3134 pt_prev = ptype; 3141 pt_prev = ptype;
3135 } 3142 }
3136 } 3143 }
3137 3144
3138 #ifdef CONFIG_NET_CLS_ACT 3145 #ifdef CONFIG_NET_CLS_ACT
3139 skb = handle_ing(skb, &pt_prev, &ret, orig_dev); 3146 skb = handle_ing(skb, &pt_prev, &ret, orig_dev);
3140 if (!skb) 3147 if (!skb)
3141 goto out; 3148 goto out;
3142 ncls: 3149 ncls:
3143 #endif 3150 #endif
3144 3151
3145 rx_handler = rcu_dereference(skb->dev->rx_handler); 3152 rx_handler = rcu_dereference(skb->dev->rx_handler);
3146 if (rx_handler) { 3153 if (rx_handler) {
3147 if (pt_prev) { 3154 if (pt_prev) {
3148 ret = deliver_skb(skb, pt_prev, orig_dev); 3155 ret = deliver_skb(skb, pt_prev, orig_dev);
3149 pt_prev = NULL; 3156 pt_prev = NULL;
3150 } 3157 }
3151 switch (rx_handler(&skb)) { 3158 switch (rx_handler(&skb)) {
3152 case RX_HANDLER_CONSUMED: 3159 case RX_HANDLER_CONSUMED:
3153 goto out; 3160 goto out;
3154 case RX_HANDLER_ANOTHER: 3161 case RX_HANDLER_ANOTHER:
3155 goto another_round; 3162 goto another_round;
3156 case RX_HANDLER_EXACT: 3163 case RX_HANDLER_EXACT:
3157 deliver_exact = true; 3164 deliver_exact = true;
3158 case RX_HANDLER_PASS: 3165 case RX_HANDLER_PASS:
3159 break; 3166 break;
3160 default: 3167 default:
3161 BUG(); 3168 BUG();
3162 } 3169 }
3163 } 3170 }
3164 3171
3165 if (vlan_tx_tag_present(skb)) { 3172 if (vlan_tx_tag_present(skb)) {
3166 if (pt_prev) { 3173 if (pt_prev) {
3167 ret = deliver_skb(skb, pt_prev, orig_dev); 3174 ret = deliver_skb(skb, pt_prev, orig_dev);
3168 pt_prev = NULL; 3175 pt_prev = NULL;
3169 } 3176 }
3170 if (vlan_do_receive(&skb)) { 3177 if (vlan_do_receive(&skb)) {
3171 ret = __netif_receive_skb(skb); 3178 ret = __netif_receive_skb(skb);
3172 goto out; 3179 goto out;
3173 } else if (unlikely(!skb)) 3180 } else if (unlikely(!skb))
3174 goto out; 3181 goto out;
3175 } 3182 }
3176 3183
3177 /* deliver only exact match when indicated */ 3184 /* deliver only exact match when indicated */
3178 null_or_dev = deliver_exact ? skb->dev : NULL; 3185 null_or_dev = deliver_exact ? skb->dev : NULL;
3179 3186
3180 type = skb->protocol; 3187 type = skb->protocol;
3181 list_for_each_entry_rcu(ptype, 3188 list_for_each_entry_rcu(ptype,
3182 &ptype_base[ntohs(type) & PTYPE_HASH_MASK], list) { 3189 &ptype_base[ntohs(type) & PTYPE_HASH_MASK], list) {
3183 if (ptype->type == type && 3190 if (ptype->type == type &&
3184 (ptype->dev == null_or_dev || ptype->dev == skb->dev || 3191 (ptype->dev == null_or_dev || ptype->dev == skb->dev ||
3185 ptype->dev == orig_dev)) { 3192 ptype->dev == orig_dev)) {
3186 if (pt_prev) 3193 if (pt_prev)
3187 ret = deliver_skb(skb, pt_prev, orig_dev); 3194 ret = deliver_skb(skb, pt_prev, orig_dev);
3188 pt_prev = ptype; 3195 pt_prev = ptype;
3189 } 3196 }
3190 } 3197 }
3191 3198
3192 if (pt_prev) { 3199 if (pt_prev) {
3193 ret = pt_prev->func(skb, skb->dev, pt_prev, orig_dev); 3200 ret = pt_prev->func(skb, skb->dev, pt_prev, orig_dev);
3194 } else { 3201 } else {
3195 atomic_long_inc(&skb->dev->rx_dropped); 3202 atomic_long_inc(&skb->dev->rx_dropped);
3196 kfree_skb(skb); 3203 kfree_skb(skb);
3197 /* Jamal, now you will not able to escape explaining 3204 /* Jamal, now you will not able to escape explaining
3198 * me how you were going to use this. :-) 3205 * me how you were going to use this. :-)
3199 */ 3206 */
3200 ret = NET_RX_DROP; 3207 ret = NET_RX_DROP;
3201 } 3208 }
3202 3209
3203 out: 3210 out:
3204 rcu_read_unlock(); 3211 rcu_read_unlock();
3205 return ret; 3212 return ret;
3206 } 3213 }
3207 3214
3208 /** 3215 /**
3209 * netif_receive_skb - process receive buffer from network 3216 * netif_receive_skb - process receive buffer from network
3210 * @skb: buffer to process 3217 * @skb: buffer to process
3211 * 3218 *
3212 * netif_receive_skb() is the main receive data processing function. 3219 * netif_receive_skb() is the main receive data processing function.
3213 * It always succeeds. The buffer may be dropped during processing 3220 * It always succeeds. The buffer may be dropped during processing
3214 * for congestion control or by the protocol layers. 3221 * for congestion control or by the protocol layers.
3215 * 3222 *
3216 * This function may only be called from softirq context and interrupts 3223 * This function may only be called from softirq context and interrupts
3217 * should be enabled. 3224 * should be enabled.
3218 * 3225 *
3219 * Return values (usually ignored): 3226 * Return values (usually ignored):
3220 * NET_RX_SUCCESS: no congestion 3227 * NET_RX_SUCCESS: no congestion
3221 * NET_RX_DROP: packet was dropped 3228 * NET_RX_DROP: packet was dropped
3222 */ 3229 */
3223 int netif_receive_skb(struct sk_buff *skb) 3230 int netif_receive_skb(struct sk_buff *skb)
3224 { 3231 {
3225 if (netdev_tstamp_prequeue) 3232 if (netdev_tstamp_prequeue)
3226 net_timestamp_check(skb); 3233 net_timestamp_check(skb);
3227 3234
3228 if (skb_defer_rx_timestamp(skb)) 3235 if (skb_defer_rx_timestamp(skb))
3229 return NET_RX_SUCCESS; 3236 return NET_RX_SUCCESS;
3230 3237
3231 #ifdef CONFIG_RPS 3238 #ifdef CONFIG_RPS
3232 { 3239 {
3233 struct rps_dev_flow voidflow, *rflow = &voidflow; 3240 struct rps_dev_flow voidflow, *rflow = &voidflow;
3234 int cpu, ret; 3241 int cpu, ret;
3235 3242
3236 rcu_read_lock(); 3243 rcu_read_lock();
3237 3244
3238 cpu = get_rps_cpu(skb->dev, skb, &rflow); 3245 cpu = get_rps_cpu(skb->dev, skb, &rflow);
3239 3246
3240 if (cpu >= 0) { 3247 if (cpu >= 0) {
3241 ret = enqueue_to_backlog(skb, cpu, &rflow->last_qtail); 3248 ret = enqueue_to_backlog(skb, cpu, &rflow->last_qtail);
3242 rcu_read_unlock(); 3249 rcu_read_unlock();
3243 } else { 3250 } else {
3244 rcu_read_unlock(); 3251 rcu_read_unlock();
3245 ret = __netif_receive_skb(skb); 3252 ret = __netif_receive_skb(skb);
3246 } 3253 }
3247 3254
3248 return ret; 3255 return ret;
3249 } 3256 }
3250 #else 3257 #else
3251 return __netif_receive_skb(skb); 3258 return __netif_receive_skb(skb);
3252 #endif 3259 #endif
3253 } 3260 }
3254 EXPORT_SYMBOL(netif_receive_skb); 3261 EXPORT_SYMBOL(netif_receive_skb);
3255 3262
3256 /* Network device is going away, flush any packets still pending 3263 /* Network device is going away, flush any packets still pending
3257 * Called with irqs disabled. 3264 * Called with irqs disabled.
3258 */ 3265 */
3259 static void flush_backlog(void *arg) 3266 static void flush_backlog(void *arg)
3260 { 3267 {
3261 struct net_device *dev = arg; 3268 struct net_device *dev = arg;
3262 struct softnet_data *sd = &__get_cpu_var(softnet_data); 3269 struct softnet_data *sd = &__get_cpu_var(softnet_data);
3263 struct sk_buff *skb, *tmp; 3270 struct sk_buff *skb, *tmp;
3264 3271
3265 rps_lock(sd); 3272 rps_lock(sd);
3266 skb_queue_walk_safe(&sd->input_pkt_queue, skb, tmp) { 3273 skb_queue_walk_safe(&sd->input_pkt_queue, skb, tmp) {
3267 if (skb->dev == dev) { 3274 if (skb->dev == dev) {
3268 __skb_unlink(skb, &sd->input_pkt_queue); 3275 __skb_unlink(skb, &sd->input_pkt_queue);
3269 kfree_skb(skb); 3276 kfree_skb(skb);
3270 input_queue_head_incr(sd); 3277 input_queue_head_incr(sd);
3271 } 3278 }
3272 } 3279 }
3273 rps_unlock(sd); 3280 rps_unlock(sd);
3274 3281
3275 skb_queue_walk_safe(&sd->process_queue, skb, tmp) { 3282 skb_queue_walk_safe(&sd->process_queue, skb, tmp) {
3276 if (skb->dev == dev) { 3283 if (skb->dev == dev) {
3277 __skb_unlink(skb, &sd->process_queue); 3284 __skb_unlink(skb, &sd->process_queue);
3278 kfree_skb(skb); 3285 kfree_skb(skb);
3279 input_queue_head_incr(sd); 3286 input_queue_head_incr(sd);
3280 } 3287 }
3281 } 3288 }
3282 } 3289 }
3283 3290
3284 static int napi_gro_complete(struct sk_buff *skb) 3291 static int napi_gro_complete(struct sk_buff *skb)
3285 { 3292 {
3286 struct packet_type *ptype; 3293 struct packet_type *ptype;
3287 __be16 type = skb->protocol; 3294 __be16 type = skb->protocol;
3288 struct list_head *head = &ptype_base[ntohs(type) & PTYPE_HASH_MASK]; 3295 struct list_head *head = &ptype_base[ntohs(type) & PTYPE_HASH_MASK];
3289 int err = -ENOENT; 3296 int err = -ENOENT;
3290 3297
3291 if (NAPI_GRO_CB(skb)->count == 1) { 3298 if (NAPI_GRO_CB(skb)->count == 1) {
3292 skb_shinfo(skb)->gso_size = 0; 3299 skb_shinfo(skb)->gso_size = 0;
3293 goto out; 3300 goto out;
3294 } 3301 }
3295 3302
3296 rcu_read_lock(); 3303 rcu_read_lock();
3297 list_for_each_entry_rcu(ptype, head, list) { 3304 list_for_each_entry_rcu(ptype, head, list) {
3298 if (ptype->type != type || ptype->dev || !ptype->gro_complete) 3305 if (ptype->type != type || ptype->dev || !ptype->gro_complete)
3299 continue; 3306 continue;
3300 3307
3301 err = ptype->gro_complete(skb); 3308 err = ptype->gro_complete(skb);
3302 break; 3309 break;
3303 } 3310 }
3304 rcu_read_unlock(); 3311 rcu_read_unlock();
3305 3312
3306 if (err) { 3313 if (err) {
3307 WARN_ON(&ptype->list == head); 3314 WARN_ON(&ptype->list == head);
3308 kfree_skb(skb); 3315 kfree_skb(skb);
3309 return NET_RX_SUCCESS; 3316 return NET_RX_SUCCESS;
3310 } 3317 }
3311 3318
3312 out: 3319 out:
3313 return netif_receive_skb(skb); 3320 return netif_receive_skb(skb);
3314 } 3321 }
3315 3322
3316 inline void napi_gro_flush(struct napi_struct *napi) 3323 inline void napi_gro_flush(struct napi_struct *napi)
3317 { 3324 {
3318 struct sk_buff *skb, *next; 3325 struct sk_buff *skb, *next;
3319 3326
3320 for (skb = napi->gro_list; skb; skb = next) { 3327 for (skb = napi->gro_list; skb; skb = next) {
3321 next = skb->next; 3328 next = skb->next;
3322 skb->next = NULL; 3329 skb->next = NULL;
3323 napi_gro_complete(skb); 3330 napi_gro_complete(skb);
3324 } 3331 }
3325 3332
3326 napi->gro_count = 0; 3333 napi->gro_count = 0;
3327 napi->gro_list = NULL; 3334 napi->gro_list = NULL;
3328 } 3335 }
3329 EXPORT_SYMBOL(napi_gro_flush); 3336 EXPORT_SYMBOL(napi_gro_flush);
3330 3337
3331 enum gro_result dev_gro_receive(struct napi_struct *napi, struct sk_buff *skb) 3338 enum gro_result dev_gro_receive(struct napi_struct *napi, struct sk_buff *skb)
3332 { 3339 {
3333 struct sk_buff **pp = NULL; 3340 struct sk_buff **pp = NULL;
3334 struct packet_type *ptype; 3341 struct packet_type *ptype;
3335 __be16 type = skb->protocol; 3342 __be16 type = skb->protocol;
3336 struct list_head *head = &ptype_base[ntohs(type) & PTYPE_HASH_MASK]; 3343 struct list_head *head = &ptype_base[ntohs(type) & PTYPE_HASH_MASK];
3337 int same_flow; 3344 int same_flow;
3338 int mac_len; 3345 int mac_len;
3339 enum gro_result ret; 3346 enum gro_result ret;
3340 3347
3341 if (!(skb->dev->features & NETIF_F_GRO) || netpoll_rx_on(skb)) 3348 if (!(skb->dev->features & NETIF_F_GRO) || netpoll_rx_on(skb))
3342 goto normal; 3349 goto normal;
3343 3350
3344 if (skb_is_gso(skb) || skb_has_frag_list(skb)) 3351 if (skb_is_gso(skb) || skb_has_frag_list(skb))
3345 goto normal; 3352 goto normal;
3346 3353
3347 rcu_read_lock(); 3354 rcu_read_lock();
3348 list_for_each_entry_rcu(ptype, head, list) { 3355 list_for_each_entry_rcu(ptype, head, list) {
3349 if (ptype->type != type || ptype->dev || !ptype->gro_receive) 3356 if (ptype->type != type || ptype->dev || !ptype->gro_receive)
3350 continue; 3357 continue;
3351 3358
3352 skb_set_network_header(skb, skb_gro_offset(skb)); 3359 skb_set_network_header(skb, skb_gro_offset(skb));
3353 mac_len = skb->network_header - skb->mac_header; 3360 mac_len = skb->network_header - skb->mac_header;
3354 skb->mac_len = mac_len; 3361 skb->mac_len = mac_len;
3355 NAPI_GRO_CB(skb)->same_flow = 0; 3362 NAPI_GRO_CB(skb)->same_flow = 0;
3356 NAPI_GRO_CB(skb)->flush = 0; 3363 NAPI_GRO_CB(skb)->flush = 0;
3357 NAPI_GRO_CB(skb)->free = 0; 3364 NAPI_GRO_CB(skb)->free = 0;
3358 3365
3359 pp = ptype->gro_receive(&napi->gro_list, skb); 3366 pp = ptype->gro_receive(&napi->gro_list, skb);
3360 break; 3367 break;
3361 } 3368 }
3362 rcu_read_unlock(); 3369 rcu_read_unlock();
3363 3370
3364 if (&ptype->list == head) 3371 if (&ptype->list == head)
3365 goto normal; 3372 goto normal;
3366 3373
3367 same_flow = NAPI_GRO_CB(skb)->same_flow; 3374 same_flow = NAPI_GRO_CB(skb)->same_flow;
3368 ret = NAPI_GRO_CB(skb)->free ? GRO_MERGED_FREE : GRO_MERGED; 3375 ret = NAPI_GRO_CB(skb)->free ? GRO_MERGED_FREE : GRO_MERGED;
3369 3376
3370 if (pp) { 3377 if (pp) {
3371 struct sk_buff *nskb = *pp; 3378 struct sk_buff *nskb = *pp;
3372 3379
3373 *pp = nskb->next; 3380 *pp = nskb->next;
3374 nskb->next = NULL; 3381 nskb->next = NULL;
3375 napi_gro_complete(nskb); 3382 napi_gro_complete(nskb);
3376 napi->gro_count--; 3383 napi->gro_count--;
3377 } 3384 }
3378 3385
3379 if (same_flow) 3386 if (same_flow)
3380 goto ok; 3387 goto ok;
3381 3388
3382 if (NAPI_GRO_CB(skb)->flush || napi->gro_count >= MAX_GRO_SKBS) 3389 if (NAPI_GRO_CB(skb)->flush || napi->gro_count >= MAX_GRO_SKBS)
3383 goto normal; 3390 goto normal;
3384 3391
3385 napi->gro_count++; 3392 napi->gro_count++;
3386 NAPI_GRO_CB(skb)->count = 1; 3393 NAPI_GRO_CB(skb)->count = 1;
3387 skb_shinfo(skb)->gso_size = skb_gro_len(skb); 3394 skb_shinfo(skb)->gso_size = skb_gro_len(skb);
3388 skb->next = napi->gro_list; 3395 skb->next = napi->gro_list;
3389 napi->gro_list = skb; 3396 napi->gro_list = skb;
3390 ret = GRO_HELD; 3397 ret = GRO_HELD;
3391 3398
3392 pull: 3399 pull:
3393 if (skb_headlen(skb) < skb_gro_offset(skb)) { 3400 if (skb_headlen(skb) < skb_gro_offset(skb)) {
3394 int grow = skb_gro_offset(skb) - skb_headlen(skb); 3401 int grow = skb_gro_offset(skb) - skb_headlen(skb);
3395 3402
3396 BUG_ON(skb->end - skb->tail < grow); 3403 BUG_ON(skb->end - skb->tail < grow);
3397 3404
3398 memcpy(skb_tail_pointer(skb), NAPI_GRO_CB(skb)->frag0, grow); 3405 memcpy(skb_tail_pointer(skb), NAPI_GRO_CB(skb)->frag0, grow);
3399 3406
3400 skb->tail += grow; 3407 skb->tail += grow;
3401 skb->data_len -= grow; 3408 skb->data_len -= grow;
3402 3409
3403 skb_shinfo(skb)->frags[0].page_offset += grow; 3410 skb_shinfo(skb)->frags[0].page_offset += grow;
3404 skb_shinfo(skb)->frags[0].size -= grow; 3411 skb_shinfo(skb)->frags[0].size -= grow;
3405 3412
3406 if (unlikely(!skb_shinfo(skb)->frags[0].size)) { 3413 if (unlikely(!skb_shinfo(skb)->frags[0].size)) {
3407 put_page(skb_shinfo(skb)->frags[0].page); 3414 put_page(skb_shinfo(skb)->frags[0].page);
3408 memmove(skb_shinfo(skb)->frags, 3415 memmove(skb_shinfo(skb)->frags,
3409 skb_shinfo(skb)->frags + 1, 3416 skb_shinfo(skb)->frags + 1,
3410 --skb_shinfo(skb)->nr_frags * sizeof(skb_frag_t)); 3417 --skb_shinfo(skb)->nr_frags * sizeof(skb_frag_t));
3411 } 3418 }
3412 } 3419 }
3413 3420
3414 ok: 3421 ok:
3415 return ret; 3422 return ret;
3416 3423
3417 normal: 3424 normal:
3418 ret = GRO_NORMAL; 3425 ret = GRO_NORMAL;
3419 goto pull; 3426 goto pull;
3420 } 3427 }
3421 EXPORT_SYMBOL(dev_gro_receive); 3428 EXPORT_SYMBOL(dev_gro_receive);
3422 3429
3423 static inline gro_result_t 3430 static inline gro_result_t
3424 __napi_gro_receive(struct napi_struct *napi, struct sk_buff *skb) 3431 __napi_gro_receive(struct napi_struct *napi, struct sk_buff *skb)
3425 { 3432 {
3426 struct sk_buff *p; 3433 struct sk_buff *p;
3427 3434
3428 for (p = napi->gro_list; p; p = p->next) { 3435 for (p = napi->gro_list; p; p = p->next) {
3429 unsigned long diffs; 3436 unsigned long diffs;
3430 3437
3431 diffs = (unsigned long)p->dev ^ (unsigned long)skb->dev; 3438 diffs = (unsigned long)p->dev ^ (unsigned long)skb->dev;
3432 diffs |= p->vlan_tci ^ skb->vlan_tci; 3439 diffs |= p->vlan_tci ^ skb->vlan_tci;
3433 diffs |= compare_ether_header(skb_mac_header(p), 3440 diffs |= compare_ether_header(skb_mac_header(p),
3434 skb_gro_mac_header(skb)); 3441 skb_gro_mac_header(skb));
3435 NAPI_GRO_CB(p)->same_flow = !diffs; 3442 NAPI_GRO_CB(p)->same_flow = !diffs;
3436 NAPI_GRO_CB(p)->flush = 0; 3443 NAPI_GRO_CB(p)->flush = 0;
3437 } 3444 }
3438 3445
3439 return dev_gro_receive(napi, skb); 3446 return dev_gro_receive(napi, skb);
3440 } 3447 }
3441 3448
3442 gro_result_t napi_skb_finish(gro_result_t ret, struct sk_buff *skb) 3449 gro_result_t napi_skb_finish(gro_result_t ret, struct sk_buff *skb)
3443 { 3450 {
3444 switch (ret) { 3451 switch (ret) {
3445 case GRO_NORMAL: 3452 case GRO_NORMAL:
3446 if (netif_receive_skb(skb)) 3453 if (netif_receive_skb(skb))
3447 ret = GRO_DROP; 3454 ret = GRO_DROP;
3448 break; 3455 break;
3449 3456
3450 case GRO_DROP: 3457 case GRO_DROP:
3451 case GRO_MERGED_FREE: 3458 case GRO_MERGED_FREE:
3452 kfree_skb(skb); 3459 kfree_skb(skb);
3453 break; 3460 break;
3454 3461
3455 case GRO_HELD: 3462 case GRO_HELD:
3456 case GRO_MERGED: 3463 case GRO_MERGED:
3457 break; 3464 break;
3458 } 3465 }
3459 3466
3460 return ret; 3467 return ret;
3461 } 3468 }
3462 EXPORT_SYMBOL(napi_skb_finish); 3469 EXPORT_SYMBOL(napi_skb_finish);
3463 3470
3464 void skb_gro_reset_offset(struct sk_buff *skb) 3471 void skb_gro_reset_offset(struct sk_buff *skb)
3465 { 3472 {
3466 NAPI_GRO_CB(skb)->data_offset = 0; 3473 NAPI_GRO_CB(skb)->data_offset = 0;
3467 NAPI_GRO_CB(skb)->frag0 = NULL; 3474 NAPI_GRO_CB(skb)->frag0 = NULL;
3468 NAPI_GRO_CB(skb)->frag0_len = 0; 3475 NAPI_GRO_CB(skb)->frag0_len = 0;
3469 3476
3470 if (skb->mac_header == skb->tail && 3477 if (skb->mac_header == skb->tail &&
3471 !PageHighMem(skb_shinfo(skb)->frags[0].page)) { 3478 !PageHighMem(skb_shinfo(skb)->frags[0].page)) {
3472 NAPI_GRO_CB(skb)->frag0 = 3479 NAPI_GRO_CB(skb)->frag0 =
3473 page_address(skb_shinfo(skb)->frags[0].page) + 3480 page_address(skb_shinfo(skb)->frags[0].page) +
3474 skb_shinfo(skb)->frags[0].page_offset; 3481 skb_shinfo(skb)->frags[0].page_offset;
3475 NAPI_GRO_CB(skb)->frag0_len = skb_shinfo(skb)->frags[0].size; 3482 NAPI_GRO_CB(skb)->frag0_len = skb_shinfo(skb)->frags[0].size;
3476 } 3483 }
3477 } 3484 }
3478 EXPORT_SYMBOL(skb_gro_reset_offset); 3485 EXPORT_SYMBOL(skb_gro_reset_offset);
3479 3486
3480 gro_result_t napi_gro_receive(struct napi_struct *napi, struct sk_buff *skb) 3487 gro_result_t napi_gro_receive(struct napi_struct *napi, struct sk_buff *skb)
3481 { 3488 {
3482 skb_gro_reset_offset(skb); 3489 skb_gro_reset_offset(skb);
3483 3490
3484 return napi_skb_finish(__napi_gro_receive(napi, skb), skb); 3491 return napi_skb_finish(__napi_gro_receive(napi, skb), skb);
3485 } 3492 }
3486 EXPORT_SYMBOL(napi_gro_receive); 3493 EXPORT_SYMBOL(napi_gro_receive);
3487 3494
3488 static void napi_reuse_skb(struct napi_struct *napi, struct sk_buff *skb) 3495 static void napi_reuse_skb(struct napi_struct *napi, struct sk_buff *skb)
3489 { 3496 {
3490 __skb_pull(skb, skb_headlen(skb)); 3497 __skb_pull(skb, skb_headlen(skb));
3491 skb_reserve(skb, NET_IP_ALIGN - skb_headroom(skb)); 3498 skb_reserve(skb, NET_IP_ALIGN - skb_headroom(skb));
3492 skb->vlan_tci = 0; 3499 skb->vlan_tci = 0;
3493 skb->dev = napi->dev; 3500 skb->dev = napi->dev;
3494 skb->skb_iif = 0; 3501 skb->skb_iif = 0;
3495 3502
3496 napi->skb = skb; 3503 napi->skb = skb;
3497 } 3504 }
3498 3505
3499 struct sk_buff *napi_get_frags(struct napi_struct *napi) 3506 struct sk_buff *napi_get_frags(struct napi_struct *napi)
3500 { 3507 {
3501 struct sk_buff *skb = napi->skb; 3508 struct sk_buff *skb = napi->skb;
3502 3509
3503 if (!skb) { 3510 if (!skb) {
3504 skb = netdev_alloc_skb_ip_align(napi->dev, GRO_MAX_HEAD); 3511 skb = netdev_alloc_skb_ip_align(napi->dev, GRO_MAX_HEAD);
3505 if (skb) 3512 if (skb)
3506 napi->skb = skb; 3513 napi->skb = skb;
3507 } 3514 }
3508 return skb; 3515 return skb;
3509 } 3516 }
3510 EXPORT_SYMBOL(napi_get_frags); 3517 EXPORT_SYMBOL(napi_get_frags);
3511 3518
3512 gro_result_t napi_frags_finish(struct napi_struct *napi, struct sk_buff *skb, 3519 gro_result_t napi_frags_finish(struct napi_struct *napi, struct sk_buff *skb,
3513 gro_result_t ret) 3520 gro_result_t ret)
3514 { 3521 {
3515 switch (ret) { 3522 switch (ret) {
3516 case GRO_NORMAL: 3523 case GRO_NORMAL:
3517 case GRO_HELD: 3524 case GRO_HELD:
3518 skb->protocol = eth_type_trans(skb, skb->dev); 3525 skb->protocol = eth_type_trans(skb, skb->dev);
3519 3526
3520 if (ret == GRO_HELD) 3527 if (ret == GRO_HELD)
3521 skb_gro_pull(skb, -ETH_HLEN); 3528 skb_gro_pull(skb, -ETH_HLEN);
3522 else if (netif_receive_skb(skb)) 3529 else if (netif_receive_skb(skb))
3523 ret = GRO_DROP; 3530 ret = GRO_DROP;
3524 break; 3531 break;
3525 3532
3526 case GRO_DROP: 3533 case GRO_DROP:
3527 case GRO_MERGED_FREE: 3534 case GRO_MERGED_FREE:
3528 napi_reuse_skb(napi, skb); 3535 napi_reuse_skb(napi, skb);
3529 break; 3536 break;
3530 3537
3531 case GRO_MERGED: 3538 case GRO_MERGED:
3532 break; 3539 break;
3533 } 3540 }
3534 3541
3535 return ret; 3542 return ret;
3536 } 3543 }
3537 EXPORT_SYMBOL(napi_frags_finish); 3544 EXPORT_SYMBOL(napi_frags_finish);
3538 3545
3539 struct sk_buff *napi_frags_skb(struct napi_struct *napi) 3546 struct sk_buff *napi_frags_skb(struct napi_struct *napi)
3540 { 3547 {
3541 struct sk_buff *skb = napi->skb; 3548 struct sk_buff *skb = napi->skb;
3542 struct ethhdr *eth; 3549 struct ethhdr *eth;
3543 unsigned int hlen; 3550 unsigned int hlen;
3544 unsigned int off; 3551 unsigned int off;
3545 3552
3546 napi->skb = NULL; 3553 napi->skb = NULL;
3547 3554
3548 skb_reset_mac_header(skb); 3555 skb_reset_mac_header(skb);
3549 skb_gro_reset_offset(skb); 3556 skb_gro_reset_offset(skb);
3550 3557
3551 off = skb_gro_offset(skb); 3558 off = skb_gro_offset(skb);
3552 hlen = off + sizeof(*eth); 3559 hlen = off + sizeof(*eth);
3553 eth = skb_gro_header_fast(skb, off); 3560 eth = skb_gro_header_fast(skb, off);
3554 if (skb_gro_header_hard(skb, hlen)) { 3561 if (skb_gro_header_hard(skb, hlen)) {
3555 eth = skb_gro_header_slow(skb, hlen, off); 3562 eth = skb_gro_header_slow(skb, hlen, off);
3556 if (unlikely(!eth)) { 3563 if (unlikely(!eth)) {
3557 napi_reuse_skb(napi, skb); 3564 napi_reuse_skb(napi, skb);
3558 skb = NULL; 3565 skb = NULL;
3559 goto out; 3566 goto out;
3560 } 3567 }
3561 } 3568 }
3562 3569
3563 skb_gro_pull(skb, sizeof(*eth)); 3570 skb_gro_pull(skb, sizeof(*eth));
3564 3571
3565 /* 3572 /*
3566 * This works because the only protocols we care about don't require 3573 * This works because the only protocols we care about don't require
3567 * special handling. We'll fix it up properly at the end. 3574 * special handling. We'll fix it up properly at the end.
3568 */ 3575 */
3569 skb->protocol = eth->h_proto; 3576 skb->protocol = eth->h_proto;
3570 3577
3571 out: 3578 out:
3572 return skb; 3579 return skb;
3573 } 3580 }
3574 EXPORT_SYMBOL(napi_frags_skb); 3581 EXPORT_SYMBOL(napi_frags_skb);
3575 3582
3576 gro_result_t napi_gro_frags(struct napi_struct *napi) 3583 gro_result_t napi_gro_frags(struct napi_struct *napi)
3577 { 3584 {
3578 struct sk_buff *skb = napi_frags_skb(napi); 3585 struct sk_buff *skb = napi_frags_skb(napi);
3579 3586
3580 if (!skb) 3587 if (!skb)
3581 return GRO_DROP; 3588 return GRO_DROP;
3582 3589
3583 return napi_frags_finish(napi, skb, __napi_gro_receive(napi, skb)); 3590 return napi_frags_finish(napi, skb, __napi_gro_receive(napi, skb));
3584 } 3591 }
3585 EXPORT_SYMBOL(napi_gro_frags); 3592 EXPORT_SYMBOL(napi_gro_frags);
3586 3593
3587 /* 3594 /*
3588 * net_rps_action sends any pending IPI's for rps. 3595 * net_rps_action sends any pending IPI's for rps.
3589 * Note: called with local irq disabled, but exits with local irq enabled. 3596 * Note: called with local irq disabled, but exits with local irq enabled.
3590 */ 3597 */
3591 static void net_rps_action_and_irq_enable(struct softnet_data *sd) 3598 static void net_rps_action_and_irq_enable(struct softnet_data *sd)
3592 { 3599 {
3593 #ifdef CONFIG_RPS 3600 #ifdef CONFIG_RPS
3594 struct softnet_data *remsd = sd->rps_ipi_list; 3601 struct softnet_data *remsd = sd->rps_ipi_list;
3595 3602
3596 if (remsd) { 3603 if (remsd) {
3597 sd->rps_ipi_list = NULL; 3604 sd->rps_ipi_list = NULL;
3598 3605
3599 local_irq_enable(); 3606 local_irq_enable();
3600 3607
3601 /* Send pending IPI's to kick RPS processing on remote cpus. */ 3608 /* Send pending IPI's to kick RPS processing on remote cpus. */
3602 while (remsd) { 3609 while (remsd) {
3603 struct softnet_data *next = remsd->rps_ipi_next; 3610 struct softnet_data *next = remsd->rps_ipi_next;
3604 3611
3605 if (cpu_online(remsd->cpu)) 3612 if (cpu_online(remsd->cpu))
3606 __smp_call_function_single(remsd->cpu, 3613 __smp_call_function_single(remsd->cpu,
3607 &remsd->csd, 0); 3614 &remsd->csd, 0);
3608 remsd = next; 3615 remsd = next;
3609 } 3616 }
3610 } else 3617 } else
3611 #endif 3618 #endif
3612 local_irq_enable(); 3619 local_irq_enable();
3613 } 3620 }
3614 3621
3615 static int process_backlog(struct napi_struct *napi, int quota) 3622 static int process_backlog(struct napi_struct *napi, int quota)
3616 { 3623 {
3617 int work = 0; 3624 int work = 0;
3618 struct softnet_data *sd = container_of(napi, struct softnet_data, backlog); 3625 struct softnet_data *sd = container_of(napi, struct softnet_data, backlog);
3619 3626
3620 #ifdef CONFIG_RPS 3627 #ifdef CONFIG_RPS
3621 /* Check if we have pending ipi, its better to send them now, 3628 /* Check if we have pending ipi, its better to send them now,
3622 * not waiting net_rx_action() end. 3629 * not waiting net_rx_action() end.
3623 */ 3630 */
3624 if (sd->rps_ipi_list) { 3631 if (sd->rps_ipi_list) {
3625 local_irq_disable(); 3632 local_irq_disable();
3626 net_rps_action_and_irq_enable(sd); 3633 net_rps_action_and_irq_enable(sd);
3627 } 3634 }
3628 #endif 3635 #endif
3629 napi->weight = weight_p; 3636 napi->weight = weight_p;
3630 local_irq_disable(); 3637 local_irq_disable();
3631 while (work < quota) { 3638 while (work < quota) {
3632 struct sk_buff *skb; 3639 struct sk_buff *skb;
3633 unsigned int qlen; 3640 unsigned int qlen;
3634 3641
3635 while ((skb = __skb_dequeue(&sd->process_queue))) { 3642 while ((skb = __skb_dequeue(&sd->process_queue))) {
3636 local_irq_enable(); 3643 local_irq_enable();
3637 __netif_receive_skb(skb); 3644 __netif_receive_skb(skb);
3638 local_irq_disable(); 3645 local_irq_disable();
3639 input_queue_head_incr(sd); 3646 input_queue_head_incr(sd);
3640 if (++work >= quota) { 3647 if (++work >= quota) {
3641 local_irq_enable(); 3648 local_irq_enable();
3642 return work; 3649 return work;
3643 } 3650 }
3644 } 3651 }
3645 3652
3646 rps_lock(sd); 3653 rps_lock(sd);
3647 qlen = skb_queue_len(&sd->input_pkt_queue); 3654 qlen = skb_queue_len(&sd->input_pkt_queue);
3648 if (qlen) 3655 if (qlen)
3649 skb_queue_splice_tail_init(&sd->input_pkt_queue, 3656 skb_queue_splice_tail_init(&sd->input_pkt_queue,
3650 &sd->process_queue); 3657 &sd->process_queue);
3651 3658
3652 if (qlen < quota - work) { 3659 if (qlen < quota - work) {
3653 /* 3660 /*
3654 * Inline a custom version of __napi_complete(). 3661 * Inline a custom version of __napi_complete().
3655 * only current cpu owns and manipulates this napi, 3662 * only current cpu owns and manipulates this napi,
3656 * and NAPI_STATE_SCHED is the only possible flag set on backlog. 3663 * and NAPI_STATE_SCHED is the only possible flag set on backlog.
3657 * we can use a plain write instead of clear_bit(), 3664 * we can use a plain write instead of clear_bit(),
3658 * and we dont need an smp_mb() memory barrier. 3665 * and we dont need an smp_mb() memory barrier.
3659 */ 3666 */
3660 list_del(&napi->poll_list); 3667 list_del(&napi->poll_list);
3661 napi->state = 0; 3668 napi->state = 0;
3662 3669
3663 quota = work + qlen; 3670 quota = work + qlen;
3664 } 3671 }
3665 rps_unlock(sd); 3672 rps_unlock(sd);
3666 } 3673 }
3667 local_irq_enable(); 3674 local_irq_enable();
3668 3675
3669 return work; 3676 return work;
3670 } 3677 }
3671 3678
3672 /** 3679 /**
3673 * __napi_schedule - schedule for receive 3680 * __napi_schedule - schedule for receive
3674 * @n: entry to schedule 3681 * @n: entry to schedule
3675 * 3682 *
3676 * The entry's receive function will be scheduled to run 3683 * The entry's receive function will be scheduled to run
3677 */ 3684 */
3678 void __napi_schedule(struct napi_struct *n) 3685 void __napi_schedule(struct napi_struct *n)
3679 { 3686 {
3680 unsigned long flags; 3687 unsigned long flags;
3681 3688
3682 local_irq_save(flags); 3689 local_irq_save(flags);
3683 ____napi_schedule(&__get_cpu_var(softnet_data), n); 3690 ____napi_schedule(&__get_cpu_var(softnet_data), n);
3684 local_irq_restore(flags); 3691 local_irq_restore(flags);
3685 } 3692 }
3686 EXPORT_SYMBOL(__napi_schedule); 3693 EXPORT_SYMBOL(__napi_schedule);
3687 3694
3688 void __napi_complete(struct napi_struct *n) 3695 void __napi_complete(struct napi_struct *n)
3689 { 3696 {
3690 BUG_ON(!test_bit(NAPI_STATE_SCHED, &n->state)); 3697 BUG_ON(!test_bit(NAPI_STATE_SCHED, &n->state));
3691 BUG_ON(n->gro_list); 3698 BUG_ON(n->gro_list);
3692 3699
3693 list_del(&n->poll_list); 3700 list_del(&n->poll_list);
3694 smp_mb__before_clear_bit(); 3701 smp_mb__before_clear_bit();
3695 clear_bit(NAPI_STATE_SCHED, &n->state); 3702 clear_bit(NAPI_STATE_SCHED, &n->state);
3696 } 3703 }
3697 EXPORT_SYMBOL(__napi_complete); 3704 EXPORT_SYMBOL(__napi_complete);
3698 3705
3699 void napi_complete(struct napi_struct *n) 3706 void napi_complete(struct napi_struct *n)
3700 { 3707 {
3701 unsigned long flags; 3708 unsigned long flags;
3702 3709
3703 /* 3710 /*
3704 * don't let napi dequeue from the cpu poll list 3711 * don't let napi dequeue from the cpu poll list
3705 * just in case its running on a different cpu 3712 * just in case its running on a different cpu
3706 */ 3713 */
3707 if (unlikely(test_bit(NAPI_STATE_NPSVC, &n->state))) 3714 if (unlikely(test_bit(NAPI_STATE_NPSVC, &n->state)))
3708 return; 3715 return;
3709 3716
3710 napi_gro_flush(n); 3717 napi_gro_flush(n);
3711 local_irq_save(flags); 3718 local_irq_save(flags);
3712 __napi_complete(n); 3719 __napi_complete(n);
3713 local_irq_restore(flags); 3720 local_irq_restore(flags);
3714 } 3721 }
3715 EXPORT_SYMBOL(napi_complete); 3722 EXPORT_SYMBOL(napi_complete);
3716 3723
3717 void netif_napi_add(struct net_device *dev, struct napi_struct *napi, 3724 void netif_napi_add(struct net_device *dev, struct napi_struct *napi,
3718 int (*poll)(struct napi_struct *, int), int weight) 3725 int (*poll)(struct napi_struct *, int), int weight)
3719 { 3726 {
3720 INIT_LIST_HEAD(&napi->poll_list); 3727 INIT_LIST_HEAD(&napi->poll_list);
3721 napi->gro_count = 0; 3728 napi->gro_count = 0;
3722 napi->gro_list = NULL; 3729 napi->gro_list = NULL;
3723 napi->skb = NULL; 3730 napi->skb = NULL;
3724 napi->poll = poll; 3731 napi->poll = poll;
3725 napi->weight = weight; 3732 napi->weight = weight;
3726 list_add(&napi->dev_list, &dev->napi_list); 3733 list_add(&napi->dev_list, &dev->napi_list);
3727 napi->dev = dev; 3734 napi->dev = dev;
3728 #ifdef CONFIG_NETPOLL 3735 #ifdef CONFIG_NETPOLL
3729 spin_lock_init(&napi->poll_lock); 3736 spin_lock_init(&napi->poll_lock);
3730 napi->poll_owner = -1; 3737 napi->poll_owner = -1;
3731 #endif 3738 #endif
3732 set_bit(NAPI_STATE_SCHED, &napi->state); 3739 set_bit(NAPI_STATE_SCHED, &napi->state);
3733 } 3740 }
3734 EXPORT_SYMBOL(netif_napi_add); 3741 EXPORT_SYMBOL(netif_napi_add);
3735 3742
3736 void netif_napi_del(struct napi_struct *napi) 3743 void netif_napi_del(struct napi_struct *napi)
3737 { 3744 {
3738 struct sk_buff *skb, *next; 3745 struct sk_buff *skb, *next;
3739 3746
3740 list_del_init(&napi->dev_list); 3747 list_del_init(&napi->dev_list);
3741 napi_free_frags(napi); 3748 napi_free_frags(napi);
3742 3749
3743 for (skb = napi->gro_list; skb; skb = next) { 3750 for (skb = napi->gro_list; skb; skb = next) {
3744 next = skb->next; 3751 next = skb->next;
3745 skb->next = NULL; 3752 skb->next = NULL;
3746 kfree_skb(skb); 3753 kfree_skb(skb);
3747 } 3754 }
3748 3755
3749 napi->gro_list = NULL; 3756 napi->gro_list = NULL;
3750 napi->gro_count = 0; 3757 napi->gro_count = 0;
3751 } 3758 }
3752 EXPORT_SYMBOL(netif_napi_del); 3759 EXPORT_SYMBOL(netif_napi_del);
3753 3760
3754 static void net_rx_action(struct softirq_action *h) 3761 static void net_rx_action(struct softirq_action *h)
3755 { 3762 {
3756 struct softnet_data *sd = &__get_cpu_var(softnet_data); 3763 struct softnet_data *sd = &__get_cpu_var(softnet_data);
3757 unsigned long time_limit = jiffies + 2; 3764 unsigned long time_limit = jiffies + 2;
3758 int budget = netdev_budget; 3765 int budget = netdev_budget;
3759 void *have; 3766 void *have;
3760 3767
3761 local_irq_disable(); 3768 local_irq_disable();
3762 3769
3763 while (!list_empty(&sd->poll_list)) { 3770 while (!list_empty(&sd->poll_list)) {
3764 struct napi_struct *n; 3771 struct napi_struct *n;
3765 int work, weight; 3772 int work, weight;
3766 3773
3767 /* If softirq window is exhuasted then punt. 3774 /* If softirq window is exhuasted then punt.
3768 * Allow this to run for 2 jiffies since which will allow 3775 * Allow this to run for 2 jiffies since which will allow
3769 * an average latency of 1.5/HZ. 3776 * an average latency of 1.5/HZ.
3770 */ 3777 */
3771 if (unlikely(budget <= 0 || time_after(jiffies, time_limit))) 3778 if (unlikely(budget <= 0 || time_after(jiffies, time_limit)))
3772 goto softnet_break; 3779 goto softnet_break;
3773 3780
3774 local_irq_enable(); 3781 local_irq_enable();
3775 3782
3776 /* Even though interrupts have been re-enabled, this 3783 /* Even though interrupts have been re-enabled, this
3777 * access is safe because interrupts can only add new 3784 * access is safe because interrupts can only add new
3778 * entries to the tail of this list, and only ->poll() 3785 * entries to the tail of this list, and only ->poll()
3779 * calls can remove this head entry from the list. 3786 * calls can remove this head entry from the list.
3780 */ 3787 */
3781 n = list_first_entry(&sd->poll_list, struct napi_struct, poll_list); 3788 n = list_first_entry(&sd->poll_list, struct napi_struct, poll_list);
3782 3789
3783 have = netpoll_poll_lock(n); 3790 have = netpoll_poll_lock(n);
3784 3791
3785 weight = n->weight; 3792 weight = n->weight;
3786 3793
3787 /* This NAPI_STATE_SCHED test is for avoiding a race 3794 /* This NAPI_STATE_SCHED test is for avoiding a race
3788 * with netpoll's poll_napi(). Only the entity which 3795 * with netpoll's poll_napi(). Only the entity which
3789 * obtains the lock and sees NAPI_STATE_SCHED set will 3796 * obtains the lock and sees NAPI_STATE_SCHED set will
3790 * actually make the ->poll() call. Therefore we avoid 3797 * actually make the ->poll() call. Therefore we avoid
3791 * accidentally calling ->poll() when NAPI is not scheduled. 3798 * accidentally calling ->poll() when NAPI is not scheduled.
3792 */ 3799 */
3793 work = 0; 3800 work = 0;
3794 if (test_bit(NAPI_STATE_SCHED, &n->state)) { 3801 if (test_bit(NAPI_STATE_SCHED, &n->state)) {
3795 work = n->poll(n, weight); 3802 work = n->poll(n, weight);
3796 trace_napi_poll(n); 3803 trace_napi_poll(n);
3797 } 3804 }
3798 3805
3799 WARN_ON_ONCE(work > weight); 3806 WARN_ON_ONCE(work > weight);
3800 3807
3801 budget -= work; 3808 budget -= work;
3802 3809
3803 local_irq_disable(); 3810 local_irq_disable();
3804 3811
3805 /* Drivers must not modify the NAPI state if they 3812 /* Drivers must not modify the NAPI state if they
3806 * consume the entire weight. In such cases this code 3813 * consume the entire weight. In such cases this code
3807 * still "owns" the NAPI instance and therefore can 3814 * still "owns" the NAPI instance and therefore can
3808 * move the instance around on the list at-will. 3815 * move the instance around on the list at-will.
3809 */ 3816 */
3810 if (unlikely(work == weight)) { 3817 if (unlikely(work == weight)) {
3811 if (unlikely(napi_disable_pending(n))) { 3818 if (unlikely(napi_disable_pending(n))) {
3812 local_irq_enable(); 3819 local_irq_enable();
3813 napi_complete(n); 3820 napi_complete(n);
3814 local_irq_disable(); 3821 local_irq_disable();
3815 } else 3822 } else
3816 list_move_tail(&n->poll_list, &sd->poll_list); 3823 list_move_tail(&n->poll_list, &sd->poll_list);
3817 } 3824 }
3818 3825
3819 netpoll_poll_unlock(have); 3826 netpoll_poll_unlock(have);
3820 } 3827 }
3821 out: 3828 out:
3822 net_rps_action_and_irq_enable(sd); 3829 net_rps_action_and_irq_enable(sd);
3823 3830
3824 #ifdef CONFIG_NET_DMA 3831 #ifdef CONFIG_NET_DMA
3825 /* 3832 /*
3826 * There may not be any more sk_buffs coming right now, so push 3833 * There may not be any more sk_buffs coming right now, so push
3827 * any pending DMA copies to hardware 3834 * any pending DMA copies to hardware
3828 */ 3835 */
3829 dma_issue_pending_all(); 3836 dma_issue_pending_all();
3830 #endif 3837 #endif
3831 3838
3832 return; 3839 return;
3833 3840
3834 softnet_break: 3841 softnet_break:
3835 sd->time_squeeze++; 3842 sd->time_squeeze++;
3836 __raise_softirq_irqoff(NET_RX_SOFTIRQ); 3843 __raise_softirq_irqoff(NET_RX_SOFTIRQ);
3837 goto out; 3844 goto out;
3838 } 3845 }
3839 3846
3840 static gifconf_func_t *gifconf_list[NPROTO]; 3847 static gifconf_func_t *gifconf_list[NPROTO];
3841 3848
3842 /** 3849 /**
3843 * register_gifconf - register a SIOCGIF handler 3850 * register_gifconf - register a SIOCGIF handler
3844 * @family: Address family 3851 * @family: Address family
3845 * @gifconf: Function handler 3852 * @gifconf: Function handler
3846 * 3853 *
3847 * Register protocol dependent address dumping routines. The handler 3854 * Register protocol dependent address dumping routines. The handler
3848 * that is passed must not be freed or reused until it has been replaced 3855 * that is passed must not be freed or reused until it has been replaced
3849 * by another handler. 3856 * by another handler.
3850 */ 3857 */
3851 int register_gifconf(unsigned int family, gifconf_func_t *gifconf) 3858 int register_gifconf(unsigned int family, gifconf_func_t *gifconf)
3852 { 3859 {
3853 if (family >= NPROTO) 3860 if (family >= NPROTO)
3854 return -EINVAL; 3861 return -EINVAL;
3855 gifconf_list[family] = gifconf; 3862 gifconf_list[family] = gifconf;
3856 return 0; 3863 return 0;
3857 } 3864 }
3858 EXPORT_SYMBOL(register_gifconf); 3865 EXPORT_SYMBOL(register_gifconf);
3859 3866
3860 3867
3861 /* 3868 /*
3862 * Map an interface index to its name (SIOCGIFNAME) 3869 * Map an interface index to its name (SIOCGIFNAME)
3863 */ 3870 */
3864 3871
3865 /* 3872 /*
3866 * We need this ioctl for efficient implementation of the 3873 * We need this ioctl for efficient implementation of the
3867 * if_indextoname() function required by the IPv6 API. Without 3874 * if_indextoname() function required by the IPv6 API. Without
3868 * it, we would have to search all the interfaces to find a 3875 * it, we would have to search all the interfaces to find a
3869 * match. --pb 3876 * match. --pb
3870 */ 3877 */
3871 3878
3872 static int dev_ifname(struct net *net, struct ifreq __user *arg) 3879 static int dev_ifname(struct net *net, struct ifreq __user *arg)
3873 { 3880 {
3874 struct net_device *dev; 3881 struct net_device *dev;
3875 struct ifreq ifr; 3882 struct ifreq ifr;
3876 3883
3877 /* 3884 /*
3878 * Fetch the caller's info block. 3885 * Fetch the caller's info block.
3879 */ 3886 */
3880 3887
3881 if (copy_from_user(&ifr, arg, sizeof(struct ifreq))) 3888 if (copy_from_user(&ifr, arg, sizeof(struct ifreq)))
3882 return -EFAULT; 3889 return -EFAULT;
3883 3890
3884 rcu_read_lock(); 3891 rcu_read_lock();
3885 dev = dev_get_by_index_rcu(net, ifr.ifr_ifindex); 3892 dev = dev_get_by_index_rcu(net, ifr.ifr_ifindex);
3886 if (!dev) { 3893 if (!dev) {
3887 rcu_read_unlock(); 3894 rcu_read_unlock();
3888 return -ENODEV; 3895 return -ENODEV;
3889 } 3896 }
3890 3897
3891 strcpy(ifr.ifr_name, dev->name); 3898 strcpy(ifr.ifr_name, dev->name);
3892 rcu_read_unlock(); 3899 rcu_read_unlock();
3893 3900
3894 if (copy_to_user(arg, &ifr, sizeof(struct ifreq))) 3901 if (copy_to_user(arg, &ifr, sizeof(struct ifreq)))
3895 return -EFAULT; 3902 return -EFAULT;
3896 return 0; 3903 return 0;
3897 } 3904 }
3898 3905
3899 /* 3906 /*
3900 * Perform a SIOCGIFCONF call. This structure will change 3907 * Perform a SIOCGIFCONF call. This structure will change
3901 * size eventually, and there is nothing I can do about it. 3908 * size eventually, and there is nothing I can do about it.
3902 * Thus we will need a 'compatibility mode'. 3909 * Thus we will need a 'compatibility mode'.
3903 */ 3910 */
3904 3911
3905 static int dev_ifconf(struct net *net, char __user *arg) 3912 static int dev_ifconf(struct net *net, char __user *arg)
3906 { 3913 {
3907 struct ifconf ifc; 3914 struct ifconf ifc;
3908 struct net_device *dev; 3915 struct net_device *dev;
3909 char __user *pos; 3916 char __user *pos;
3910 int len; 3917 int len;
3911 int total; 3918 int total;
3912 int i; 3919 int i;
3913 3920
3914 /* 3921 /*
3915 * Fetch the caller's info block. 3922 * Fetch the caller's info block.
3916 */ 3923 */
3917 3924
3918 if (copy_from_user(&ifc, arg, sizeof(struct ifconf))) 3925 if (copy_from_user(&ifc, arg, sizeof(struct ifconf)))
3919 return -EFAULT; 3926 return -EFAULT;
3920 3927
3921 pos = ifc.ifc_buf; 3928 pos = ifc.ifc_buf;
3922 len = ifc.ifc_len; 3929 len = ifc.ifc_len;
3923 3930
3924 /* 3931 /*
3925 * Loop over the interfaces, and write an info block for each. 3932 * Loop over the interfaces, and write an info block for each.
3926 */ 3933 */
3927 3934
3928 total = 0; 3935 total = 0;
3929 for_each_netdev(net, dev) { 3936 for_each_netdev(net, dev) {
3930 for (i = 0; i < NPROTO; i++) { 3937 for (i = 0; i < NPROTO; i++) {
3931 if (gifconf_list[i]) { 3938 if (gifconf_list[i]) {
3932 int done; 3939 int done;
3933 if (!pos) 3940 if (!pos)
3934 done = gifconf_list[i](dev, NULL, 0); 3941 done = gifconf_list[i](dev, NULL, 0);
3935 else 3942 else
3936 done = gifconf_list[i](dev, pos + total, 3943 done = gifconf_list[i](dev, pos + total,
3937 len - total); 3944 len - total);
3938 if (done < 0) 3945 if (done < 0)
3939 return -EFAULT; 3946 return -EFAULT;
3940 total += done; 3947 total += done;
3941 } 3948 }
3942 } 3949 }
3943 } 3950 }
3944 3951
3945 /* 3952 /*
3946 * All done. Write the updated control block back to the caller. 3953 * All done. Write the updated control block back to the caller.
3947 */ 3954 */
3948 ifc.ifc_len = total; 3955 ifc.ifc_len = total;
3949 3956
3950 /* 3957 /*
3951 * Both BSD and Solaris return 0 here, so we do too. 3958 * Both BSD and Solaris return 0 here, so we do too.
3952 */ 3959 */
3953 return copy_to_user(arg, &ifc, sizeof(struct ifconf)) ? -EFAULT : 0; 3960 return copy_to_user(arg, &ifc, sizeof(struct ifconf)) ? -EFAULT : 0;
3954 } 3961 }
3955 3962
3956 #ifdef CONFIG_PROC_FS 3963 #ifdef CONFIG_PROC_FS
3957 /* 3964 /*
3958 * This is invoked by the /proc filesystem handler to display a device 3965 * This is invoked by the /proc filesystem handler to display a device
3959 * in detail. 3966 * in detail.
3960 */ 3967 */
3961 void *dev_seq_start(struct seq_file *seq, loff_t *pos) 3968 void *dev_seq_start(struct seq_file *seq, loff_t *pos)
3962 __acquires(RCU) 3969 __acquires(RCU)
3963 { 3970 {
3964 struct net *net = seq_file_net(seq); 3971 struct net *net = seq_file_net(seq);
3965 loff_t off; 3972 loff_t off;
3966 struct net_device *dev; 3973 struct net_device *dev;
3967 3974
3968 rcu_read_lock(); 3975 rcu_read_lock();
3969 if (!*pos) 3976 if (!*pos)
3970 return SEQ_START_TOKEN; 3977 return SEQ_START_TOKEN;
3971 3978
3972 off = 1; 3979 off = 1;
3973 for_each_netdev_rcu(net, dev) 3980 for_each_netdev_rcu(net, dev)
3974 if (off++ == *pos) 3981 if (off++ == *pos)
3975 return dev; 3982 return dev;
3976 3983
3977 return NULL; 3984 return NULL;
3978 } 3985 }
3979 3986
3980 void *dev_seq_next(struct seq_file *seq, void *v, loff_t *pos) 3987 void *dev_seq_next(struct seq_file *seq, void *v, loff_t *pos)
3981 { 3988 {
3982 struct net_device *dev = v; 3989 struct net_device *dev = v;
3983 3990
3984 if (v == SEQ_START_TOKEN) 3991 if (v == SEQ_START_TOKEN)
3985 dev = first_net_device_rcu(seq_file_net(seq)); 3992 dev = first_net_device_rcu(seq_file_net(seq));
3986 else 3993 else
3987 dev = next_net_device_rcu(dev); 3994 dev = next_net_device_rcu(dev);
3988 3995
3989 ++*pos; 3996 ++*pos;
3990 return dev; 3997 return dev;
3991 } 3998 }
3992 3999
3993 void dev_seq_stop(struct seq_file *seq, void *v) 4000 void dev_seq_stop(struct seq_file *seq, void *v)
3994 __releases(RCU) 4001 __releases(RCU)
3995 { 4002 {
3996 rcu_read_unlock(); 4003 rcu_read_unlock();
3997 } 4004 }
3998 4005
3999 static void dev_seq_printf_stats(struct seq_file *seq, struct net_device *dev) 4006 static void dev_seq_printf_stats(struct seq_file *seq, struct net_device *dev)
4000 { 4007 {
4001 struct rtnl_link_stats64 temp; 4008 struct rtnl_link_stats64 temp;
4002 const struct rtnl_link_stats64 *stats = dev_get_stats(dev, &temp); 4009 const struct rtnl_link_stats64 *stats = dev_get_stats(dev, &temp);
4003 4010
4004 seq_printf(seq, "%6s: %7llu %7llu %4llu %4llu %4llu %5llu %10llu %9llu " 4011 seq_printf(seq, "%6s: %7llu %7llu %4llu %4llu %4llu %5llu %10llu %9llu "
4005 "%8llu %7llu %4llu %4llu %4llu %5llu %7llu %10llu\n", 4012 "%8llu %7llu %4llu %4llu %4llu %5llu %7llu %10llu\n",
4006 dev->name, stats->rx_bytes, stats->rx_packets, 4013 dev->name, stats->rx_bytes, stats->rx_packets,
4007 stats->rx_errors, 4014 stats->rx_errors,
4008 stats->rx_dropped + stats->rx_missed_errors, 4015 stats->rx_dropped + stats->rx_missed_errors,
4009 stats->rx_fifo_errors, 4016 stats->rx_fifo_errors,
4010 stats->rx_length_errors + stats->rx_over_errors + 4017 stats->rx_length_errors + stats->rx_over_errors +
4011 stats->rx_crc_errors + stats->rx_frame_errors, 4018 stats->rx_crc_errors + stats->rx_frame_errors,
4012 stats->rx_compressed, stats->multicast, 4019 stats->rx_compressed, stats->multicast,
4013 stats->tx_bytes, stats->tx_packets, 4020 stats->tx_bytes, stats->tx_packets,
4014 stats->tx_errors, stats->tx_dropped, 4021 stats->tx_errors, stats->tx_dropped,
4015 stats->tx_fifo_errors, stats->collisions, 4022 stats->tx_fifo_errors, stats->collisions,
4016 stats->tx_carrier_errors + 4023 stats->tx_carrier_errors +
4017 stats->tx_aborted_errors + 4024 stats->tx_aborted_errors +
4018 stats->tx_window_errors + 4025 stats->tx_window_errors +
4019 stats->tx_heartbeat_errors, 4026 stats->tx_heartbeat_errors,
4020 stats->tx_compressed); 4027 stats->tx_compressed);
4021 } 4028 }
4022 4029
4023 /* 4030 /*
4024 * Called from the PROCfs module. This now uses the new arbitrary sized 4031 * Called from the PROCfs module. This now uses the new arbitrary sized
4025 * /proc/net interface to create /proc/net/dev 4032 * /proc/net interface to create /proc/net/dev
4026 */ 4033 */
4027 static int dev_seq_show(struct seq_file *seq, void *v) 4034 static int dev_seq_show(struct seq_file *seq, void *v)
4028 { 4035 {
4029 if (v == SEQ_START_TOKEN) 4036 if (v == SEQ_START_TOKEN)
4030 seq_puts(seq, "Inter-| Receive " 4037 seq_puts(seq, "Inter-| Receive "
4031 " | Transmit\n" 4038 " | Transmit\n"
4032 " face |bytes packets errs drop fifo frame " 4039 " face |bytes packets errs drop fifo frame "
4033 "compressed multicast|bytes packets errs " 4040 "compressed multicast|bytes packets errs "
4034 "drop fifo colls carrier compressed\n"); 4041 "drop fifo colls carrier compressed\n");
4035 else 4042 else
4036 dev_seq_printf_stats(seq, v); 4043 dev_seq_printf_stats(seq, v);
4037 return 0; 4044 return 0;
4038 } 4045 }
4039 4046
4040 static struct softnet_data *softnet_get_online(loff_t *pos) 4047 static struct softnet_data *softnet_get_online(loff_t *pos)
4041 { 4048 {
4042 struct softnet_data *sd = NULL; 4049 struct softnet_data *sd = NULL;
4043 4050
4044 while (*pos < nr_cpu_ids) 4051 while (*pos < nr_cpu_ids)
4045 if (cpu_online(*pos)) { 4052 if (cpu_online(*pos)) {
4046 sd = &per_cpu(softnet_data, *pos); 4053 sd = &per_cpu(softnet_data, *pos);
4047 break; 4054 break;
4048 } else 4055 } else
4049 ++*pos; 4056 ++*pos;
4050 return sd; 4057 return sd;
4051 } 4058 }
4052 4059
4053 static void *softnet_seq_start(struct seq_file *seq, loff_t *pos) 4060 static void *softnet_seq_start(struct seq_file *seq, loff_t *pos)
4054 { 4061 {
4055 return softnet_get_online(pos); 4062 return softnet_get_online(pos);
4056 } 4063 }
4057 4064
4058 static void *softnet_seq_next(struct seq_file *seq, void *v, loff_t *pos) 4065 static void *softnet_seq_next(struct seq_file *seq, void *v, loff_t *pos)
4059 { 4066 {
4060 ++*pos; 4067 ++*pos;
4061 return softnet_get_online(pos); 4068 return softnet_get_online(pos);
4062 } 4069 }
4063 4070
4064 static void softnet_seq_stop(struct seq_file *seq, void *v) 4071 static void softnet_seq_stop(struct seq_file *seq, void *v)
4065 { 4072 {
4066 } 4073 }
4067 4074
4068 static int softnet_seq_show(struct seq_file *seq, void *v) 4075 static int softnet_seq_show(struct seq_file *seq, void *v)
4069 { 4076 {
4070 struct softnet_data *sd = v; 4077 struct softnet_data *sd = v;
4071 4078
4072 seq_printf(seq, "%08x %08x %08x %08x %08x %08x %08x %08x %08x %08x\n", 4079 seq_printf(seq, "%08x %08x %08x %08x %08x %08x %08x %08x %08x %08x\n",
4073 sd->processed, sd->dropped, sd->time_squeeze, 0, 4080 sd->processed, sd->dropped, sd->time_squeeze, 0,
4074 0, 0, 0, 0, /* was fastroute */ 4081 0, 0, 0, 0, /* was fastroute */
4075 sd->cpu_collision, sd->received_rps); 4082 sd->cpu_collision, sd->received_rps);
4076 return 0; 4083 return 0;
4077 } 4084 }
4078 4085
4079 static const struct seq_operations dev_seq_ops = { 4086 static const struct seq_operations dev_seq_ops = {
4080 .start = dev_seq_start, 4087 .start = dev_seq_start,
4081 .next = dev_seq_next, 4088 .next = dev_seq_next,
4082 .stop = dev_seq_stop, 4089 .stop = dev_seq_stop,
4083 .show = dev_seq_show, 4090 .show = dev_seq_show,
4084 }; 4091 };
4085 4092
4086 static int dev_seq_open(struct inode *inode, struct file *file) 4093 static int dev_seq_open(struct inode *inode, struct file *file)
4087 { 4094 {
4088 return seq_open_net(inode, file, &dev_seq_ops, 4095 return seq_open_net(inode, file, &dev_seq_ops,
4089 sizeof(struct seq_net_private)); 4096 sizeof(struct seq_net_private));
4090 } 4097 }
4091 4098
4092 static const struct file_operations dev_seq_fops = { 4099 static const struct file_operations dev_seq_fops = {
4093 .owner = THIS_MODULE, 4100 .owner = THIS_MODULE,
4094 .open = dev_seq_open, 4101 .open = dev_seq_open,
4095 .read = seq_read, 4102 .read = seq_read,
4096 .llseek = seq_lseek, 4103 .llseek = seq_lseek,
4097 .release = seq_release_net, 4104 .release = seq_release_net,
4098 }; 4105 };
4099 4106
4100 static const struct seq_operations softnet_seq_ops = { 4107 static const struct seq_operations softnet_seq_ops = {
4101 .start = softnet_seq_start, 4108 .start = softnet_seq_start,
4102 .next = softnet_seq_next, 4109 .next = softnet_seq_next,
4103 .stop = softnet_seq_stop, 4110 .stop = softnet_seq_stop,
4104 .show = softnet_seq_show, 4111 .show = softnet_seq_show,
4105 }; 4112 };
4106 4113
4107 static int softnet_seq_open(struct inode *inode, struct file *file) 4114 static int softnet_seq_open(struct inode *inode, struct file *file)
4108 { 4115 {
4109 return seq_open(file, &softnet_seq_ops); 4116 return seq_open(file, &softnet_seq_ops);
4110 } 4117 }
4111 4118
4112 static const struct file_operations softnet_seq_fops = { 4119 static const struct file_operations softnet_seq_fops = {
4113 .owner = THIS_MODULE, 4120 .owner = THIS_MODULE,
4114 .open = softnet_seq_open, 4121 .open = softnet_seq_open,
4115 .read = seq_read, 4122 .read = seq_read,
4116 .llseek = seq_lseek, 4123 .llseek = seq_lseek,
4117 .release = seq_release, 4124 .release = seq_release,
4118 }; 4125 };
4119 4126
4120 static void *ptype_get_idx(loff_t pos) 4127 static void *ptype_get_idx(loff_t pos)
4121 { 4128 {
4122 struct packet_type *pt = NULL; 4129 struct packet_type *pt = NULL;
4123 loff_t i = 0; 4130 loff_t i = 0;
4124 int t; 4131 int t;
4125 4132
4126 list_for_each_entry_rcu(pt, &ptype_all, list) { 4133 list_for_each_entry_rcu(pt, &ptype_all, list) {
4127 if (i == pos) 4134 if (i == pos)
4128 return pt; 4135 return pt;
4129 ++i; 4136 ++i;
4130 } 4137 }
4131 4138
4132 for (t = 0; t < PTYPE_HASH_SIZE; t++) { 4139 for (t = 0; t < PTYPE_HASH_SIZE; t++) {
4133 list_for_each_entry_rcu(pt, &ptype_base[t], list) { 4140 list_for_each_entry_rcu(pt, &ptype_base[t], list) {
4134 if (i == pos) 4141 if (i == pos)
4135 return pt; 4142 return pt;
4136 ++i; 4143 ++i;
4137 } 4144 }
4138 } 4145 }
4139 return NULL; 4146 return NULL;
4140 } 4147 }
4141 4148
4142 static void *ptype_seq_start(struct seq_file *seq, loff_t *pos) 4149 static void *ptype_seq_start(struct seq_file *seq, loff_t *pos)
4143 __acquires(RCU) 4150 __acquires(RCU)
4144 { 4151 {
4145 rcu_read_lock(); 4152 rcu_read_lock();
4146 return *pos ? ptype_get_idx(*pos - 1) : SEQ_START_TOKEN; 4153 return *pos ? ptype_get_idx(*pos - 1) : SEQ_START_TOKEN;
4147 } 4154 }
4148 4155
4149 static void *ptype_seq_next(struct seq_file *seq, void *v, loff_t *pos) 4156 static void *ptype_seq_next(struct seq_file *seq, void *v, loff_t *pos)
4150 { 4157 {
4151 struct packet_type *pt; 4158 struct packet_type *pt;
4152 struct list_head *nxt; 4159 struct list_head *nxt;
4153 int hash; 4160 int hash;
4154 4161
4155 ++*pos; 4162 ++*pos;
4156 if (v == SEQ_START_TOKEN) 4163 if (v == SEQ_START_TOKEN)
4157 return ptype_get_idx(0); 4164 return ptype_get_idx(0);
4158 4165
4159 pt = v; 4166 pt = v;
4160 nxt = pt->list.next; 4167 nxt = pt->list.next;
4161 if (pt->type == htons(ETH_P_ALL)) { 4168 if (pt->type == htons(ETH_P_ALL)) {
4162 if (nxt != &ptype_all) 4169 if (nxt != &ptype_all)
4163 goto found; 4170 goto found;
4164 hash = 0; 4171 hash = 0;
4165 nxt = ptype_base[0].next; 4172 nxt = ptype_base[0].next;
4166 } else 4173 } else
4167 hash = ntohs(pt->type) & PTYPE_HASH_MASK; 4174 hash = ntohs(pt->type) & PTYPE_HASH_MASK;
4168 4175
4169 while (nxt == &ptype_base[hash]) { 4176 while (nxt == &ptype_base[hash]) {
4170 if (++hash >= PTYPE_HASH_SIZE) 4177 if (++hash >= PTYPE_HASH_SIZE)
4171 return NULL; 4178 return NULL;
4172 nxt = ptype_base[hash].next; 4179 nxt = ptype_base[hash].next;
4173 } 4180 }
4174 found: 4181 found:
4175 return list_entry(nxt, struct packet_type, list); 4182 return list_entry(nxt, struct packet_type, list);
4176 } 4183 }
4177 4184
4178 static void ptype_seq_stop(struct seq_file *seq, void *v) 4185 static void ptype_seq_stop(struct seq_file *seq, void *v)
4179 __releases(RCU) 4186 __releases(RCU)
4180 { 4187 {
4181 rcu_read_unlock(); 4188 rcu_read_unlock();
4182 } 4189 }
4183 4190
4184 static int ptype_seq_show(struct seq_file *seq, void *v) 4191 static int ptype_seq_show(struct seq_file *seq, void *v)
4185 { 4192 {
4186 struct packet_type *pt = v; 4193 struct packet_type *pt = v;
4187 4194
4188 if (v == SEQ_START_TOKEN) 4195 if (v == SEQ_START_TOKEN)
4189 seq_puts(seq, "Type Device Function\n"); 4196 seq_puts(seq, "Type Device Function\n");
4190 else if (pt->dev == NULL || dev_net(pt->dev) == seq_file_net(seq)) { 4197 else if (pt->dev == NULL || dev_net(pt->dev) == seq_file_net(seq)) {
4191 if (pt->type == htons(ETH_P_ALL)) 4198 if (pt->type == htons(ETH_P_ALL))
4192 seq_puts(seq, "ALL "); 4199 seq_puts(seq, "ALL ");
4193 else 4200 else
4194 seq_printf(seq, "%04x", ntohs(pt->type)); 4201 seq_printf(seq, "%04x", ntohs(pt->type));
4195 4202
4196 seq_printf(seq, " %-8s %pF\n", 4203 seq_printf(seq, " %-8s %pF\n",
4197 pt->dev ? pt->dev->name : "", pt->func); 4204 pt->dev ? pt->dev->name : "", pt->func);
4198 } 4205 }
4199 4206
4200 return 0; 4207 return 0;
4201 } 4208 }
4202 4209
4203 static const struct seq_operations ptype_seq_ops = { 4210 static const struct seq_operations ptype_seq_ops = {
4204 .start = ptype_seq_start, 4211 .start = ptype_seq_start,
4205 .next = ptype_seq_next, 4212 .next = ptype_seq_next,
4206 .stop = ptype_seq_stop, 4213 .stop = ptype_seq_stop,
4207 .show = ptype_seq_show, 4214 .show = ptype_seq_show,
4208 }; 4215 };
4209 4216
4210 static int ptype_seq_open(struct inode *inode, struct file *file) 4217 static int ptype_seq_open(struct inode *inode, struct file *file)
4211 { 4218 {
4212 return seq_open_net(inode, file, &ptype_seq_ops, 4219 return seq_open_net(inode, file, &ptype_seq_ops,
4213 sizeof(struct seq_net_private)); 4220 sizeof(struct seq_net_private));
4214 } 4221 }
4215 4222
4216 static const struct file_operations ptype_seq_fops = { 4223 static const struct file_operations ptype_seq_fops = {
4217 .owner = THIS_MODULE, 4224 .owner = THIS_MODULE,
4218 .open = ptype_seq_open, 4225 .open = ptype_seq_open,
4219 .read = seq_read, 4226 .read = seq_read,
4220 .llseek = seq_lseek, 4227 .llseek = seq_lseek,
4221 .release = seq_release_net, 4228 .release = seq_release_net,
4222 }; 4229 };
4223 4230
4224 4231
4225 static int __net_init dev_proc_net_init(struct net *net) 4232 static int __net_init dev_proc_net_init(struct net *net)
4226 { 4233 {
4227 int rc = -ENOMEM; 4234 int rc = -ENOMEM;
4228 4235
4229 if (!proc_net_fops_create(net, "dev", S_IRUGO, &dev_seq_fops)) 4236 if (!proc_net_fops_create(net, "dev", S_IRUGO, &dev_seq_fops))
4230 goto out; 4237 goto out;
4231 if (!proc_net_fops_create(net, "softnet_stat", S_IRUGO, &softnet_seq_fops)) 4238 if (!proc_net_fops_create(net, "softnet_stat", S_IRUGO, &softnet_seq_fops))
4232 goto out_dev; 4239 goto out_dev;
4233 if (!proc_net_fops_create(net, "ptype", S_IRUGO, &ptype_seq_fops)) 4240 if (!proc_net_fops_create(net, "ptype", S_IRUGO, &ptype_seq_fops))
4234 goto out_softnet; 4241 goto out_softnet;
4235 4242
4236 if (wext_proc_init(net)) 4243 if (wext_proc_init(net))
4237 goto out_ptype; 4244 goto out_ptype;
4238 rc = 0; 4245 rc = 0;
4239 out: 4246 out:
4240 return rc; 4247 return rc;
4241 out_ptype: 4248 out_ptype:
4242 proc_net_remove(net, "ptype"); 4249 proc_net_remove(net, "ptype");
4243 out_softnet: 4250 out_softnet:
4244 proc_net_remove(net, "softnet_stat"); 4251 proc_net_remove(net, "softnet_stat");
4245 out_dev: 4252 out_dev:
4246 proc_net_remove(net, "dev"); 4253 proc_net_remove(net, "dev");
4247 goto out; 4254 goto out;
4248 } 4255 }
4249 4256
4250 static void __net_exit dev_proc_net_exit(struct net *net) 4257 static void __net_exit dev_proc_net_exit(struct net *net)
4251 { 4258 {
4252 wext_proc_exit(net); 4259 wext_proc_exit(net);
4253 4260
4254 proc_net_remove(net, "ptype"); 4261 proc_net_remove(net, "ptype");
4255 proc_net_remove(net, "softnet_stat"); 4262 proc_net_remove(net, "softnet_stat");
4256 proc_net_remove(net, "dev"); 4263 proc_net_remove(net, "dev");
4257 } 4264 }
4258 4265
4259 static struct pernet_operations __net_initdata dev_proc_ops = { 4266 static struct pernet_operations __net_initdata dev_proc_ops = {
4260 .init = dev_proc_net_init, 4267 .init = dev_proc_net_init,
4261 .exit = dev_proc_net_exit, 4268 .exit = dev_proc_net_exit,
4262 }; 4269 };
4263 4270
4264 static int __init dev_proc_init(void) 4271 static int __init dev_proc_init(void)
4265 { 4272 {
4266 return register_pernet_subsys(&dev_proc_ops); 4273 return register_pernet_subsys(&dev_proc_ops);
4267 } 4274 }
4268 #else 4275 #else
4269 #define dev_proc_init() 0 4276 #define dev_proc_init() 0
4270 #endif /* CONFIG_PROC_FS */ 4277 #endif /* CONFIG_PROC_FS */
4271 4278
4272 4279
4273 /** 4280 /**
4274 * netdev_set_master - set up master pointer 4281 * netdev_set_master - set up master pointer
4275 * @slave: slave device 4282 * @slave: slave device
4276 * @master: new master device 4283 * @master: new master device
4277 * 4284 *
4278 * Changes the master device of the slave. Pass %NULL to break the 4285 * Changes the master device of the slave. Pass %NULL to break the
4279 * bonding. The caller must hold the RTNL semaphore. On a failure 4286 * bonding. The caller must hold the RTNL semaphore. On a failure
4280 * a negative errno code is returned. On success the reference counts 4287 * a negative errno code is returned. On success the reference counts
4281 * are adjusted and the function returns zero. 4288 * are adjusted and the function returns zero.
4282 */ 4289 */
4283 int netdev_set_master(struct net_device *slave, struct net_device *master) 4290 int netdev_set_master(struct net_device *slave, struct net_device *master)
4284 { 4291 {
4285 struct net_device *old = slave->master; 4292 struct net_device *old = slave->master;
4286 4293
4287 ASSERT_RTNL(); 4294 ASSERT_RTNL();
4288 4295
4289 if (master) { 4296 if (master) {
4290 if (old) 4297 if (old)
4291 return -EBUSY; 4298 return -EBUSY;
4292 dev_hold(master); 4299 dev_hold(master);
4293 } 4300 }
4294 4301
4295 slave->master = master; 4302 slave->master = master;
4296 4303
4297 if (old) 4304 if (old)
4298 dev_put(old); 4305 dev_put(old);
4299 return 0; 4306 return 0;
4300 } 4307 }
4301 EXPORT_SYMBOL(netdev_set_master); 4308 EXPORT_SYMBOL(netdev_set_master);
4302 4309
4303 /** 4310 /**
4304 * netdev_set_bond_master - set up bonding master/slave pair 4311 * netdev_set_bond_master - set up bonding master/slave pair
4305 * @slave: slave device 4312 * @slave: slave device
4306 * @master: new master device 4313 * @master: new master device
4307 * 4314 *
4308 * Changes the master device of the slave. Pass %NULL to break the 4315 * Changes the master device of the slave. Pass %NULL to break the
4309 * bonding. The caller must hold the RTNL semaphore. On a failure 4316 * bonding. The caller must hold the RTNL semaphore. On a failure
4310 * a negative errno code is returned. On success %RTM_NEWLINK is sent 4317 * a negative errno code is returned. On success %RTM_NEWLINK is sent
4311 * to the routing socket and the function returns zero. 4318 * to the routing socket and the function returns zero.
4312 */ 4319 */
4313 int netdev_set_bond_master(struct net_device *slave, struct net_device *master) 4320 int netdev_set_bond_master(struct net_device *slave, struct net_device *master)
4314 { 4321 {
4315 int err; 4322 int err;
4316 4323
4317 ASSERT_RTNL(); 4324 ASSERT_RTNL();
4318 4325
4319 err = netdev_set_master(slave, master); 4326 err = netdev_set_master(slave, master);
4320 if (err) 4327 if (err)
4321 return err; 4328 return err;
4322 if (master) 4329 if (master)
4323 slave->flags |= IFF_SLAVE; 4330 slave->flags |= IFF_SLAVE;
4324 else 4331 else
4325 slave->flags &= ~IFF_SLAVE; 4332 slave->flags &= ~IFF_SLAVE;
4326 4333
4327 rtmsg_ifinfo(RTM_NEWLINK, slave, IFF_SLAVE); 4334 rtmsg_ifinfo(RTM_NEWLINK, slave, IFF_SLAVE);
4328 return 0; 4335 return 0;
4329 } 4336 }
4330 EXPORT_SYMBOL(netdev_set_bond_master); 4337 EXPORT_SYMBOL(netdev_set_bond_master);
4331 4338
4332 static void dev_change_rx_flags(struct net_device *dev, int flags) 4339 static void dev_change_rx_flags(struct net_device *dev, int flags)
4333 { 4340 {
4334 const struct net_device_ops *ops = dev->netdev_ops; 4341 const struct net_device_ops *ops = dev->netdev_ops;
4335 4342
4336 if ((dev->flags & IFF_UP) && ops->ndo_change_rx_flags) 4343 if ((dev->flags & IFF_UP) && ops->ndo_change_rx_flags)
4337 ops->ndo_change_rx_flags(dev, flags); 4344 ops->ndo_change_rx_flags(dev, flags);
4338 } 4345 }
4339 4346
4340 static int __dev_set_promiscuity(struct net_device *dev, int inc) 4347 static int __dev_set_promiscuity(struct net_device *dev, int inc)
4341 { 4348 {
4342 unsigned short old_flags = dev->flags; 4349 unsigned short old_flags = dev->flags;
4343 uid_t uid; 4350 uid_t uid;
4344 gid_t gid; 4351 gid_t gid;
4345 4352
4346 ASSERT_RTNL(); 4353 ASSERT_RTNL();
4347 4354
4348 dev->flags |= IFF_PROMISC; 4355 dev->flags |= IFF_PROMISC;
4349 dev->promiscuity += inc; 4356 dev->promiscuity += inc;
4350 if (dev->promiscuity == 0) { 4357 if (dev->promiscuity == 0) {
4351 /* 4358 /*
4352 * Avoid overflow. 4359 * Avoid overflow.
4353 * If inc causes overflow, untouch promisc and return error. 4360 * If inc causes overflow, untouch promisc and return error.
4354 */ 4361 */
4355 if (inc < 0) 4362 if (inc < 0)
4356 dev->flags &= ~IFF_PROMISC; 4363 dev->flags &= ~IFF_PROMISC;
4357 else { 4364 else {
4358 dev->promiscuity -= inc; 4365 dev->promiscuity -= inc;
4359 printk(KERN_WARNING "%s: promiscuity touches roof, " 4366 printk(KERN_WARNING "%s: promiscuity touches roof, "
4360 "set promiscuity failed, promiscuity feature " 4367 "set promiscuity failed, promiscuity feature "
4361 "of device might be broken.\n", dev->name); 4368 "of device might be broken.\n", dev->name);
4362 return -EOVERFLOW; 4369 return -EOVERFLOW;
4363 } 4370 }
4364 } 4371 }
4365 if (dev->flags != old_flags) { 4372 if (dev->flags != old_flags) {
4366 printk(KERN_INFO "device %s %s promiscuous mode\n", 4373 printk(KERN_INFO "device %s %s promiscuous mode\n",
4367 dev->name, (dev->flags & IFF_PROMISC) ? "entered" : 4374 dev->name, (dev->flags & IFF_PROMISC) ? "entered" :
4368 "left"); 4375 "left");
4369 if (audit_enabled) { 4376 if (audit_enabled) {
4370 current_uid_gid(&uid, &gid); 4377 current_uid_gid(&uid, &gid);
4371 audit_log(current->audit_context, GFP_ATOMIC, 4378 audit_log(current->audit_context, GFP_ATOMIC,
4372 AUDIT_ANOM_PROMISCUOUS, 4379 AUDIT_ANOM_PROMISCUOUS,
4373 "dev=%s prom=%d old_prom=%d auid=%u uid=%u gid=%u ses=%u", 4380 "dev=%s prom=%d old_prom=%d auid=%u uid=%u gid=%u ses=%u",
4374 dev->name, (dev->flags & IFF_PROMISC), 4381 dev->name, (dev->flags & IFF_PROMISC),
4375 (old_flags & IFF_PROMISC), 4382 (old_flags & IFF_PROMISC),
4376 audit_get_loginuid(current), 4383 audit_get_loginuid(current),
4377 uid, gid, 4384 uid, gid,
4378 audit_get_sessionid(current)); 4385 audit_get_sessionid(current));
4379 } 4386 }
4380 4387
4381 dev_change_rx_flags(dev, IFF_PROMISC); 4388 dev_change_rx_flags(dev, IFF_PROMISC);
4382 } 4389 }
4383 return 0; 4390 return 0;
4384 } 4391 }
4385 4392
4386 /** 4393 /**
4387 * dev_set_promiscuity - update promiscuity count on a device 4394 * dev_set_promiscuity - update promiscuity count on a device
4388 * @dev: device 4395 * @dev: device
4389 * @inc: modifier 4396 * @inc: modifier
4390 * 4397 *
4391 * Add or remove promiscuity from a device. While the count in the device 4398 * Add or remove promiscuity from a device. While the count in the device
4392 * remains above zero the interface remains promiscuous. Once it hits zero 4399 * remains above zero the interface remains promiscuous. Once it hits zero
4393 * the device reverts back to normal filtering operation. A negative inc 4400 * the device reverts back to normal filtering operation. A negative inc
4394 * value is used to drop promiscuity on the device. 4401 * value is used to drop promiscuity on the device.
4395 * Return 0 if successful or a negative errno code on error. 4402 * Return 0 if successful or a negative errno code on error.
4396 */ 4403 */
4397 int dev_set_promiscuity(struct net_device *dev, int inc) 4404 int dev_set_promiscuity(struct net_device *dev, int inc)
4398 { 4405 {
4399 unsigned short old_flags = dev->flags; 4406 unsigned short old_flags = dev->flags;
4400 int err; 4407 int err;
4401 4408
4402 err = __dev_set_promiscuity(dev, inc); 4409 err = __dev_set_promiscuity(dev, inc);
4403 if (err < 0) 4410 if (err < 0)
4404 return err; 4411 return err;
4405 if (dev->flags != old_flags) 4412 if (dev->flags != old_flags)
4406 dev_set_rx_mode(dev); 4413 dev_set_rx_mode(dev);
4407 return err; 4414 return err;
4408 } 4415 }
4409 EXPORT_SYMBOL(dev_set_promiscuity); 4416 EXPORT_SYMBOL(dev_set_promiscuity);
4410 4417
4411 /** 4418 /**
4412 * dev_set_allmulti - update allmulti count on a device 4419 * dev_set_allmulti - update allmulti count on a device
4413 * @dev: device 4420 * @dev: device
4414 * @inc: modifier 4421 * @inc: modifier
4415 * 4422 *
4416 * Add or remove reception of all multicast frames to a device. While the 4423 * Add or remove reception of all multicast frames to a device. While the
4417 * count in the device remains above zero the interface remains listening 4424 * count in the device remains above zero the interface remains listening
4418 * to all interfaces. Once it hits zero the device reverts back to normal 4425 * to all interfaces. Once it hits zero the device reverts back to normal
4419 * filtering operation. A negative @inc value is used to drop the counter 4426 * filtering operation. A negative @inc value is used to drop the counter
4420 * when releasing a resource needing all multicasts. 4427 * when releasing a resource needing all multicasts.
4421 * Return 0 if successful or a negative errno code on error. 4428 * Return 0 if successful or a negative errno code on error.
4422 */ 4429 */
4423 4430
4424 int dev_set_allmulti(struct net_device *dev, int inc) 4431 int dev_set_allmulti(struct net_device *dev, int inc)
4425 { 4432 {
4426 unsigned short old_flags = dev->flags; 4433 unsigned short old_flags = dev->flags;
4427 4434
4428 ASSERT_RTNL(); 4435 ASSERT_RTNL();
4429 4436
4430 dev->flags |= IFF_ALLMULTI; 4437 dev->flags |= IFF_ALLMULTI;
4431 dev->allmulti += inc; 4438 dev->allmulti += inc;
4432 if (dev->allmulti == 0) { 4439 if (dev->allmulti == 0) {
4433 /* 4440 /*
4434 * Avoid overflow. 4441 * Avoid overflow.
4435 * If inc causes overflow, untouch allmulti and return error. 4442 * If inc causes overflow, untouch allmulti and return error.
4436 */ 4443 */
4437 if (inc < 0) 4444 if (inc < 0)
4438 dev->flags &= ~IFF_ALLMULTI; 4445 dev->flags &= ~IFF_ALLMULTI;
4439 else { 4446 else {
4440 dev->allmulti -= inc; 4447 dev->allmulti -= inc;
4441 printk(KERN_WARNING "%s: allmulti touches roof, " 4448 printk(KERN_WARNING "%s: allmulti touches roof, "
4442 "set allmulti failed, allmulti feature of " 4449 "set allmulti failed, allmulti feature of "
4443 "device might be broken.\n", dev->name); 4450 "device might be broken.\n", dev->name);
4444 return -EOVERFLOW; 4451 return -EOVERFLOW;
4445 } 4452 }
4446 } 4453 }
4447 if (dev->flags ^ old_flags) { 4454 if (dev->flags ^ old_flags) {
4448 dev_change_rx_flags(dev, IFF_ALLMULTI); 4455 dev_change_rx_flags(dev, IFF_ALLMULTI);
4449 dev_set_rx_mode(dev); 4456 dev_set_rx_mode(dev);
4450 } 4457 }
4451 return 0; 4458 return 0;
4452 } 4459 }
4453 EXPORT_SYMBOL(dev_set_allmulti); 4460 EXPORT_SYMBOL(dev_set_allmulti);
4454 4461
4455 /* 4462 /*
4456 * Upload unicast and multicast address lists to device and 4463 * Upload unicast and multicast address lists to device and
4457 * configure RX filtering. When the device doesn't support unicast 4464 * configure RX filtering. When the device doesn't support unicast
4458 * filtering it is put in promiscuous mode while unicast addresses 4465 * filtering it is put in promiscuous mode while unicast addresses
4459 * are present. 4466 * are present.
4460 */ 4467 */
4461 void __dev_set_rx_mode(struct net_device *dev) 4468 void __dev_set_rx_mode(struct net_device *dev)
4462 { 4469 {
4463 const struct net_device_ops *ops = dev->netdev_ops; 4470 const struct net_device_ops *ops = dev->netdev_ops;
4464 4471
4465 /* dev_open will call this function so the list will stay sane. */ 4472 /* dev_open will call this function so the list will stay sane. */
4466 if (!(dev->flags&IFF_UP)) 4473 if (!(dev->flags&IFF_UP))
4467 return; 4474 return;
4468 4475
4469 if (!netif_device_present(dev)) 4476 if (!netif_device_present(dev))
4470 return; 4477 return;
4471 4478
4472 if (ops->ndo_set_rx_mode) 4479 if (ops->ndo_set_rx_mode)
4473 ops->ndo_set_rx_mode(dev); 4480 ops->ndo_set_rx_mode(dev);
4474 else { 4481 else {
4475 /* Unicast addresses changes may only happen under the rtnl, 4482 /* Unicast addresses changes may only happen under the rtnl,
4476 * therefore calling __dev_set_promiscuity here is safe. 4483 * therefore calling __dev_set_promiscuity here is safe.
4477 */ 4484 */
4478 if (!netdev_uc_empty(dev) && !dev->uc_promisc) { 4485 if (!netdev_uc_empty(dev) && !dev->uc_promisc) {
4479 __dev_set_promiscuity(dev, 1); 4486 __dev_set_promiscuity(dev, 1);
4480 dev->uc_promisc = 1; 4487 dev->uc_promisc = 1;
4481 } else if (netdev_uc_empty(dev) && dev->uc_promisc) { 4488 } else if (netdev_uc_empty(dev) && dev->uc_promisc) {
4482 __dev_set_promiscuity(dev, -1); 4489 __dev_set_promiscuity(dev, -1);
4483 dev->uc_promisc = 0; 4490 dev->uc_promisc = 0;
4484 } 4491 }
4485 4492
4486 if (ops->ndo_set_multicast_list) 4493 if (ops->ndo_set_multicast_list)
4487 ops->ndo_set_multicast_list(dev); 4494 ops->ndo_set_multicast_list(dev);
4488 } 4495 }
4489 } 4496 }
4490 4497
4491 void dev_set_rx_mode(struct net_device *dev) 4498 void dev_set_rx_mode(struct net_device *dev)
4492 { 4499 {
4493 netif_addr_lock_bh(dev); 4500 netif_addr_lock_bh(dev);
4494 __dev_set_rx_mode(dev); 4501 __dev_set_rx_mode(dev);
4495 netif_addr_unlock_bh(dev); 4502 netif_addr_unlock_bh(dev);
4496 } 4503 }
4497 4504
4498 /** 4505 /**
4499 * dev_ethtool_get_settings - call device's ethtool_ops::get_settings() 4506 * dev_ethtool_get_settings - call device's ethtool_ops::get_settings()
4500 * @dev: device 4507 * @dev: device
4501 * @cmd: memory area for ethtool_ops::get_settings() result 4508 * @cmd: memory area for ethtool_ops::get_settings() result
4502 * 4509 *
4503 * The cmd arg is initialized properly (cleared and 4510 * The cmd arg is initialized properly (cleared and
4504 * ethtool_cmd::cmd field set to ETHTOOL_GSET). 4511 * ethtool_cmd::cmd field set to ETHTOOL_GSET).
4505 * 4512 *
4506 * Return device's ethtool_ops::get_settings() result value or 4513 * Return device's ethtool_ops::get_settings() result value or
4507 * -EOPNOTSUPP when device doesn't expose 4514 * -EOPNOTSUPP when device doesn't expose
4508 * ethtool_ops::get_settings() operation. 4515 * ethtool_ops::get_settings() operation.
4509 */ 4516 */
4510 int dev_ethtool_get_settings(struct net_device *dev, 4517 int dev_ethtool_get_settings(struct net_device *dev,
4511 struct ethtool_cmd *cmd) 4518 struct ethtool_cmd *cmd)
4512 { 4519 {
4513 if (!dev->ethtool_ops || !dev->ethtool_ops->get_settings) 4520 if (!dev->ethtool_ops || !dev->ethtool_ops->get_settings)
4514 return -EOPNOTSUPP; 4521 return -EOPNOTSUPP;
4515 4522
4516 memset(cmd, 0, sizeof(struct ethtool_cmd)); 4523 memset(cmd, 0, sizeof(struct ethtool_cmd));
4517 cmd->cmd = ETHTOOL_GSET; 4524 cmd->cmd = ETHTOOL_GSET;
4518 return dev->ethtool_ops->get_settings(dev, cmd); 4525 return dev->ethtool_ops->get_settings(dev, cmd);
4519 } 4526 }
4520 EXPORT_SYMBOL(dev_ethtool_get_settings); 4527 EXPORT_SYMBOL(dev_ethtool_get_settings);
4521 4528
4522 /** 4529 /**
4523 * dev_get_flags - get flags reported to userspace 4530 * dev_get_flags - get flags reported to userspace
4524 * @dev: device 4531 * @dev: device
4525 * 4532 *
4526 * Get the combination of flag bits exported through APIs to userspace. 4533 * Get the combination of flag bits exported through APIs to userspace.
4527 */ 4534 */
4528 unsigned dev_get_flags(const struct net_device *dev) 4535 unsigned dev_get_flags(const struct net_device *dev)
4529 { 4536 {
4530 unsigned flags; 4537 unsigned flags;
4531 4538
4532 flags = (dev->flags & ~(IFF_PROMISC | 4539 flags = (dev->flags & ~(IFF_PROMISC |
4533 IFF_ALLMULTI | 4540 IFF_ALLMULTI |
4534 IFF_RUNNING | 4541 IFF_RUNNING |
4535 IFF_LOWER_UP | 4542 IFF_LOWER_UP |
4536 IFF_DORMANT)) | 4543 IFF_DORMANT)) |
4537 (dev->gflags & (IFF_PROMISC | 4544 (dev->gflags & (IFF_PROMISC |
4538 IFF_ALLMULTI)); 4545 IFF_ALLMULTI));
4539 4546
4540 if (netif_running(dev)) { 4547 if (netif_running(dev)) {
4541 if (netif_oper_up(dev)) 4548 if (netif_oper_up(dev))
4542 flags |= IFF_RUNNING; 4549 flags |= IFF_RUNNING;
4543 if (netif_carrier_ok(dev)) 4550 if (netif_carrier_ok(dev))
4544 flags |= IFF_LOWER_UP; 4551 flags |= IFF_LOWER_UP;
4545 if (netif_dormant(dev)) 4552 if (netif_dormant(dev))
4546 flags |= IFF_DORMANT; 4553 flags |= IFF_DORMANT;
4547 } 4554 }
4548 4555
4549 return flags; 4556 return flags;
4550 } 4557 }
4551 EXPORT_SYMBOL(dev_get_flags); 4558 EXPORT_SYMBOL(dev_get_flags);
4552 4559
4553 int __dev_change_flags(struct net_device *dev, unsigned int flags) 4560 int __dev_change_flags(struct net_device *dev, unsigned int flags)
4554 { 4561 {
4555 int old_flags = dev->flags; 4562 int old_flags = dev->flags;
4556 int ret; 4563 int ret;
4557 4564
4558 ASSERT_RTNL(); 4565 ASSERT_RTNL();
4559 4566
4560 /* 4567 /*
4561 * Set the flags on our device. 4568 * Set the flags on our device.
4562 */ 4569 */
4563 4570
4564 dev->flags = (flags & (IFF_DEBUG | IFF_NOTRAILERS | IFF_NOARP | 4571 dev->flags = (flags & (IFF_DEBUG | IFF_NOTRAILERS | IFF_NOARP |
4565 IFF_DYNAMIC | IFF_MULTICAST | IFF_PORTSEL | 4572 IFF_DYNAMIC | IFF_MULTICAST | IFF_PORTSEL |
4566 IFF_AUTOMEDIA)) | 4573 IFF_AUTOMEDIA)) |
4567 (dev->flags & (IFF_UP | IFF_VOLATILE | IFF_PROMISC | 4574 (dev->flags & (IFF_UP | IFF_VOLATILE | IFF_PROMISC |
4568 IFF_ALLMULTI)); 4575 IFF_ALLMULTI));
4569 4576
4570 /* 4577 /*
4571 * Load in the correct multicast list now the flags have changed. 4578 * Load in the correct multicast list now the flags have changed.
4572 */ 4579 */
4573 4580
4574 if ((old_flags ^ flags) & IFF_MULTICAST) 4581 if ((old_flags ^ flags) & IFF_MULTICAST)
4575 dev_change_rx_flags(dev, IFF_MULTICAST); 4582 dev_change_rx_flags(dev, IFF_MULTICAST);
4576 4583
4577 dev_set_rx_mode(dev); 4584 dev_set_rx_mode(dev);
4578 4585
4579 /* 4586 /*
4580 * Have we downed the interface. We handle IFF_UP ourselves 4587 * Have we downed the interface. We handle IFF_UP ourselves
4581 * according to user attempts to set it, rather than blindly 4588 * according to user attempts to set it, rather than blindly
4582 * setting it. 4589 * setting it.
4583 */ 4590 */
4584 4591
4585 ret = 0; 4592 ret = 0;
4586 if ((old_flags ^ flags) & IFF_UP) { /* Bit is different ? */ 4593 if ((old_flags ^ flags) & IFF_UP) { /* Bit is different ? */
4587 ret = ((old_flags & IFF_UP) ? __dev_close : __dev_open)(dev); 4594 ret = ((old_flags & IFF_UP) ? __dev_close : __dev_open)(dev);
4588 4595
4589 if (!ret) 4596 if (!ret)
4590 dev_set_rx_mode(dev); 4597 dev_set_rx_mode(dev);
4591 } 4598 }
4592 4599
4593 if ((flags ^ dev->gflags) & IFF_PROMISC) { 4600 if ((flags ^ dev->gflags) & IFF_PROMISC) {
4594 int inc = (flags & IFF_PROMISC) ? 1 : -1; 4601 int inc = (flags & IFF_PROMISC) ? 1 : -1;
4595 4602
4596 dev->gflags ^= IFF_PROMISC; 4603 dev->gflags ^= IFF_PROMISC;
4597 dev_set_promiscuity(dev, inc); 4604 dev_set_promiscuity(dev, inc);
4598 } 4605 }
4599 4606
4600 /* NOTE: order of synchronization of IFF_PROMISC and IFF_ALLMULTI 4607 /* NOTE: order of synchronization of IFF_PROMISC and IFF_ALLMULTI
4601 is important. Some (broken) drivers set IFF_PROMISC, when 4608 is important. Some (broken) drivers set IFF_PROMISC, when
4602 IFF_ALLMULTI is requested not asking us and not reporting. 4609 IFF_ALLMULTI is requested not asking us and not reporting.
4603 */ 4610 */
4604 if ((flags ^ dev->gflags) & IFF_ALLMULTI) { 4611 if ((flags ^ dev->gflags) & IFF_ALLMULTI) {
4605 int inc = (flags & IFF_ALLMULTI) ? 1 : -1; 4612 int inc = (flags & IFF_ALLMULTI) ? 1 : -1;
4606 4613
4607 dev->gflags ^= IFF_ALLMULTI; 4614 dev->gflags ^= IFF_ALLMULTI;
4608 dev_set_allmulti(dev, inc); 4615 dev_set_allmulti(dev, inc);
4609 } 4616 }
4610 4617
4611 return ret; 4618 return ret;
4612 } 4619 }
4613 4620
4614 void __dev_notify_flags(struct net_device *dev, unsigned int old_flags) 4621 void __dev_notify_flags(struct net_device *dev, unsigned int old_flags)
4615 { 4622 {
4616 unsigned int changes = dev->flags ^ old_flags; 4623 unsigned int changes = dev->flags ^ old_flags;
4617 4624
4618 if (changes & IFF_UP) { 4625 if (changes & IFF_UP) {
4619 if (dev->flags & IFF_UP) 4626 if (dev->flags & IFF_UP)
4620 call_netdevice_notifiers(NETDEV_UP, dev); 4627 call_netdevice_notifiers(NETDEV_UP, dev);
4621 else 4628 else
4622 call_netdevice_notifiers(NETDEV_DOWN, dev); 4629 call_netdevice_notifiers(NETDEV_DOWN, dev);
4623 } 4630 }
4624 4631
4625 if (dev->flags & IFF_UP && 4632 if (dev->flags & IFF_UP &&
4626 (changes & ~(IFF_UP | IFF_PROMISC | IFF_ALLMULTI | IFF_VOLATILE))) 4633 (changes & ~(IFF_UP | IFF_PROMISC | IFF_ALLMULTI | IFF_VOLATILE)))
4627 call_netdevice_notifiers(NETDEV_CHANGE, dev); 4634 call_netdevice_notifiers(NETDEV_CHANGE, dev);
4628 } 4635 }
4629 4636
4630 /** 4637 /**
4631 * dev_change_flags - change device settings 4638 * dev_change_flags - change device settings
4632 * @dev: device 4639 * @dev: device
4633 * @flags: device state flags 4640 * @flags: device state flags
4634 * 4641 *
4635 * Change settings on device based state flags. The flags are 4642 * Change settings on device based state flags. The flags are
4636 * in the userspace exported format. 4643 * in the userspace exported format.
4637 */ 4644 */
4638 int dev_change_flags(struct net_device *dev, unsigned flags) 4645 int dev_change_flags(struct net_device *dev, unsigned flags)
4639 { 4646 {
4640 int ret, changes; 4647 int ret, changes;
4641 int old_flags = dev->flags; 4648 int old_flags = dev->flags;
4642 4649
4643 ret = __dev_change_flags(dev, flags); 4650 ret = __dev_change_flags(dev, flags);
4644 if (ret < 0) 4651 if (ret < 0)
4645 return ret; 4652 return ret;
4646 4653
4647 changes = old_flags ^ dev->flags; 4654 changes = old_flags ^ dev->flags;
4648 if (changes) 4655 if (changes)
4649 rtmsg_ifinfo(RTM_NEWLINK, dev, changes); 4656 rtmsg_ifinfo(RTM_NEWLINK, dev, changes);
4650 4657
4651 __dev_notify_flags(dev, old_flags); 4658 __dev_notify_flags(dev, old_flags);
4652 return ret; 4659 return ret;
4653 } 4660 }
4654 EXPORT_SYMBOL(dev_change_flags); 4661 EXPORT_SYMBOL(dev_change_flags);
4655 4662
4656 /** 4663 /**
4657 * dev_set_mtu - Change maximum transfer unit 4664 * dev_set_mtu - Change maximum transfer unit
4658 * @dev: device 4665 * @dev: device
4659 * @new_mtu: new transfer unit 4666 * @new_mtu: new transfer unit
4660 * 4667 *
4661 * Change the maximum transfer size of the network device. 4668 * Change the maximum transfer size of the network device.
4662 */ 4669 */
4663 int dev_set_mtu(struct net_device *dev, int new_mtu) 4670 int dev_set_mtu(struct net_device *dev, int new_mtu)
4664 { 4671 {
4665 const struct net_device_ops *ops = dev->netdev_ops; 4672 const struct net_device_ops *ops = dev->netdev_ops;
4666 int err; 4673 int err;
4667 4674
4668 if (new_mtu == dev->mtu) 4675 if (new_mtu == dev->mtu)
4669 return 0; 4676 return 0;
4670 4677
4671 /* MTU must be positive. */ 4678 /* MTU must be positive. */
4672 if (new_mtu < 0) 4679 if (new_mtu < 0)
4673 return -EINVAL; 4680 return -EINVAL;
4674 4681
4675 if (!netif_device_present(dev)) 4682 if (!netif_device_present(dev))
4676 return -ENODEV; 4683 return -ENODEV;
4677 4684
4678 err = 0; 4685 err = 0;
4679 if (ops->ndo_change_mtu) 4686 if (ops->ndo_change_mtu)
4680 err = ops->ndo_change_mtu(dev, new_mtu); 4687 err = ops->ndo_change_mtu(dev, new_mtu);
4681 else 4688 else
4682 dev->mtu = new_mtu; 4689 dev->mtu = new_mtu;
4683 4690
4684 if (!err && dev->flags & IFF_UP) 4691 if (!err && dev->flags & IFF_UP)
4685 call_netdevice_notifiers(NETDEV_CHANGEMTU, dev); 4692 call_netdevice_notifiers(NETDEV_CHANGEMTU, dev);
4686 return err; 4693 return err;
4687 } 4694 }
4688 EXPORT_SYMBOL(dev_set_mtu); 4695 EXPORT_SYMBOL(dev_set_mtu);
4689 4696
4690 /** 4697 /**
4691 * dev_set_group - Change group this device belongs to 4698 * dev_set_group - Change group this device belongs to
4692 * @dev: device 4699 * @dev: device
4693 * @new_group: group this device should belong to 4700 * @new_group: group this device should belong to
4694 */ 4701 */
4695 void dev_set_group(struct net_device *dev, int new_group) 4702 void dev_set_group(struct net_device *dev, int new_group)
4696 { 4703 {
4697 dev->group = new_group; 4704 dev->group = new_group;
4698 } 4705 }
4699 EXPORT_SYMBOL(dev_set_group); 4706 EXPORT_SYMBOL(dev_set_group);
4700 4707
4701 /** 4708 /**
4702 * dev_set_mac_address - Change Media Access Control Address 4709 * dev_set_mac_address - Change Media Access Control Address
4703 * @dev: device 4710 * @dev: device
4704 * @sa: new address 4711 * @sa: new address
4705 * 4712 *
4706 * Change the hardware (MAC) address of the device 4713 * Change the hardware (MAC) address of the device
4707 */ 4714 */
4708 int dev_set_mac_address(struct net_device *dev, struct sockaddr *sa) 4715 int dev_set_mac_address(struct net_device *dev, struct sockaddr *sa)
4709 { 4716 {
4710 const struct net_device_ops *ops = dev->netdev_ops; 4717 const struct net_device_ops *ops = dev->netdev_ops;
4711 int err; 4718 int err;
4712 4719
4713 if (!ops->ndo_set_mac_address) 4720 if (!ops->ndo_set_mac_address)
4714 return -EOPNOTSUPP; 4721 return -EOPNOTSUPP;
4715 if (sa->sa_family != dev->type) 4722 if (sa->sa_family != dev->type)
4716 return -EINVAL; 4723 return -EINVAL;
4717 if (!netif_device_present(dev)) 4724 if (!netif_device_present(dev))
4718 return -ENODEV; 4725 return -ENODEV;
4719 err = ops->ndo_set_mac_address(dev, sa); 4726 err = ops->ndo_set_mac_address(dev, sa);
4720 if (!err) 4727 if (!err)
4721 call_netdevice_notifiers(NETDEV_CHANGEADDR, dev); 4728 call_netdevice_notifiers(NETDEV_CHANGEADDR, dev);
4722 return err; 4729 return err;
4723 } 4730 }
4724 EXPORT_SYMBOL(dev_set_mac_address); 4731 EXPORT_SYMBOL(dev_set_mac_address);
4725 4732
4726 /* 4733 /*
4727 * Perform the SIOCxIFxxx calls, inside rcu_read_lock() 4734 * Perform the SIOCxIFxxx calls, inside rcu_read_lock()
4728 */ 4735 */
4729 static int dev_ifsioc_locked(struct net *net, struct ifreq *ifr, unsigned int cmd) 4736 static int dev_ifsioc_locked(struct net *net, struct ifreq *ifr, unsigned int cmd)
4730 { 4737 {
4731 int err; 4738 int err;
4732 struct net_device *dev = dev_get_by_name_rcu(net, ifr->ifr_name); 4739 struct net_device *dev = dev_get_by_name_rcu(net, ifr->ifr_name);
4733 4740
4734 if (!dev) 4741 if (!dev)
4735 return -ENODEV; 4742 return -ENODEV;
4736 4743
4737 switch (cmd) { 4744 switch (cmd) {
4738 case SIOCGIFFLAGS: /* Get interface flags */ 4745 case SIOCGIFFLAGS: /* Get interface flags */
4739 ifr->ifr_flags = (short) dev_get_flags(dev); 4746 ifr->ifr_flags = (short) dev_get_flags(dev);
4740 return 0; 4747 return 0;
4741 4748
4742 case SIOCGIFMETRIC: /* Get the metric on the interface 4749 case SIOCGIFMETRIC: /* Get the metric on the interface
4743 (currently unused) */ 4750 (currently unused) */
4744 ifr->ifr_metric = 0; 4751 ifr->ifr_metric = 0;
4745 return 0; 4752 return 0;
4746 4753
4747 case SIOCGIFMTU: /* Get the MTU of a device */ 4754 case SIOCGIFMTU: /* Get the MTU of a device */
4748 ifr->ifr_mtu = dev->mtu; 4755 ifr->ifr_mtu = dev->mtu;
4749 return 0; 4756 return 0;
4750 4757
4751 case SIOCGIFHWADDR: 4758 case SIOCGIFHWADDR:
4752 if (!dev->addr_len) 4759 if (!dev->addr_len)
4753 memset(ifr->ifr_hwaddr.sa_data, 0, sizeof ifr->ifr_hwaddr.sa_data); 4760 memset(ifr->ifr_hwaddr.sa_data, 0, sizeof ifr->ifr_hwaddr.sa_data);
4754 else 4761 else
4755 memcpy(ifr->ifr_hwaddr.sa_data, dev->dev_addr, 4762 memcpy(ifr->ifr_hwaddr.sa_data, dev->dev_addr,
4756 min(sizeof ifr->ifr_hwaddr.sa_data, (size_t) dev->addr_len)); 4763 min(sizeof ifr->ifr_hwaddr.sa_data, (size_t) dev->addr_len));
4757 ifr->ifr_hwaddr.sa_family = dev->type; 4764 ifr->ifr_hwaddr.sa_family = dev->type;
4758 return 0; 4765 return 0;
4759 4766
4760 case SIOCGIFSLAVE: 4767 case SIOCGIFSLAVE:
4761 err = -EINVAL; 4768 err = -EINVAL;
4762 break; 4769 break;
4763 4770
4764 case SIOCGIFMAP: 4771 case SIOCGIFMAP:
4765 ifr->ifr_map.mem_start = dev->mem_start; 4772 ifr->ifr_map.mem_start = dev->mem_start;
4766 ifr->ifr_map.mem_end = dev->mem_end; 4773 ifr->ifr_map.mem_end = dev->mem_end;
4767 ifr->ifr_map.base_addr = dev->base_addr; 4774 ifr->ifr_map.base_addr = dev->base_addr;
4768 ifr->ifr_map.irq = dev->irq; 4775 ifr->ifr_map.irq = dev->irq;
4769 ifr->ifr_map.dma = dev->dma; 4776 ifr->ifr_map.dma = dev->dma;
4770 ifr->ifr_map.port = dev->if_port; 4777 ifr->ifr_map.port = dev->if_port;
4771 return 0; 4778 return 0;
4772 4779
4773 case SIOCGIFINDEX: 4780 case SIOCGIFINDEX:
4774 ifr->ifr_ifindex = dev->ifindex; 4781 ifr->ifr_ifindex = dev->ifindex;
4775 return 0; 4782 return 0;
4776 4783
4777 case SIOCGIFTXQLEN: 4784 case SIOCGIFTXQLEN:
4778 ifr->ifr_qlen = dev->tx_queue_len; 4785 ifr->ifr_qlen = dev->tx_queue_len;
4779 return 0; 4786 return 0;
4780 4787
4781 default: 4788 default:
4782 /* dev_ioctl() should ensure this case 4789 /* dev_ioctl() should ensure this case
4783 * is never reached 4790 * is never reached
4784 */ 4791 */
4785 WARN_ON(1); 4792 WARN_ON(1);
4786 err = -ENOTTY; 4793 err = -ENOTTY;
4787 break; 4794 break;
4788 4795
4789 } 4796 }
4790 return err; 4797 return err;
4791 } 4798 }
4792 4799
4793 /* 4800 /*
4794 * Perform the SIOCxIFxxx calls, inside rtnl_lock() 4801 * Perform the SIOCxIFxxx calls, inside rtnl_lock()
4795 */ 4802 */
4796 static int dev_ifsioc(struct net *net, struct ifreq *ifr, unsigned int cmd) 4803 static int dev_ifsioc(struct net *net, struct ifreq *ifr, unsigned int cmd)
4797 { 4804 {
4798 int err; 4805 int err;
4799 struct net_device *dev = __dev_get_by_name(net, ifr->ifr_name); 4806 struct net_device *dev = __dev_get_by_name(net, ifr->ifr_name);
4800 const struct net_device_ops *ops; 4807 const struct net_device_ops *ops;
4801 4808
4802 if (!dev) 4809 if (!dev)
4803 return -ENODEV; 4810 return -ENODEV;
4804 4811
4805 ops = dev->netdev_ops; 4812 ops = dev->netdev_ops;
4806 4813
4807 switch (cmd) { 4814 switch (cmd) {
4808 case SIOCSIFFLAGS: /* Set interface flags */ 4815 case SIOCSIFFLAGS: /* Set interface flags */
4809 return dev_change_flags(dev, ifr->ifr_flags); 4816 return dev_change_flags(dev, ifr->ifr_flags);
4810 4817
4811 case SIOCSIFMETRIC: /* Set the metric on the interface 4818 case SIOCSIFMETRIC: /* Set the metric on the interface
4812 (currently unused) */ 4819 (currently unused) */
4813 return -EOPNOTSUPP; 4820 return -EOPNOTSUPP;
4814 4821
4815 case SIOCSIFMTU: /* Set the MTU of a device */ 4822 case SIOCSIFMTU: /* Set the MTU of a device */
4816 return dev_set_mtu(dev, ifr->ifr_mtu); 4823 return dev_set_mtu(dev, ifr->ifr_mtu);
4817 4824
4818 case SIOCSIFHWADDR: 4825 case SIOCSIFHWADDR:
4819 return dev_set_mac_address(dev, &ifr->ifr_hwaddr); 4826 return dev_set_mac_address(dev, &ifr->ifr_hwaddr);
4820 4827
4821 case SIOCSIFHWBROADCAST: 4828 case SIOCSIFHWBROADCAST:
4822 if (ifr->ifr_hwaddr.sa_family != dev->type) 4829 if (ifr->ifr_hwaddr.sa_family != dev->type)
4823 return -EINVAL; 4830 return -EINVAL;
4824 memcpy(dev->broadcast, ifr->ifr_hwaddr.sa_data, 4831 memcpy(dev->broadcast, ifr->ifr_hwaddr.sa_data,
4825 min(sizeof ifr->ifr_hwaddr.sa_data, (size_t) dev->addr_len)); 4832 min(sizeof ifr->ifr_hwaddr.sa_data, (size_t) dev->addr_len));
4826 call_netdevice_notifiers(NETDEV_CHANGEADDR, dev); 4833 call_netdevice_notifiers(NETDEV_CHANGEADDR, dev);
4827 return 0; 4834 return 0;
4828 4835
4829 case SIOCSIFMAP: 4836 case SIOCSIFMAP:
4830 if (ops->ndo_set_config) { 4837 if (ops->ndo_set_config) {
4831 if (!netif_device_present(dev)) 4838 if (!netif_device_present(dev))
4832 return -ENODEV; 4839 return -ENODEV;
4833 return ops->ndo_set_config(dev, &ifr->ifr_map); 4840 return ops->ndo_set_config(dev, &ifr->ifr_map);
4834 } 4841 }
4835 return -EOPNOTSUPP; 4842 return -EOPNOTSUPP;
4836 4843
4837 case SIOCADDMULTI: 4844 case SIOCADDMULTI:
4838 if ((!ops->ndo_set_multicast_list && !ops->ndo_set_rx_mode) || 4845 if ((!ops->ndo_set_multicast_list && !ops->ndo_set_rx_mode) ||
4839 ifr->ifr_hwaddr.sa_family != AF_UNSPEC) 4846 ifr->ifr_hwaddr.sa_family != AF_UNSPEC)
4840 return -EINVAL; 4847 return -EINVAL;
4841 if (!netif_device_present(dev)) 4848 if (!netif_device_present(dev))
4842 return -ENODEV; 4849 return -ENODEV;
4843 return dev_mc_add_global(dev, ifr->ifr_hwaddr.sa_data); 4850 return dev_mc_add_global(dev, ifr->ifr_hwaddr.sa_data);
4844 4851
4845 case SIOCDELMULTI: 4852 case SIOCDELMULTI:
4846 if ((!ops->ndo_set_multicast_list && !ops->ndo_set_rx_mode) || 4853 if ((!ops->ndo_set_multicast_list && !ops->ndo_set_rx_mode) ||
4847 ifr->ifr_hwaddr.sa_family != AF_UNSPEC) 4854 ifr->ifr_hwaddr.sa_family != AF_UNSPEC)
4848 return -EINVAL; 4855 return -EINVAL;
4849 if (!netif_device_present(dev)) 4856 if (!netif_device_present(dev))
4850 return -ENODEV; 4857 return -ENODEV;
4851 return dev_mc_del_global(dev, ifr->ifr_hwaddr.sa_data); 4858 return dev_mc_del_global(dev, ifr->ifr_hwaddr.sa_data);
4852 4859
4853 case SIOCSIFTXQLEN: 4860 case SIOCSIFTXQLEN:
4854 if (ifr->ifr_qlen < 0) 4861 if (ifr->ifr_qlen < 0)
4855 return -EINVAL; 4862 return -EINVAL;
4856 dev->tx_queue_len = ifr->ifr_qlen; 4863 dev->tx_queue_len = ifr->ifr_qlen;
4857 return 0; 4864 return 0;
4858 4865
4859 case SIOCSIFNAME: 4866 case SIOCSIFNAME:
4860 ifr->ifr_newname[IFNAMSIZ-1] = '\0'; 4867 ifr->ifr_newname[IFNAMSIZ-1] = '\0';
4861 return dev_change_name(dev, ifr->ifr_newname); 4868 return dev_change_name(dev, ifr->ifr_newname);
4862 4869
4863 /* 4870 /*
4864 * Unknown or private ioctl 4871 * Unknown or private ioctl
4865 */ 4872 */
4866 default: 4873 default:
4867 if ((cmd >= SIOCDEVPRIVATE && 4874 if ((cmd >= SIOCDEVPRIVATE &&
4868 cmd <= SIOCDEVPRIVATE + 15) || 4875 cmd <= SIOCDEVPRIVATE + 15) ||
4869 cmd == SIOCBONDENSLAVE || 4876 cmd == SIOCBONDENSLAVE ||
4870 cmd == SIOCBONDRELEASE || 4877 cmd == SIOCBONDRELEASE ||
4871 cmd == SIOCBONDSETHWADDR || 4878 cmd == SIOCBONDSETHWADDR ||
4872 cmd == SIOCBONDSLAVEINFOQUERY || 4879 cmd == SIOCBONDSLAVEINFOQUERY ||
4873 cmd == SIOCBONDINFOQUERY || 4880 cmd == SIOCBONDINFOQUERY ||
4874 cmd == SIOCBONDCHANGEACTIVE || 4881 cmd == SIOCBONDCHANGEACTIVE ||
4875 cmd == SIOCGMIIPHY || 4882 cmd == SIOCGMIIPHY ||
4876 cmd == SIOCGMIIREG || 4883 cmd == SIOCGMIIREG ||
4877 cmd == SIOCSMIIREG || 4884 cmd == SIOCSMIIREG ||
4878 cmd == SIOCBRADDIF || 4885 cmd == SIOCBRADDIF ||
4879 cmd == SIOCBRDELIF || 4886 cmd == SIOCBRDELIF ||
4880 cmd == SIOCSHWTSTAMP || 4887 cmd == SIOCSHWTSTAMP ||
4881 cmd == SIOCWANDEV) { 4888 cmd == SIOCWANDEV) {
4882 err = -EOPNOTSUPP; 4889 err = -EOPNOTSUPP;
4883 if (ops->ndo_do_ioctl) { 4890 if (ops->ndo_do_ioctl) {
4884 if (netif_device_present(dev)) 4891 if (netif_device_present(dev))
4885 err = ops->ndo_do_ioctl(dev, ifr, cmd); 4892 err = ops->ndo_do_ioctl(dev, ifr, cmd);
4886 else 4893 else
4887 err = -ENODEV; 4894 err = -ENODEV;
4888 } 4895 }
4889 } else 4896 } else
4890 err = -EINVAL; 4897 err = -EINVAL;
4891 4898
4892 } 4899 }
4893 return err; 4900 return err;
4894 } 4901 }
4895 4902
4896 /* 4903 /*
4897 * This function handles all "interface"-type I/O control requests. The actual 4904 * This function handles all "interface"-type I/O control requests. The actual
4898 * 'doing' part of this is dev_ifsioc above. 4905 * 'doing' part of this is dev_ifsioc above.
4899 */ 4906 */
4900 4907
4901 /** 4908 /**
4902 * dev_ioctl - network device ioctl 4909 * dev_ioctl - network device ioctl
4903 * @net: the applicable net namespace 4910 * @net: the applicable net namespace
4904 * @cmd: command to issue 4911 * @cmd: command to issue
4905 * @arg: pointer to a struct ifreq in user space 4912 * @arg: pointer to a struct ifreq in user space
4906 * 4913 *
4907 * Issue ioctl functions to devices. This is normally called by the 4914 * Issue ioctl functions to devices. This is normally called by the
4908 * user space syscall interfaces but can sometimes be useful for 4915 * user space syscall interfaces but can sometimes be useful for
4909 * other purposes. The return value is the return from the syscall if 4916 * other purposes. The return value is the return from the syscall if
4910 * positive or a negative errno code on error. 4917 * positive or a negative errno code on error.
4911 */ 4918 */
4912 4919
4913 int dev_ioctl(struct net *net, unsigned int cmd, void __user *arg) 4920 int dev_ioctl(struct net *net, unsigned int cmd, void __user *arg)
4914 { 4921 {
4915 struct ifreq ifr; 4922 struct ifreq ifr;
4916 int ret; 4923 int ret;
4917 char *colon; 4924 char *colon;
4918 4925
4919 /* One special case: SIOCGIFCONF takes ifconf argument 4926 /* One special case: SIOCGIFCONF takes ifconf argument
4920 and requires shared lock, because it sleeps writing 4927 and requires shared lock, because it sleeps writing
4921 to user space. 4928 to user space.
4922 */ 4929 */
4923 4930
4924 if (cmd == SIOCGIFCONF) { 4931 if (cmd == SIOCGIFCONF) {
4925 rtnl_lock(); 4932 rtnl_lock();
4926 ret = dev_ifconf(net, (char __user *) arg); 4933 ret = dev_ifconf(net, (char __user *) arg);
4927 rtnl_unlock(); 4934 rtnl_unlock();
4928 return ret; 4935 return ret;
4929 } 4936 }
4930 if (cmd == SIOCGIFNAME) 4937 if (cmd == SIOCGIFNAME)
4931 return dev_ifname(net, (struct ifreq __user *)arg); 4938 return dev_ifname(net, (struct ifreq __user *)arg);
4932 4939
4933 if (copy_from_user(&ifr, arg, sizeof(struct ifreq))) 4940 if (copy_from_user(&ifr, arg, sizeof(struct ifreq)))
4934 return -EFAULT; 4941 return -EFAULT;
4935 4942
4936 ifr.ifr_name[IFNAMSIZ-1] = 0; 4943 ifr.ifr_name[IFNAMSIZ-1] = 0;
4937 4944
4938 colon = strchr(ifr.ifr_name, ':'); 4945 colon = strchr(ifr.ifr_name, ':');
4939 if (colon) 4946 if (colon)
4940 *colon = 0; 4947 *colon = 0;
4941 4948
4942 /* 4949 /*
4943 * See which interface the caller is talking about. 4950 * See which interface the caller is talking about.
4944 */ 4951 */
4945 4952
4946 switch (cmd) { 4953 switch (cmd) {
4947 /* 4954 /*
4948 * These ioctl calls: 4955 * These ioctl calls:
4949 * - can be done by all. 4956 * - can be done by all.
4950 * - atomic and do not require locking. 4957 * - atomic and do not require locking.
4951 * - return a value 4958 * - return a value
4952 */ 4959 */
4953 case SIOCGIFFLAGS: 4960 case SIOCGIFFLAGS:
4954 case SIOCGIFMETRIC: 4961 case SIOCGIFMETRIC:
4955 case SIOCGIFMTU: 4962 case SIOCGIFMTU:
4956 case SIOCGIFHWADDR: 4963 case SIOCGIFHWADDR:
4957 case SIOCGIFSLAVE: 4964 case SIOCGIFSLAVE:
4958 case SIOCGIFMAP: 4965 case SIOCGIFMAP:
4959 case SIOCGIFINDEX: 4966 case SIOCGIFINDEX:
4960 case SIOCGIFTXQLEN: 4967 case SIOCGIFTXQLEN:
4961 dev_load(net, ifr.ifr_name); 4968 dev_load(net, ifr.ifr_name);
4962 rcu_read_lock(); 4969 rcu_read_lock();
4963 ret = dev_ifsioc_locked(net, &ifr, cmd); 4970 ret = dev_ifsioc_locked(net, &ifr, cmd);
4964 rcu_read_unlock(); 4971 rcu_read_unlock();
4965 if (!ret) { 4972 if (!ret) {
4966 if (colon) 4973 if (colon)
4967 *colon = ':'; 4974 *colon = ':';
4968 if (copy_to_user(arg, &ifr, 4975 if (copy_to_user(arg, &ifr,
4969 sizeof(struct ifreq))) 4976 sizeof(struct ifreq)))
4970 ret = -EFAULT; 4977 ret = -EFAULT;
4971 } 4978 }
4972 return ret; 4979 return ret;
4973 4980
4974 case SIOCETHTOOL: 4981 case SIOCETHTOOL:
4975 dev_load(net, ifr.ifr_name); 4982 dev_load(net, ifr.ifr_name);
4976 rtnl_lock(); 4983 rtnl_lock();
4977 ret = dev_ethtool(net, &ifr); 4984 ret = dev_ethtool(net, &ifr);
4978 rtnl_unlock(); 4985 rtnl_unlock();
4979 if (!ret) { 4986 if (!ret) {
4980 if (colon) 4987 if (colon)
4981 *colon = ':'; 4988 *colon = ':';
4982 if (copy_to_user(arg, &ifr, 4989 if (copy_to_user(arg, &ifr,
4983 sizeof(struct ifreq))) 4990 sizeof(struct ifreq)))
4984 ret = -EFAULT; 4991 ret = -EFAULT;
4985 } 4992 }
4986 return ret; 4993 return ret;
4987 4994
4988 /* 4995 /*
4989 * These ioctl calls: 4996 * These ioctl calls:
4990 * - require superuser power. 4997 * - require superuser power.
4991 * - require strict serialization. 4998 * - require strict serialization.
4992 * - return a value 4999 * - return a value
4993 */ 5000 */
4994 case SIOCGMIIPHY: 5001 case SIOCGMIIPHY:
4995 case SIOCGMIIREG: 5002 case SIOCGMIIREG:
4996 case SIOCSIFNAME: 5003 case SIOCSIFNAME:
4997 if (!capable(CAP_NET_ADMIN)) 5004 if (!capable(CAP_NET_ADMIN))
4998 return -EPERM; 5005 return -EPERM;
4999 dev_load(net, ifr.ifr_name); 5006 dev_load(net, ifr.ifr_name);
5000 rtnl_lock(); 5007 rtnl_lock();
5001 ret = dev_ifsioc(net, &ifr, cmd); 5008 ret = dev_ifsioc(net, &ifr, cmd);
5002 rtnl_unlock(); 5009 rtnl_unlock();
5003 if (!ret) { 5010 if (!ret) {
5004 if (colon) 5011 if (colon)
5005 *colon = ':'; 5012 *colon = ':';
5006 if (copy_to_user(arg, &ifr, 5013 if (copy_to_user(arg, &ifr,
5007 sizeof(struct ifreq))) 5014 sizeof(struct ifreq)))
5008 ret = -EFAULT; 5015 ret = -EFAULT;
5009 } 5016 }
5010 return ret; 5017 return ret;
5011 5018
5012 /* 5019 /*
5013 * These ioctl calls: 5020 * These ioctl calls:
5014 * - require superuser power. 5021 * - require superuser power.
5015 * - require strict serialization. 5022 * - require strict serialization.
5016 * - do not return a value 5023 * - do not return a value
5017 */ 5024 */
5018 case SIOCSIFFLAGS: 5025 case SIOCSIFFLAGS:
5019 case SIOCSIFMETRIC: 5026 case SIOCSIFMETRIC:
5020 case SIOCSIFMTU: 5027 case SIOCSIFMTU:
5021 case SIOCSIFMAP: 5028 case SIOCSIFMAP:
5022 case SIOCSIFHWADDR: 5029 case SIOCSIFHWADDR:
5023 case SIOCSIFSLAVE: 5030 case SIOCSIFSLAVE:
5024 case SIOCADDMULTI: 5031 case SIOCADDMULTI:
5025 case SIOCDELMULTI: 5032 case SIOCDELMULTI:
5026 case SIOCSIFHWBROADCAST: 5033 case SIOCSIFHWBROADCAST:
5027 case SIOCSIFTXQLEN: 5034 case SIOCSIFTXQLEN:
5028 case SIOCSMIIREG: 5035 case SIOCSMIIREG:
5029 case SIOCBONDENSLAVE: 5036 case SIOCBONDENSLAVE:
5030 case SIOCBONDRELEASE: 5037 case SIOCBONDRELEASE:
5031 case SIOCBONDSETHWADDR: 5038 case SIOCBONDSETHWADDR:
5032 case SIOCBONDCHANGEACTIVE: 5039 case SIOCBONDCHANGEACTIVE:
5033 case SIOCBRADDIF: 5040 case SIOCBRADDIF:
5034 case SIOCBRDELIF: 5041 case SIOCBRDELIF:
5035 case SIOCSHWTSTAMP: 5042 case SIOCSHWTSTAMP:
5036 if (!capable(CAP_NET_ADMIN)) 5043 if (!capable(CAP_NET_ADMIN))
5037 return -EPERM; 5044 return -EPERM;
5038 /* fall through */ 5045 /* fall through */
5039 case SIOCBONDSLAVEINFOQUERY: 5046 case SIOCBONDSLAVEINFOQUERY:
5040 case SIOCBONDINFOQUERY: 5047 case SIOCBONDINFOQUERY:
5041 dev_load(net, ifr.ifr_name); 5048 dev_load(net, ifr.ifr_name);
5042 rtnl_lock(); 5049 rtnl_lock();
5043 ret = dev_ifsioc(net, &ifr, cmd); 5050 ret = dev_ifsioc(net, &ifr, cmd);
5044 rtnl_unlock(); 5051 rtnl_unlock();
5045 return ret; 5052 return ret;
5046 5053
5047 case SIOCGIFMEM: 5054 case SIOCGIFMEM:
5048 /* Get the per device memory space. We can add this but 5055 /* Get the per device memory space. We can add this but
5049 * currently do not support it */ 5056 * currently do not support it */
5050 case SIOCSIFMEM: 5057 case SIOCSIFMEM:
5051 /* Set the per device memory buffer space. 5058 /* Set the per device memory buffer space.
5052 * Not applicable in our case */ 5059 * Not applicable in our case */
5053 case SIOCSIFLINK: 5060 case SIOCSIFLINK:
5054 return -ENOTTY; 5061 return -ENOTTY;
5055 5062
5056 /* 5063 /*
5057 * Unknown or private ioctl. 5064 * Unknown or private ioctl.
5058 */ 5065 */
5059 default: 5066 default:
5060 if (cmd == SIOCWANDEV || 5067 if (cmd == SIOCWANDEV ||
5061 (cmd >= SIOCDEVPRIVATE && 5068 (cmd >= SIOCDEVPRIVATE &&
5062 cmd <= SIOCDEVPRIVATE + 15)) { 5069 cmd <= SIOCDEVPRIVATE + 15)) {
5063 dev_load(net, ifr.ifr_name); 5070 dev_load(net, ifr.ifr_name);
5064 rtnl_lock(); 5071 rtnl_lock();
5065 ret = dev_ifsioc(net, &ifr, cmd); 5072 ret = dev_ifsioc(net, &ifr, cmd);
5066 rtnl_unlock(); 5073 rtnl_unlock();
5067 if (!ret && copy_to_user(arg, &ifr, 5074 if (!ret && copy_to_user(arg, &ifr,
5068 sizeof(struct ifreq))) 5075 sizeof(struct ifreq)))
5069 ret = -EFAULT; 5076 ret = -EFAULT;
5070 return ret; 5077 return ret;
5071 } 5078 }
5072 /* Take care of Wireless Extensions */ 5079 /* Take care of Wireless Extensions */
5073 if (cmd >= SIOCIWFIRST && cmd <= SIOCIWLAST) 5080 if (cmd >= SIOCIWFIRST && cmd <= SIOCIWLAST)
5074 return wext_handle_ioctl(net, &ifr, cmd, arg); 5081 return wext_handle_ioctl(net, &ifr, cmd, arg);
5075 return -ENOTTY; 5082 return -ENOTTY;
5076 } 5083 }
5077 } 5084 }
5078 5085
5079 5086
5080 /** 5087 /**
5081 * dev_new_index - allocate an ifindex 5088 * dev_new_index - allocate an ifindex
5082 * @net: the applicable net namespace 5089 * @net: the applicable net namespace
5083 * 5090 *
5084 * Returns a suitable unique value for a new device interface 5091 * Returns a suitable unique value for a new device interface
5085 * number. The caller must hold the rtnl semaphore or the 5092 * number. The caller must hold the rtnl semaphore or the
5086 * dev_base_lock to be sure it remains unique. 5093 * dev_base_lock to be sure it remains unique.
5087 */ 5094 */
5088 static int dev_new_index(struct net *net) 5095 static int dev_new_index(struct net *net)
5089 { 5096 {
5090 static int ifindex; 5097 static int ifindex;
5091 for (;;) { 5098 for (;;) {
5092 if (++ifindex <= 0) 5099 if (++ifindex <= 0)
5093 ifindex = 1; 5100 ifindex = 1;
5094 if (!__dev_get_by_index(net, ifindex)) 5101 if (!__dev_get_by_index(net, ifindex))
5095 return ifindex; 5102 return ifindex;
5096 } 5103 }
5097 } 5104 }
5098 5105
5099 /* Delayed registration/unregisteration */ 5106 /* Delayed registration/unregisteration */
5100 static LIST_HEAD(net_todo_list); 5107 static LIST_HEAD(net_todo_list);
5101 5108
5102 static void net_set_todo(struct net_device *dev) 5109 static void net_set_todo(struct net_device *dev)
5103 { 5110 {
5104 list_add_tail(&dev->todo_list, &net_todo_list); 5111 list_add_tail(&dev->todo_list, &net_todo_list);
5105 } 5112 }
5106 5113
5107 static void rollback_registered_many(struct list_head *head) 5114 static void rollback_registered_many(struct list_head *head)
5108 { 5115 {
5109 struct net_device *dev, *tmp; 5116 struct net_device *dev, *tmp;
5110 5117
5111 BUG_ON(dev_boot_phase); 5118 BUG_ON(dev_boot_phase);
5112 ASSERT_RTNL(); 5119 ASSERT_RTNL();
5113 5120
5114 list_for_each_entry_safe(dev, tmp, head, unreg_list) { 5121 list_for_each_entry_safe(dev, tmp, head, unreg_list) {
5115 /* Some devices call without registering 5122 /* Some devices call without registering
5116 * for initialization unwind. Remove those 5123 * for initialization unwind. Remove those
5117 * devices and proceed with the remaining. 5124 * devices and proceed with the remaining.
5118 */ 5125 */
5119 if (dev->reg_state == NETREG_UNINITIALIZED) { 5126 if (dev->reg_state == NETREG_UNINITIALIZED) {
5120 pr_debug("unregister_netdevice: device %s/%p never " 5127 pr_debug("unregister_netdevice: device %s/%p never "
5121 "was registered\n", dev->name, dev); 5128 "was registered\n", dev->name, dev);
5122 5129
5123 WARN_ON(1); 5130 WARN_ON(1);
5124 list_del(&dev->unreg_list); 5131 list_del(&dev->unreg_list);
5125 continue; 5132 continue;
5126 } 5133 }
5127 dev->dismantle = true; 5134 dev->dismantle = true;
5128 BUG_ON(dev->reg_state != NETREG_REGISTERED); 5135 BUG_ON(dev->reg_state != NETREG_REGISTERED);
5129 } 5136 }
5130 5137
5131 /* If device is running, close it first. */ 5138 /* If device is running, close it first. */
5132 dev_close_many(head); 5139 dev_close_many(head);
5133 5140
5134 list_for_each_entry(dev, head, unreg_list) { 5141 list_for_each_entry(dev, head, unreg_list) {
5135 /* And unlink it from device chain. */ 5142 /* And unlink it from device chain. */
5136 unlist_netdevice(dev); 5143 unlist_netdevice(dev);
5137 5144
5138 dev->reg_state = NETREG_UNREGISTERING; 5145 dev->reg_state = NETREG_UNREGISTERING;
5139 } 5146 }
5140 5147
5141 synchronize_net(); 5148 synchronize_net();
5142 5149
5143 list_for_each_entry(dev, head, unreg_list) { 5150 list_for_each_entry(dev, head, unreg_list) {
5144 /* Shutdown queueing discipline. */ 5151 /* Shutdown queueing discipline. */
5145 dev_shutdown(dev); 5152 dev_shutdown(dev);
5146 5153
5147 5154
5148 /* Notify protocols, that we are about to destroy 5155 /* Notify protocols, that we are about to destroy
5149 this device. They should clean all the things. 5156 this device. They should clean all the things.
5150 */ 5157 */
5151 call_netdevice_notifiers(NETDEV_UNREGISTER, dev); 5158 call_netdevice_notifiers(NETDEV_UNREGISTER, dev);
5152 5159
5153 if (!dev->rtnl_link_ops || 5160 if (!dev->rtnl_link_ops ||
5154 dev->rtnl_link_state == RTNL_LINK_INITIALIZED) 5161 dev->rtnl_link_state == RTNL_LINK_INITIALIZED)
5155 rtmsg_ifinfo(RTM_DELLINK, dev, ~0U); 5162 rtmsg_ifinfo(RTM_DELLINK, dev, ~0U);
5156 5163
5157 /* 5164 /*
5158 * Flush the unicast and multicast chains 5165 * Flush the unicast and multicast chains
5159 */ 5166 */
5160 dev_uc_flush(dev); 5167 dev_uc_flush(dev);
5161 dev_mc_flush(dev); 5168 dev_mc_flush(dev);
5162 5169
5163 if (dev->netdev_ops->ndo_uninit) 5170 if (dev->netdev_ops->ndo_uninit)
5164 dev->netdev_ops->ndo_uninit(dev); 5171 dev->netdev_ops->ndo_uninit(dev);
5165 5172
5166 /* Notifier chain MUST detach us from master device. */ 5173 /* Notifier chain MUST detach us from master device. */
5167 WARN_ON(dev->master); 5174 WARN_ON(dev->master);
5168 5175
5169 /* Remove entries from kobject tree */ 5176 /* Remove entries from kobject tree */
5170 netdev_unregister_kobject(dev); 5177 netdev_unregister_kobject(dev);
5171 } 5178 }
5172 5179
5173 /* Process any work delayed until the end of the batch */ 5180 /* Process any work delayed until the end of the batch */
5174 dev = list_first_entry(head, struct net_device, unreg_list); 5181 dev = list_first_entry(head, struct net_device, unreg_list);
5175 call_netdevice_notifiers(NETDEV_UNREGISTER_BATCH, dev); 5182 call_netdevice_notifiers(NETDEV_UNREGISTER_BATCH, dev);
5176 5183
5177 rcu_barrier(); 5184 rcu_barrier();
5178 5185
5179 list_for_each_entry(dev, head, unreg_list) 5186 list_for_each_entry(dev, head, unreg_list)
5180 dev_put(dev); 5187 dev_put(dev);
5181 } 5188 }
5182 5189
5183 static void rollback_registered(struct net_device *dev) 5190 static void rollback_registered(struct net_device *dev)
5184 { 5191 {
5185 LIST_HEAD(single); 5192 LIST_HEAD(single);
5186 5193
5187 list_add(&dev->unreg_list, &single); 5194 list_add(&dev->unreg_list, &single);
5188 rollback_registered_many(&single); 5195 rollback_registered_many(&single);
5189 list_del(&single); 5196 list_del(&single);
5190 } 5197 }
5191 5198
5192 u32 netdev_fix_features(struct net_device *dev, u32 features) 5199 u32 netdev_fix_features(struct net_device *dev, u32 features)
5193 { 5200 {
5194 /* Fix illegal checksum combinations */ 5201 /* Fix illegal checksum combinations */
5195 if ((features & NETIF_F_HW_CSUM) && 5202 if ((features & NETIF_F_HW_CSUM) &&
5196 (features & (NETIF_F_IP_CSUM|NETIF_F_IPV6_CSUM))) { 5203 (features & (NETIF_F_IP_CSUM|NETIF_F_IPV6_CSUM))) {
5197 netdev_warn(dev, "mixed HW and IP checksum settings.\n"); 5204 netdev_warn(dev, "mixed HW and IP checksum settings.\n");
5198 features &= ~(NETIF_F_IP_CSUM|NETIF_F_IPV6_CSUM); 5205 features &= ~(NETIF_F_IP_CSUM|NETIF_F_IPV6_CSUM);
5199 } 5206 }
5200 5207
5201 if ((features & NETIF_F_NO_CSUM) && 5208 if ((features & NETIF_F_NO_CSUM) &&
5202 (features & (NETIF_F_HW_CSUM|NETIF_F_IP_CSUM|NETIF_F_IPV6_CSUM))) { 5209 (features & (NETIF_F_HW_CSUM|NETIF_F_IP_CSUM|NETIF_F_IPV6_CSUM))) {
5203 netdev_warn(dev, "mixed no checksumming and other settings.\n"); 5210 netdev_warn(dev, "mixed no checksumming and other settings.\n");
5204 features &= ~(NETIF_F_IP_CSUM|NETIF_F_IPV6_CSUM|NETIF_F_HW_CSUM); 5211 features &= ~(NETIF_F_IP_CSUM|NETIF_F_IPV6_CSUM|NETIF_F_HW_CSUM);
5205 } 5212 }
5206 5213
5207 /* Fix illegal SG+CSUM combinations. */ 5214 /* Fix illegal SG+CSUM combinations. */
5208 if ((features & NETIF_F_SG) && 5215 if ((features & NETIF_F_SG) &&
5209 !(features & NETIF_F_ALL_CSUM)) { 5216 !(features & NETIF_F_ALL_CSUM)) {
5210 netdev_dbg(dev, 5217 netdev_dbg(dev,
5211 "Dropping NETIF_F_SG since no checksum feature.\n"); 5218 "Dropping NETIF_F_SG since no checksum feature.\n");
5212 features &= ~NETIF_F_SG; 5219 features &= ~NETIF_F_SG;
5213 } 5220 }
5214 5221
5215 /* TSO requires that SG is present as well. */ 5222 /* TSO requires that SG is present as well. */
5216 if ((features & NETIF_F_ALL_TSO) && !(features & NETIF_F_SG)) { 5223 if ((features & NETIF_F_ALL_TSO) && !(features & NETIF_F_SG)) {
5217 netdev_dbg(dev, "Dropping TSO features since no SG feature.\n"); 5224 netdev_dbg(dev, "Dropping TSO features since no SG feature.\n");
5218 features &= ~NETIF_F_ALL_TSO; 5225 features &= ~NETIF_F_ALL_TSO;
5219 } 5226 }
5220 5227
5221 /* TSO ECN requires that TSO is present as well. */ 5228 /* TSO ECN requires that TSO is present as well. */
5222 if ((features & NETIF_F_ALL_TSO) == NETIF_F_TSO_ECN) 5229 if ((features & NETIF_F_ALL_TSO) == NETIF_F_TSO_ECN)
5223 features &= ~NETIF_F_TSO_ECN; 5230 features &= ~NETIF_F_TSO_ECN;
5224 5231
5225 /* Software GSO depends on SG. */ 5232 /* Software GSO depends on SG. */
5226 if ((features & NETIF_F_GSO) && !(features & NETIF_F_SG)) { 5233 if ((features & NETIF_F_GSO) && !(features & NETIF_F_SG)) {
5227 netdev_dbg(dev, "Dropping NETIF_F_GSO since no SG feature.\n"); 5234 netdev_dbg(dev, "Dropping NETIF_F_GSO since no SG feature.\n");
5228 features &= ~NETIF_F_GSO; 5235 features &= ~NETIF_F_GSO;
5229 } 5236 }
5230 5237
5231 /* UFO needs SG and checksumming */ 5238 /* UFO needs SG and checksumming */
5232 if (features & NETIF_F_UFO) { 5239 if (features & NETIF_F_UFO) {
5233 /* maybe split UFO into V4 and V6? */ 5240 /* maybe split UFO into V4 and V6? */
5234 if (!((features & NETIF_F_GEN_CSUM) || 5241 if (!((features & NETIF_F_GEN_CSUM) ||
5235 (features & (NETIF_F_IP_CSUM|NETIF_F_IPV6_CSUM)) 5242 (features & (NETIF_F_IP_CSUM|NETIF_F_IPV6_CSUM))
5236 == (NETIF_F_IP_CSUM|NETIF_F_IPV6_CSUM))) { 5243 == (NETIF_F_IP_CSUM|NETIF_F_IPV6_CSUM))) {
5237 netdev_dbg(dev, 5244 netdev_dbg(dev,
5238 "Dropping NETIF_F_UFO since no checksum offload features.\n"); 5245 "Dropping NETIF_F_UFO since no checksum offload features.\n");
5239 features &= ~NETIF_F_UFO; 5246 features &= ~NETIF_F_UFO;
5240 } 5247 }
5241 5248
5242 if (!(features & NETIF_F_SG)) { 5249 if (!(features & NETIF_F_SG)) {
5243 netdev_dbg(dev, 5250 netdev_dbg(dev,
5244 "Dropping NETIF_F_UFO since no NETIF_F_SG feature.\n"); 5251 "Dropping NETIF_F_UFO since no NETIF_F_SG feature.\n");
5245 features &= ~NETIF_F_UFO; 5252 features &= ~NETIF_F_UFO;
5246 } 5253 }
5247 } 5254 }
5248 5255
5249 return features; 5256 return features;
5250 } 5257 }
5251 EXPORT_SYMBOL(netdev_fix_features); 5258 EXPORT_SYMBOL(netdev_fix_features);
5252 5259
5253 int __netdev_update_features(struct net_device *dev) 5260 int __netdev_update_features(struct net_device *dev)
5254 { 5261 {
5255 u32 features; 5262 u32 features;
5256 int err = 0; 5263 int err = 0;
5257 5264
5258 ASSERT_RTNL(); 5265 ASSERT_RTNL();
5259 5266
5260 features = netdev_get_wanted_features(dev); 5267 features = netdev_get_wanted_features(dev);
5261 5268
5262 if (dev->netdev_ops->ndo_fix_features) 5269 if (dev->netdev_ops->ndo_fix_features)
5263 features = dev->netdev_ops->ndo_fix_features(dev, features); 5270 features = dev->netdev_ops->ndo_fix_features(dev, features);
5264 5271
5265 /* driver might be less strict about feature dependencies */ 5272 /* driver might be less strict about feature dependencies */
5266 features = netdev_fix_features(dev, features); 5273 features = netdev_fix_features(dev, features);
5267 5274
5268 if (dev->features == features) 5275 if (dev->features == features)
5269 return 0; 5276 return 0;
5270 5277
5271 netdev_dbg(dev, "Features changed: 0x%08x -> 0x%08x\n", 5278 netdev_dbg(dev, "Features changed: 0x%08x -> 0x%08x\n",
5272 dev->features, features); 5279 dev->features, features);
5273 5280
5274 if (dev->netdev_ops->ndo_set_features) 5281 if (dev->netdev_ops->ndo_set_features)
5275 err = dev->netdev_ops->ndo_set_features(dev, features); 5282 err = dev->netdev_ops->ndo_set_features(dev, features);
5276 5283
5277 if (unlikely(err < 0)) { 5284 if (unlikely(err < 0)) {
5278 netdev_err(dev, 5285 netdev_err(dev,
5279 "set_features() failed (%d); wanted 0x%08x, left 0x%08x\n", 5286 "set_features() failed (%d); wanted 0x%08x, left 0x%08x\n",
5280 err, features, dev->features); 5287 err, features, dev->features);
5281 return -1; 5288 return -1;
5282 } 5289 }
5283 5290
5284 if (!err) 5291 if (!err)
5285 dev->features = features; 5292 dev->features = features;
5286 5293
5287 return 1; 5294 return 1;
5288 } 5295 }
5289 5296
5290 /** 5297 /**
5291 * netdev_update_features - recalculate device features 5298 * netdev_update_features - recalculate device features
5292 * @dev: the device to check 5299 * @dev: the device to check
5293 * 5300 *
5294 * Recalculate dev->features set and send notifications if it 5301 * Recalculate dev->features set and send notifications if it
5295 * has changed. Should be called after driver or hardware dependent 5302 * has changed. Should be called after driver or hardware dependent
5296 * conditions might have changed that influence the features. 5303 * conditions might have changed that influence the features.
5297 */ 5304 */
5298 void netdev_update_features(struct net_device *dev) 5305 void netdev_update_features(struct net_device *dev)
5299 { 5306 {
5300 if (__netdev_update_features(dev)) 5307 if (__netdev_update_features(dev))
5301 netdev_features_change(dev); 5308 netdev_features_change(dev);
5302 } 5309 }
5303 EXPORT_SYMBOL(netdev_update_features); 5310 EXPORT_SYMBOL(netdev_update_features);
5304 5311
5305 /** 5312 /**
5306 * netdev_change_features - recalculate device features 5313 * netdev_change_features - recalculate device features
5307 * @dev: the device to check 5314 * @dev: the device to check
5308 * 5315 *
5309 * Recalculate dev->features set and send notifications even 5316 * Recalculate dev->features set and send notifications even
5310 * if they have not changed. Should be called instead of 5317 * if they have not changed. Should be called instead of
5311 * netdev_update_features() if also dev->vlan_features might 5318 * netdev_update_features() if also dev->vlan_features might
5312 * have changed to allow the changes to be propagated to stacked 5319 * have changed to allow the changes to be propagated to stacked
5313 * VLAN devices. 5320 * VLAN devices.
5314 */ 5321 */
5315 void netdev_change_features(struct net_device *dev) 5322 void netdev_change_features(struct net_device *dev)
5316 { 5323 {
5317 __netdev_update_features(dev); 5324 __netdev_update_features(dev);
5318 netdev_features_change(dev); 5325 netdev_features_change(dev);
5319 } 5326 }
5320 EXPORT_SYMBOL(netdev_change_features); 5327 EXPORT_SYMBOL(netdev_change_features);
5321 5328
5322 /** 5329 /**
5323 * netif_stacked_transfer_operstate - transfer operstate 5330 * netif_stacked_transfer_operstate - transfer operstate
5324 * @rootdev: the root or lower level device to transfer state from 5331 * @rootdev: the root or lower level device to transfer state from
5325 * @dev: the device to transfer operstate to 5332 * @dev: the device to transfer operstate to
5326 * 5333 *
5327 * Transfer operational state from root to device. This is normally 5334 * Transfer operational state from root to device. This is normally
5328 * called when a stacking relationship exists between the root 5335 * called when a stacking relationship exists between the root
5329 * device and the device(a leaf device). 5336 * device and the device(a leaf device).
5330 */ 5337 */
5331 void netif_stacked_transfer_operstate(const struct net_device *rootdev, 5338 void netif_stacked_transfer_operstate(const struct net_device *rootdev,
5332 struct net_device *dev) 5339 struct net_device *dev)
5333 { 5340 {
5334 if (rootdev->operstate == IF_OPER_DORMANT) 5341 if (rootdev->operstate == IF_OPER_DORMANT)
5335 netif_dormant_on(dev); 5342 netif_dormant_on(dev);
5336 else 5343 else
5337 netif_dormant_off(dev); 5344 netif_dormant_off(dev);
5338 5345
5339 if (netif_carrier_ok(rootdev)) { 5346 if (netif_carrier_ok(rootdev)) {
5340 if (!netif_carrier_ok(dev)) 5347 if (!netif_carrier_ok(dev))
5341 netif_carrier_on(dev); 5348 netif_carrier_on(dev);
5342 } else { 5349 } else {
5343 if (netif_carrier_ok(dev)) 5350 if (netif_carrier_ok(dev))
5344 netif_carrier_off(dev); 5351 netif_carrier_off(dev);
5345 } 5352 }
5346 } 5353 }
5347 EXPORT_SYMBOL(netif_stacked_transfer_operstate); 5354 EXPORT_SYMBOL(netif_stacked_transfer_operstate);
5348 5355
5349 #ifdef CONFIG_RPS 5356 #ifdef CONFIG_RPS
5350 static int netif_alloc_rx_queues(struct net_device *dev) 5357 static int netif_alloc_rx_queues(struct net_device *dev)
5351 { 5358 {
5352 unsigned int i, count = dev->num_rx_queues; 5359 unsigned int i, count = dev->num_rx_queues;
5353 struct netdev_rx_queue *rx; 5360 struct netdev_rx_queue *rx;
5354 5361
5355 BUG_ON(count < 1); 5362 BUG_ON(count < 1);
5356 5363
5357 rx = kcalloc(count, sizeof(struct netdev_rx_queue), GFP_KERNEL); 5364 rx = kcalloc(count, sizeof(struct netdev_rx_queue), GFP_KERNEL);
5358 if (!rx) { 5365 if (!rx) {
5359 pr_err("netdev: Unable to allocate %u rx queues.\n", count); 5366 pr_err("netdev: Unable to allocate %u rx queues.\n", count);
5360 return -ENOMEM; 5367 return -ENOMEM;
5361 } 5368 }
5362 dev->_rx = rx; 5369 dev->_rx = rx;
5363 5370
5364 for (i = 0; i < count; i++) 5371 for (i = 0; i < count; i++)
5365 rx[i].dev = dev; 5372 rx[i].dev = dev;
5366 return 0; 5373 return 0;
5367 } 5374 }
5368 #endif 5375 #endif
5369 5376
5370 static void netdev_init_one_queue(struct net_device *dev, 5377 static void netdev_init_one_queue(struct net_device *dev,
5371 struct netdev_queue *queue, void *_unused) 5378 struct netdev_queue *queue, void *_unused)
5372 { 5379 {
5373 /* Initialize queue lock */ 5380 /* Initialize queue lock */
5374 spin_lock_init(&queue->_xmit_lock); 5381 spin_lock_init(&queue->_xmit_lock);
5375 netdev_set_xmit_lockdep_class(&queue->_xmit_lock, dev->type); 5382 netdev_set_xmit_lockdep_class(&queue->_xmit_lock, dev->type);
5376 queue->xmit_lock_owner = -1; 5383 queue->xmit_lock_owner = -1;
5377 netdev_queue_numa_node_write(queue, NUMA_NO_NODE); 5384 netdev_queue_numa_node_write(queue, NUMA_NO_NODE);
5378 queue->dev = dev; 5385 queue->dev = dev;
5379 } 5386 }
5380 5387
5381 static int netif_alloc_netdev_queues(struct net_device *dev) 5388 static int netif_alloc_netdev_queues(struct net_device *dev)
5382 { 5389 {
5383 unsigned int count = dev->num_tx_queues; 5390 unsigned int count = dev->num_tx_queues;
5384 struct netdev_queue *tx; 5391 struct netdev_queue *tx;
5385 5392
5386 BUG_ON(count < 1); 5393 BUG_ON(count < 1);
5387 5394
5388 tx = kcalloc(count, sizeof(struct netdev_queue), GFP_KERNEL); 5395 tx = kcalloc(count, sizeof(struct netdev_queue), GFP_KERNEL);
5389 if (!tx) { 5396 if (!tx) {
5390 pr_err("netdev: Unable to allocate %u tx queues.\n", 5397 pr_err("netdev: Unable to allocate %u tx queues.\n",
5391 count); 5398 count);
5392 return -ENOMEM; 5399 return -ENOMEM;
5393 } 5400 }
5394 dev->_tx = tx; 5401 dev->_tx = tx;
5395 5402
5396 netdev_for_each_tx_queue(dev, netdev_init_one_queue, NULL); 5403 netdev_for_each_tx_queue(dev, netdev_init_one_queue, NULL);
5397 spin_lock_init(&dev->tx_global_lock); 5404 spin_lock_init(&dev->tx_global_lock);
5398 5405
5399 return 0; 5406 return 0;
5400 } 5407 }
5401 5408
5402 /** 5409 /**
5403 * register_netdevice - register a network device 5410 * register_netdevice - register a network device
5404 * @dev: device to register 5411 * @dev: device to register
5405 * 5412 *
5406 * Take a completed network device structure and add it to the kernel 5413 * Take a completed network device structure and add it to the kernel
5407 * interfaces. A %NETDEV_REGISTER message is sent to the netdev notifier 5414 * interfaces. A %NETDEV_REGISTER message is sent to the netdev notifier
5408 * chain. 0 is returned on success. A negative errno code is returned 5415 * chain. 0 is returned on success. A negative errno code is returned
5409 * on a failure to set up the device, or if the name is a duplicate. 5416 * on a failure to set up the device, or if the name is a duplicate.
5410 * 5417 *
5411 * Callers must hold the rtnl semaphore. You may want 5418 * Callers must hold the rtnl semaphore. You may want
5412 * register_netdev() instead of this. 5419 * register_netdev() instead of this.
5413 * 5420 *
5414 * BUGS: 5421 * BUGS:
5415 * The locking appears insufficient to guarantee two parallel registers 5422 * The locking appears insufficient to guarantee two parallel registers
5416 * will not get the same name. 5423 * will not get the same name.
5417 */ 5424 */
5418 5425
5419 int register_netdevice(struct net_device *dev) 5426 int register_netdevice(struct net_device *dev)
5420 { 5427 {
5421 int ret; 5428 int ret;
5422 struct net *net = dev_net(dev); 5429 struct net *net = dev_net(dev);
5423 5430
5424 BUG_ON(dev_boot_phase); 5431 BUG_ON(dev_boot_phase);
5425 ASSERT_RTNL(); 5432 ASSERT_RTNL();
5426 5433
5427 might_sleep(); 5434 might_sleep();
5428 5435
5429 /* When net_device's are persistent, this will be fatal. */ 5436 /* When net_device's are persistent, this will be fatal. */
5430 BUG_ON(dev->reg_state != NETREG_UNINITIALIZED); 5437 BUG_ON(dev->reg_state != NETREG_UNINITIALIZED);
5431 BUG_ON(!net); 5438 BUG_ON(!net);
5432 5439
5433 spin_lock_init(&dev->addr_list_lock); 5440 spin_lock_init(&dev->addr_list_lock);
5434 netdev_set_addr_lockdep_class(dev); 5441 netdev_set_addr_lockdep_class(dev);
5435 5442
5436 dev->iflink = -1; 5443 dev->iflink = -1;
5437 5444
5438 ret = dev_get_valid_name(dev, dev->name); 5445 ret = dev_get_valid_name(dev, dev->name);
5439 if (ret < 0) 5446 if (ret < 0)
5440 goto out; 5447 goto out;
5441 5448
5442 /* Init, if this function is available */ 5449 /* Init, if this function is available */
5443 if (dev->netdev_ops->ndo_init) { 5450 if (dev->netdev_ops->ndo_init) {
5444 ret = dev->netdev_ops->ndo_init(dev); 5451 ret = dev->netdev_ops->ndo_init(dev);
5445 if (ret) { 5452 if (ret) {
5446 if (ret > 0) 5453 if (ret > 0)
5447 ret = -EIO; 5454 ret = -EIO;
5448 goto out; 5455 goto out;
5449 } 5456 }
5450 } 5457 }
5451 5458
5452 dev->ifindex = dev_new_index(net); 5459 dev->ifindex = dev_new_index(net);
5453 if (dev->iflink == -1) 5460 if (dev->iflink == -1)
5454 dev->iflink = dev->ifindex; 5461 dev->iflink = dev->ifindex;
5455 5462
5456 /* Transfer changeable features to wanted_features and enable 5463 /* Transfer changeable features to wanted_features and enable
5457 * software offloads (GSO and GRO). 5464 * software offloads (GSO and GRO).
5458 */ 5465 */
5459 dev->hw_features |= NETIF_F_SOFT_FEATURES; 5466 dev->hw_features |= NETIF_F_SOFT_FEATURES;
5460 dev->features |= NETIF_F_SOFT_FEATURES; 5467 dev->features |= NETIF_F_SOFT_FEATURES;
5461 dev->wanted_features = dev->features & dev->hw_features; 5468 dev->wanted_features = dev->features & dev->hw_features;
5462 5469
5463 /* Turn on no cache copy if HW is doing checksum */ 5470 /* Turn on no cache copy if HW is doing checksum */
5464 dev->hw_features |= NETIF_F_NOCACHE_COPY; 5471 dev->hw_features |= NETIF_F_NOCACHE_COPY;
5465 if ((dev->features & NETIF_F_ALL_CSUM) && 5472 if ((dev->features & NETIF_F_ALL_CSUM) &&
5466 !(dev->features & NETIF_F_NO_CSUM)) { 5473 !(dev->features & NETIF_F_NO_CSUM)) {
5467 dev->wanted_features |= NETIF_F_NOCACHE_COPY; 5474 dev->wanted_features |= NETIF_F_NOCACHE_COPY;
5468 dev->features |= NETIF_F_NOCACHE_COPY; 5475 dev->features |= NETIF_F_NOCACHE_COPY;
5469 } 5476 }
5470 5477
5471 /* Enable GRO and NETIF_F_HIGHDMA for vlans by default, 5478 /* Enable GRO and NETIF_F_HIGHDMA for vlans by default,
5472 * vlan_dev_init() will do the dev->features check, so these features 5479 * vlan_dev_init() will do the dev->features check, so these features
5473 * are enabled only if supported by underlying device. 5480 * are enabled only if supported by underlying device.
5474 */ 5481 */
5475 dev->vlan_features |= (NETIF_F_GRO | NETIF_F_HIGHDMA); 5482 dev->vlan_features |= (NETIF_F_GRO | NETIF_F_HIGHDMA);
5476 5483
5477 ret = call_netdevice_notifiers(NETDEV_POST_INIT, dev); 5484 ret = call_netdevice_notifiers(NETDEV_POST_INIT, dev);
5478 ret = notifier_to_errno(ret); 5485 ret = notifier_to_errno(ret);
5479 if (ret) 5486 if (ret)
5480 goto err_uninit; 5487 goto err_uninit;
5481 5488
5482 ret = netdev_register_kobject(dev); 5489 ret = netdev_register_kobject(dev);
5483 if (ret) 5490 if (ret)
5484 goto err_uninit; 5491 goto err_uninit;
5485 dev->reg_state = NETREG_REGISTERED; 5492 dev->reg_state = NETREG_REGISTERED;
5486 5493
5487 __netdev_update_features(dev); 5494 __netdev_update_features(dev);
5488 5495
5489 /* 5496 /*
5490 * Default initial state at registry is that the 5497 * Default initial state at registry is that the
5491 * device is present. 5498 * device is present.
5492 */ 5499 */
5493 5500
5494 set_bit(__LINK_STATE_PRESENT, &dev->state); 5501 set_bit(__LINK_STATE_PRESENT, &dev->state);
5495 5502
5496 dev_init_scheduler(dev); 5503 dev_init_scheduler(dev);
5497 dev_hold(dev); 5504 dev_hold(dev);
5498 list_netdevice(dev); 5505 list_netdevice(dev);
5499 5506
5500 /* Notify protocols, that a new device appeared. */ 5507 /* Notify protocols, that a new device appeared. */
5501 ret = call_netdevice_notifiers(NETDEV_REGISTER, dev); 5508 ret = call_netdevice_notifiers(NETDEV_REGISTER, dev);
5502 ret = notifier_to_errno(ret); 5509 ret = notifier_to_errno(ret);
5503 if (ret) { 5510 if (ret) {
5504 rollback_registered(dev); 5511 rollback_registered(dev);
5505 dev->reg_state = NETREG_UNREGISTERED; 5512 dev->reg_state = NETREG_UNREGISTERED;
5506 } 5513 }
5507 /* 5514 /*
5508 * Prevent userspace races by waiting until the network 5515 * Prevent userspace races by waiting until the network
5509 * device is fully setup before sending notifications. 5516 * device is fully setup before sending notifications.
5510 */ 5517 */
5511 if (!dev->rtnl_link_ops || 5518 if (!dev->rtnl_link_ops ||
5512 dev->rtnl_link_state == RTNL_LINK_INITIALIZED) 5519 dev->rtnl_link_state == RTNL_LINK_INITIALIZED)
5513 rtmsg_ifinfo(RTM_NEWLINK, dev, ~0U); 5520 rtmsg_ifinfo(RTM_NEWLINK, dev, ~0U);
5514 5521
5515 out: 5522 out:
5516 return ret; 5523 return ret;
5517 5524
5518 err_uninit: 5525 err_uninit:
5519 if (dev->netdev_ops->ndo_uninit) 5526 if (dev->netdev_ops->ndo_uninit)
5520 dev->netdev_ops->ndo_uninit(dev); 5527 dev->netdev_ops->ndo_uninit(dev);
5521 goto out; 5528 goto out;
5522 } 5529 }
5523 EXPORT_SYMBOL(register_netdevice); 5530 EXPORT_SYMBOL(register_netdevice);
5524 5531
5525 /** 5532 /**
5526 * init_dummy_netdev - init a dummy network device for NAPI 5533 * init_dummy_netdev - init a dummy network device for NAPI
5527 * @dev: device to init 5534 * @dev: device to init
5528 * 5535 *
5529 * This takes a network device structure and initialize the minimum 5536 * This takes a network device structure and initialize the minimum
5530 * amount of fields so it can be used to schedule NAPI polls without 5537 * amount of fields so it can be used to schedule NAPI polls without
5531 * registering a full blown interface. This is to be used by drivers 5538 * registering a full blown interface. This is to be used by drivers
5532 * that need to tie several hardware interfaces to a single NAPI 5539 * that need to tie several hardware interfaces to a single NAPI
5533 * poll scheduler due to HW limitations. 5540 * poll scheduler due to HW limitations.
5534 */ 5541 */
5535 int init_dummy_netdev(struct net_device *dev) 5542 int init_dummy_netdev(struct net_device *dev)
5536 { 5543 {
5537 /* Clear everything. Note we don't initialize spinlocks 5544 /* Clear everything. Note we don't initialize spinlocks
5538 * are they aren't supposed to be taken by any of the 5545 * are they aren't supposed to be taken by any of the
5539 * NAPI code and this dummy netdev is supposed to be 5546 * NAPI code and this dummy netdev is supposed to be
5540 * only ever used for NAPI polls 5547 * only ever used for NAPI polls
5541 */ 5548 */
5542 memset(dev, 0, sizeof(struct net_device)); 5549 memset(dev, 0, sizeof(struct net_device));
5543 5550
5544 /* make sure we BUG if trying to hit standard 5551 /* make sure we BUG if trying to hit standard
5545 * register/unregister code path 5552 * register/unregister code path
5546 */ 5553 */
5547 dev->reg_state = NETREG_DUMMY; 5554 dev->reg_state = NETREG_DUMMY;
5548 5555
5549 /* NAPI wants this */ 5556 /* NAPI wants this */
5550 INIT_LIST_HEAD(&dev->napi_list); 5557 INIT_LIST_HEAD(&dev->napi_list);
5551 5558
5552 /* a dummy interface is started by default */ 5559 /* a dummy interface is started by default */
5553 set_bit(__LINK_STATE_PRESENT, &dev->state); 5560 set_bit(__LINK_STATE_PRESENT, &dev->state);
5554 set_bit(__LINK_STATE_START, &dev->state); 5561 set_bit(__LINK_STATE_START, &dev->state);
5555 5562
5556 /* Note : We dont allocate pcpu_refcnt for dummy devices, 5563 /* Note : We dont allocate pcpu_refcnt for dummy devices,
5557 * because users of this 'device' dont need to change 5564 * because users of this 'device' dont need to change
5558 * its refcount. 5565 * its refcount.
5559 */ 5566 */
5560 5567
5561 return 0; 5568 return 0;
5562 } 5569 }
5563 EXPORT_SYMBOL_GPL(init_dummy_netdev); 5570 EXPORT_SYMBOL_GPL(init_dummy_netdev);
5564 5571
5565 5572
5566 /** 5573 /**
5567 * register_netdev - register a network device 5574 * register_netdev - register a network device
5568 * @dev: device to register 5575 * @dev: device to register
5569 * 5576 *
5570 * Take a completed network device structure and add it to the kernel 5577 * Take a completed network device structure and add it to the kernel
5571 * interfaces. A %NETDEV_REGISTER message is sent to the netdev notifier 5578 * interfaces. A %NETDEV_REGISTER message is sent to the netdev notifier
5572 * chain. 0 is returned on success. A negative errno code is returned 5579 * chain. 0 is returned on success. A negative errno code is returned
5573 * on a failure to set up the device, or if the name is a duplicate. 5580 * on a failure to set up the device, or if the name is a duplicate.
5574 * 5581 *
5575 * This is a wrapper around register_netdevice that takes the rtnl semaphore 5582 * This is a wrapper around register_netdevice that takes the rtnl semaphore
5576 * and expands the device name if you passed a format string to 5583 * and expands the device name if you passed a format string to
5577 * alloc_netdev. 5584 * alloc_netdev.
5578 */ 5585 */
5579 int register_netdev(struct net_device *dev) 5586 int register_netdev(struct net_device *dev)
5580 { 5587 {
5581 int err; 5588 int err;
5582 5589
5583 rtnl_lock(); 5590 rtnl_lock();
5584 err = register_netdevice(dev); 5591 err = register_netdevice(dev);
5585 rtnl_unlock(); 5592 rtnl_unlock();
5586 return err; 5593 return err;
5587 } 5594 }
5588 EXPORT_SYMBOL(register_netdev); 5595 EXPORT_SYMBOL(register_netdev);
5589 5596
5590 int netdev_refcnt_read(const struct net_device *dev) 5597 int netdev_refcnt_read(const struct net_device *dev)
5591 { 5598 {
5592 int i, refcnt = 0; 5599 int i, refcnt = 0;
5593 5600
5594 for_each_possible_cpu(i) 5601 for_each_possible_cpu(i)
5595 refcnt += *per_cpu_ptr(dev->pcpu_refcnt, i); 5602 refcnt += *per_cpu_ptr(dev->pcpu_refcnt, i);
5596 return refcnt; 5603 return refcnt;
5597 } 5604 }
5598 EXPORT_SYMBOL(netdev_refcnt_read); 5605 EXPORT_SYMBOL(netdev_refcnt_read);
5599 5606
5600 /* 5607 /*
5601 * netdev_wait_allrefs - wait until all references are gone. 5608 * netdev_wait_allrefs - wait until all references are gone.
5602 * 5609 *
5603 * This is called when unregistering network devices. 5610 * This is called when unregistering network devices.
5604 * 5611 *
5605 * Any protocol or device that holds a reference should register 5612 * Any protocol or device that holds a reference should register
5606 * for netdevice notification, and cleanup and put back the 5613 * for netdevice notification, and cleanup and put back the
5607 * reference if they receive an UNREGISTER event. 5614 * reference if they receive an UNREGISTER event.
5608 * We can get stuck here if buggy protocols don't correctly 5615 * We can get stuck here if buggy protocols don't correctly
5609 * call dev_put. 5616 * call dev_put.
5610 */ 5617 */
5611 static void netdev_wait_allrefs(struct net_device *dev) 5618 static void netdev_wait_allrefs(struct net_device *dev)
5612 { 5619 {
5613 unsigned long rebroadcast_time, warning_time; 5620 unsigned long rebroadcast_time, warning_time;
5614 int refcnt; 5621 int refcnt;
5615 5622
5616 linkwatch_forget_dev(dev); 5623 linkwatch_forget_dev(dev);
5617 5624
5618 rebroadcast_time = warning_time = jiffies; 5625 rebroadcast_time = warning_time = jiffies;
5619 refcnt = netdev_refcnt_read(dev); 5626 refcnt = netdev_refcnt_read(dev);
5620 5627
5621 while (refcnt != 0) { 5628 while (refcnt != 0) {
5622 if (time_after(jiffies, rebroadcast_time + 1 * HZ)) { 5629 if (time_after(jiffies, rebroadcast_time + 1 * HZ)) {
5623 rtnl_lock(); 5630 rtnl_lock();
5624 5631
5625 /* Rebroadcast unregister notification */ 5632 /* Rebroadcast unregister notification */
5626 call_netdevice_notifiers(NETDEV_UNREGISTER, dev); 5633 call_netdevice_notifiers(NETDEV_UNREGISTER, dev);
5627 /* don't resend NETDEV_UNREGISTER_BATCH, _BATCH users 5634 /* don't resend NETDEV_UNREGISTER_BATCH, _BATCH users
5628 * should have already handle it the first time */ 5635 * should have already handle it the first time */
5629 5636
5630 if (test_bit(__LINK_STATE_LINKWATCH_PENDING, 5637 if (test_bit(__LINK_STATE_LINKWATCH_PENDING,
5631 &dev->state)) { 5638 &dev->state)) {
5632 /* We must not have linkwatch events 5639 /* We must not have linkwatch events
5633 * pending on unregister. If this 5640 * pending on unregister. If this
5634 * happens, we simply run the queue 5641 * happens, we simply run the queue
5635 * unscheduled, resulting in a noop 5642 * unscheduled, resulting in a noop
5636 * for this device. 5643 * for this device.
5637 */ 5644 */
5638 linkwatch_run_queue(); 5645 linkwatch_run_queue();
5639 } 5646 }
5640 5647
5641 __rtnl_unlock(); 5648 __rtnl_unlock();
5642 5649
5643 rebroadcast_time = jiffies; 5650 rebroadcast_time = jiffies;
5644 } 5651 }
5645 5652
5646 msleep(250); 5653 msleep(250);
5647 5654
5648 refcnt = netdev_refcnt_read(dev); 5655 refcnt = netdev_refcnt_read(dev);
5649 5656
5650 if (time_after(jiffies, warning_time + 10 * HZ)) { 5657 if (time_after(jiffies, warning_time + 10 * HZ)) {
5651 printk(KERN_EMERG "unregister_netdevice: " 5658 printk(KERN_EMERG "unregister_netdevice: "
5652 "waiting for %s to become free. Usage " 5659 "waiting for %s to become free. Usage "
5653 "count = %d\n", 5660 "count = %d\n",
5654 dev->name, refcnt); 5661 dev->name, refcnt);
5655 warning_time = jiffies; 5662 warning_time = jiffies;
5656 } 5663 }
5657 } 5664 }
5658 } 5665 }
5659 5666
5660 /* The sequence is: 5667 /* The sequence is:
5661 * 5668 *
5662 * rtnl_lock(); 5669 * rtnl_lock();
5663 * ... 5670 * ...
5664 * register_netdevice(x1); 5671 * register_netdevice(x1);
5665 * register_netdevice(x2); 5672 * register_netdevice(x2);
5666 * ... 5673 * ...
5667 * unregister_netdevice(y1); 5674 * unregister_netdevice(y1);
5668 * unregister_netdevice(y2); 5675 * unregister_netdevice(y2);
5669 * ... 5676 * ...
5670 * rtnl_unlock(); 5677 * rtnl_unlock();
5671 * free_netdev(y1); 5678 * free_netdev(y1);
5672 * free_netdev(y2); 5679 * free_netdev(y2);
5673 * 5680 *
5674 * We are invoked by rtnl_unlock(). 5681 * We are invoked by rtnl_unlock().
5675 * This allows us to deal with problems: 5682 * This allows us to deal with problems:
5676 * 1) We can delete sysfs objects which invoke hotplug 5683 * 1) We can delete sysfs objects which invoke hotplug
5677 * without deadlocking with linkwatch via keventd. 5684 * without deadlocking with linkwatch via keventd.
5678 * 2) Since we run with the RTNL semaphore not held, we can sleep 5685 * 2) Since we run with the RTNL semaphore not held, we can sleep
5679 * safely in order to wait for the netdev refcnt to drop to zero. 5686 * safely in order to wait for the netdev refcnt to drop to zero.
5680 * 5687 *
5681 * We must not return until all unregister events added during 5688 * We must not return until all unregister events added during
5682 * the interval the lock was held have been completed. 5689 * the interval the lock was held have been completed.
5683 */ 5690 */
5684 void netdev_run_todo(void) 5691 void netdev_run_todo(void)
5685 { 5692 {
5686 struct list_head list; 5693 struct list_head list;
5687 5694
5688 /* Snapshot list, allow later requests */ 5695 /* Snapshot list, allow later requests */
5689 list_replace_init(&net_todo_list, &list); 5696 list_replace_init(&net_todo_list, &list);
5690 5697
5691 __rtnl_unlock(); 5698 __rtnl_unlock();
5692 5699
5693 while (!list_empty(&list)) { 5700 while (!list_empty(&list)) {
5694 struct net_device *dev 5701 struct net_device *dev
5695 = list_first_entry(&list, struct net_device, todo_list); 5702 = list_first_entry(&list, struct net_device, todo_list);
5696 list_del(&dev->todo_list); 5703 list_del(&dev->todo_list);
5697 5704
5698 if (unlikely(dev->reg_state != NETREG_UNREGISTERING)) { 5705 if (unlikely(dev->reg_state != NETREG_UNREGISTERING)) {
5699 printk(KERN_ERR "network todo '%s' but state %d\n", 5706 printk(KERN_ERR "network todo '%s' but state %d\n",
5700 dev->name, dev->reg_state); 5707 dev->name, dev->reg_state);
5701 dump_stack(); 5708 dump_stack();
5702 continue; 5709 continue;
5703 } 5710 }
5704 5711
5705 dev->reg_state = NETREG_UNREGISTERED; 5712 dev->reg_state = NETREG_UNREGISTERED;
5706 5713
5707 on_each_cpu(flush_backlog, dev, 1); 5714 on_each_cpu(flush_backlog, dev, 1);
5708 5715
5709 netdev_wait_allrefs(dev); 5716 netdev_wait_allrefs(dev);
5710 5717
5711 /* paranoia */ 5718 /* paranoia */
5712 BUG_ON(netdev_refcnt_read(dev)); 5719 BUG_ON(netdev_refcnt_read(dev));
5713 WARN_ON(rcu_dereference_raw(dev->ip_ptr)); 5720 WARN_ON(rcu_dereference_raw(dev->ip_ptr));
5714 WARN_ON(rcu_dereference_raw(dev->ip6_ptr)); 5721 WARN_ON(rcu_dereference_raw(dev->ip6_ptr));
5715 WARN_ON(dev->dn_ptr); 5722 WARN_ON(dev->dn_ptr);
5716 5723
5717 if (dev->destructor) 5724 if (dev->destructor)
5718 dev->destructor(dev); 5725 dev->destructor(dev);
5719 5726
5720 /* Free network device */ 5727 /* Free network device */
5721 kobject_put(&dev->dev.kobj); 5728 kobject_put(&dev->dev.kobj);
5722 } 5729 }
5723 } 5730 }
5724 5731
5725 /* Convert net_device_stats to rtnl_link_stats64. They have the same 5732 /* Convert net_device_stats to rtnl_link_stats64. They have the same
5726 * fields in the same order, with only the type differing. 5733 * fields in the same order, with only the type differing.
5727 */ 5734 */
5728 static void netdev_stats_to_stats64(struct rtnl_link_stats64 *stats64, 5735 static void netdev_stats_to_stats64(struct rtnl_link_stats64 *stats64,
5729 const struct net_device_stats *netdev_stats) 5736 const struct net_device_stats *netdev_stats)
5730 { 5737 {
5731 #if BITS_PER_LONG == 64 5738 #if BITS_PER_LONG == 64
5732 BUILD_BUG_ON(sizeof(*stats64) != sizeof(*netdev_stats)); 5739 BUILD_BUG_ON(sizeof(*stats64) != sizeof(*netdev_stats));
5733 memcpy(stats64, netdev_stats, sizeof(*stats64)); 5740 memcpy(stats64, netdev_stats, sizeof(*stats64));
5734 #else 5741 #else
5735 size_t i, n = sizeof(*stats64) / sizeof(u64); 5742 size_t i, n = sizeof(*stats64) / sizeof(u64);
5736 const unsigned long *src = (const unsigned long *)netdev_stats; 5743 const unsigned long *src = (const unsigned long *)netdev_stats;
5737 u64 *dst = (u64 *)stats64; 5744 u64 *dst = (u64 *)stats64;
5738 5745
5739 BUILD_BUG_ON(sizeof(*netdev_stats) / sizeof(unsigned long) != 5746 BUILD_BUG_ON(sizeof(*netdev_stats) / sizeof(unsigned long) !=
5740 sizeof(*stats64) / sizeof(u64)); 5747 sizeof(*stats64) / sizeof(u64));
5741 for (i = 0; i < n; i++) 5748 for (i = 0; i < n; i++)
5742 dst[i] = src[i]; 5749 dst[i] = src[i];
5743 #endif 5750 #endif
5744 } 5751 }
5745 5752
5746 /** 5753 /**
5747 * dev_get_stats - get network device statistics 5754 * dev_get_stats - get network device statistics
5748 * @dev: device to get statistics from 5755 * @dev: device to get statistics from
5749 * @storage: place to store stats 5756 * @storage: place to store stats
5750 * 5757 *
5751 * Get network statistics from device. Return @storage. 5758 * Get network statistics from device. Return @storage.
5752 * The device driver may provide its own method by setting 5759 * The device driver may provide its own method by setting
5753 * dev->netdev_ops->get_stats64 or dev->netdev_ops->get_stats; 5760 * dev->netdev_ops->get_stats64 or dev->netdev_ops->get_stats;
5754 * otherwise the internal statistics structure is used. 5761 * otherwise the internal statistics structure is used.
5755 */ 5762 */
5756 struct rtnl_link_stats64 *dev_get_stats(struct net_device *dev, 5763 struct rtnl_link_stats64 *dev_get_stats(struct net_device *dev,
5757 struct rtnl_link_stats64 *storage) 5764 struct rtnl_link_stats64 *storage)
5758 { 5765 {
5759 const struct net_device_ops *ops = dev->netdev_ops; 5766 const struct net_device_ops *ops = dev->netdev_ops;
5760 5767
5761 if (ops->ndo_get_stats64) { 5768 if (ops->ndo_get_stats64) {
5762 memset(storage, 0, sizeof(*storage)); 5769 memset(storage, 0, sizeof(*storage));
5763 ops->ndo_get_stats64(dev, storage); 5770 ops->ndo_get_stats64(dev, storage);
5764 } else if (ops->ndo_get_stats) { 5771 } else if (ops->ndo_get_stats) {
5765 netdev_stats_to_stats64(storage, ops->ndo_get_stats(dev)); 5772 netdev_stats_to_stats64(storage, ops->ndo_get_stats(dev));
5766 } else { 5773 } else {
5767 netdev_stats_to_stats64(storage, &dev->stats); 5774 netdev_stats_to_stats64(storage, &dev->stats);
5768 } 5775 }
5769 storage->rx_dropped += atomic_long_read(&dev->rx_dropped); 5776 storage->rx_dropped += atomic_long_read(&dev->rx_dropped);
5770 return storage; 5777 return storage;
5771 } 5778 }
5772 EXPORT_SYMBOL(dev_get_stats); 5779 EXPORT_SYMBOL(dev_get_stats);
5773 5780
5774 struct netdev_queue *dev_ingress_queue_create(struct net_device *dev) 5781 struct netdev_queue *dev_ingress_queue_create(struct net_device *dev)
5775 { 5782 {
5776 struct netdev_queue *queue = dev_ingress_queue(dev); 5783 struct netdev_queue *queue = dev_ingress_queue(dev);
5777 5784
5778 #ifdef CONFIG_NET_CLS_ACT 5785 #ifdef CONFIG_NET_CLS_ACT
5779 if (queue) 5786 if (queue)
5780 return queue; 5787 return queue;
5781 queue = kzalloc(sizeof(*queue), GFP_KERNEL); 5788 queue = kzalloc(sizeof(*queue), GFP_KERNEL);
5782 if (!queue) 5789 if (!queue)
5783 return NULL; 5790 return NULL;
5784 netdev_init_one_queue(dev, queue, NULL); 5791 netdev_init_one_queue(dev, queue, NULL);
5785 queue->qdisc = &noop_qdisc; 5792 queue->qdisc = &noop_qdisc;
5786 queue->qdisc_sleeping = &noop_qdisc; 5793 queue->qdisc_sleeping = &noop_qdisc;
5787 rcu_assign_pointer(dev->ingress_queue, queue); 5794 rcu_assign_pointer(dev->ingress_queue, queue);
5788 #endif 5795 #endif
5789 return queue; 5796 return queue;
5790 } 5797 }
5791 5798
5792 /** 5799 /**
5793 * alloc_netdev_mqs - allocate network device 5800 * alloc_netdev_mqs - allocate network device
5794 * @sizeof_priv: size of private data to allocate space for 5801 * @sizeof_priv: size of private data to allocate space for
5795 * @name: device name format string 5802 * @name: device name format string
5796 * @setup: callback to initialize device 5803 * @setup: callback to initialize device
5797 * @txqs: the number of TX subqueues to allocate 5804 * @txqs: the number of TX subqueues to allocate
5798 * @rxqs: the number of RX subqueues to allocate 5805 * @rxqs: the number of RX subqueues to allocate
5799 * 5806 *
5800 * Allocates a struct net_device with private data area for driver use 5807 * Allocates a struct net_device with private data area for driver use
5801 * and performs basic initialization. Also allocates subquue structs 5808 * and performs basic initialization. Also allocates subquue structs
5802 * for each queue on the device. 5809 * for each queue on the device.
5803 */ 5810 */
5804 struct net_device *alloc_netdev_mqs(int sizeof_priv, const char *name, 5811 struct net_device *alloc_netdev_mqs(int sizeof_priv, const char *name,
5805 void (*setup)(struct net_device *), 5812 void (*setup)(struct net_device *),
5806 unsigned int txqs, unsigned int rxqs) 5813 unsigned int txqs, unsigned int rxqs)
5807 { 5814 {
5808 struct net_device *dev; 5815 struct net_device *dev;
5809 size_t alloc_size; 5816 size_t alloc_size;
5810 struct net_device *p; 5817 struct net_device *p;
5811 5818
5812 BUG_ON(strlen(name) >= sizeof(dev->name)); 5819 BUG_ON(strlen(name) >= sizeof(dev->name));
5813 5820
5814 if (txqs < 1) { 5821 if (txqs < 1) {
5815 pr_err("alloc_netdev: Unable to allocate device " 5822 pr_err("alloc_netdev: Unable to allocate device "
5816 "with zero queues.\n"); 5823 "with zero queues.\n");
5817 return NULL; 5824 return NULL;
5818 } 5825 }
5819 5826
5820 #ifdef CONFIG_RPS 5827 #ifdef CONFIG_RPS
5821 if (rxqs < 1) { 5828 if (rxqs < 1) {
5822 pr_err("alloc_netdev: Unable to allocate device " 5829 pr_err("alloc_netdev: Unable to allocate device "
5823 "with zero RX queues.\n"); 5830 "with zero RX queues.\n");
5824 return NULL; 5831 return NULL;
5825 } 5832 }
5826 #endif 5833 #endif
5827 5834
5828 alloc_size = sizeof(struct net_device); 5835 alloc_size = sizeof(struct net_device);
5829 if (sizeof_priv) { 5836 if (sizeof_priv) {
5830 /* ensure 32-byte alignment of private area */ 5837 /* ensure 32-byte alignment of private area */
5831 alloc_size = ALIGN(alloc_size, NETDEV_ALIGN); 5838 alloc_size = ALIGN(alloc_size, NETDEV_ALIGN);
5832 alloc_size += sizeof_priv; 5839 alloc_size += sizeof_priv;
5833 } 5840 }
5834 /* ensure 32-byte alignment of whole construct */ 5841 /* ensure 32-byte alignment of whole construct */
5835 alloc_size += NETDEV_ALIGN - 1; 5842 alloc_size += NETDEV_ALIGN - 1;
5836 5843
5837 p = kzalloc(alloc_size, GFP_KERNEL); 5844 p = kzalloc(alloc_size, GFP_KERNEL);
5838 if (!p) { 5845 if (!p) {
5839 printk(KERN_ERR "alloc_netdev: Unable to allocate device.\n"); 5846 printk(KERN_ERR "alloc_netdev: Unable to allocate device.\n");
5840 return NULL; 5847 return NULL;
5841 } 5848 }
5842 5849
5843 dev = PTR_ALIGN(p, NETDEV_ALIGN); 5850 dev = PTR_ALIGN(p, NETDEV_ALIGN);
5844 dev->padded = (char *)dev - (char *)p; 5851 dev->padded = (char *)dev - (char *)p;
5845 5852
5846 dev->pcpu_refcnt = alloc_percpu(int); 5853 dev->pcpu_refcnt = alloc_percpu(int);
5847 if (!dev->pcpu_refcnt) 5854 if (!dev->pcpu_refcnt)
5848 goto free_p; 5855 goto free_p;
5849 5856
5850 if (dev_addr_init(dev)) 5857 if (dev_addr_init(dev))
5851 goto free_pcpu; 5858 goto free_pcpu;
5852 5859
5853 dev_mc_init(dev); 5860 dev_mc_init(dev);
5854 dev_uc_init(dev); 5861 dev_uc_init(dev);
5855 5862
5856 dev_net_set(dev, &init_net); 5863 dev_net_set(dev, &init_net);
5857 5864
5858 dev->gso_max_size = GSO_MAX_SIZE; 5865 dev->gso_max_size = GSO_MAX_SIZE;
5859 5866
5860 INIT_LIST_HEAD(&dev->ethtool_ntuple_list.list); 5867 INIT_LIST_HEAD(&dev->ethtool_ntuple_list.list);
5861 dev->ethtool_ntuple_list.count = 0; 5868 dev->ethtool_ntuple_list.count = 0;
5862 INIT_LIST_HEAD(&dev->napi_list); 5869 INIT_LIST_HEAD(&dev->napi_list);
5863 INIT_LIST_HEAD(&dev->unreg_list); 5870 INIT_LIST_HEAD(&dev->unreg_list);
5864 INIT_LIST_HEAD(&dev->link_watch_list); 5871 INIT_LIST_HEAD(&dev->link_watch_list);
5865 dev->priv_flags = IFF_XMIT_DST_RELEASE; 5872 dev->priv_flags = IFF_XMIT_DST_RELEASE;
5866 setup(dev); 5873 setup(dev);
5867 5874
5868 dev->num_tx_queues = txqs; 5875 dev->num_tx_queues = txqs;
5869 dev->real_num_tx_queues = txqs; 5876 dev->real_num_tx_queues = txqs;
5870 if (netif_alloc_netdev_queues(dev)) 5877 if (netif_alloc_netdev_queues(dev))
5871 goto free_all; 5878 goto free_all;
5872 5879
5873 #ifdef CONFIG_RPS 5880 #ifdef CONFIG_RPS
5874 dev->num_rx_queues = rxqs; 5881 dev->num_rx_queues = rxqs;
5875 dev->real_num_rx_queues = rxqs; 5882 dev->real_num_rx_queues = rxqs;
5876 if (netif_alloc_rx_queues(dev)) 5883 if (netif_alloc_rx_queues(dev))
5877 goto free_all; 5884 goto free_all;
5878 #endif 5885 #endif
5879 5886
5880 strcpy(dev->name, name); 5887 strcpy(dev->name, name);
5881 dev->group = INIT_NETDEV_GROUP; 5888 dev->group = INIT_NETDEV_GROUP;
5882 return dev; 5889 return dev;
5883 5890
5884 free_all: 5891 free_all:
5885 free_netdev(dev); 5892 free_netdev(dev);
5886 return NULL; 5893 return NULL;
5887 5894
5888 free_pcpu: 5895 free_pcpu:
5889 free_percpu(dev->pcpu_refcnt); 5896 free_percpu(dev->pcpu_refcnt);
5890 kfree(dev->_tx); 5897 kfree(dev->_tx);
5891 #ifdef CONFIG_RPS 5898 #ifdef CONFIG_RPS
5892 kfree(dev->_rx); 5899 kfree(dev->_rx);
5893 #endif 5900 #endif
5894 5901
5895 free_p: 5902 free_p:
5896 kfree(p); 5903 kfree(p);
5897 return NULL; 5904 return NULL;
5898 } 5905 }
5899 EXPORT_SYMBOL(alloc_netdev_mqs); 5906 EXPORT_SYMBOL(alloc_netdev_mqs);
5900 5907
5901 /** 5908 /**
5902 * free_netdev - free network device 5909 * free_netdev - free network device
5903 * @dev: device 5910 * @dev: device
5904 * 5911 *
5905 * This function does the last stage of destroying an allocated device 5912 * This function does the last stage of destroying an allocated device
5906 * interface. The reference to the device object is released. 5913 * interface. The reference to the device object is released.
5907 * If this is the last reference then it will be freed. 5914 * If this is the last reference then it will be freed.
5908 */ 5915 */
5909 void free_netdev(struct net_device *dev) 5916 void free_netdev(struct net_device *dev)
5910 { 5917 {
5911 struct napi_struct *p, *n; 5918 struct napi_struct *p, *n;
5912 5919
5913 release_net(dev_net(dev)); 5920 release_net(dev_net(dev));
5914 5921
5915 kfree(dev->_tx); 5922 kfree(dev->_tx);
5916 #ifdef CONFIG_RPS 5923 #ifdef CONFIG_RPS
5917 kfree(dev->_rx); 5924 kfree(dev->_rx);
5918 #endif 5925 #endif
5919 5926
5920 kfree(rcu_dereference_raw(dev->ingress_queue)); 5927 kfree(rcu_dereference_raw(dev->ingress_queue));
5921 5928
5922 /* Flush device addresses */ 5929 /* Flush device addresses */
5923 dev_addr_flush(dev); 5930 dev_addr_flush(dev);
5924 5931
5925 /* Clear ethtool n-tuple list */ 5932 /* Clear ethtool n-tuple list */
5926 ethtool_ntuple_flush(dev); 5933 ethtool_ntuple_flush(dev);
5927 5934
5928 list_for_each_entry_safe(p, n, &dev->napi_list, dev_list) 5935 list_for_each_entry_safe(p, n, &dev->napi_list, dev_list)
5929 netif_napi_del(p); 5936 netif_napi_del(p);
5930 5937
5931 free_percpu(dev->pcpu_refcnt); 5938 free_percpu(dev->pcpu_refcnt);
5932 dev->pcpu_refcnt = NULL; 5939 dev->pcpu_refcnt = NULL;
5933 5940
5934 /* Compatibility with error handling in drivers */ 5941 /* Compatibility with error handling in drivers */
5935 if (dev->reg_state == NETREG_UNINITIALIZED) { 5942 if (dev->reg_state == NETREG_UNINITIALIZED) {
5936 kfree((char *)dev - dev->padded); 5943 kfree((char *)dev - dev->padded);
5937 return; 5944 return;
5938 } 5945 }
5939 5946
5940 BUG_ON(dev->reg_state != NETREG_UNREGISTERED); 5947 BUG_ON(dev->reg_state != NETREG_UNREGISTERED);
5941 dev->reg_state = NETREG_RELEASED; 5948 dev->reg_state = NETREG_RELEASED;
5942 5949
5943 /* will free via device release */ 5950 /* will free via device release */
5944 put_device(&dev->dev); 5951 put_device(&dev->dev);
5945 } 5952 }
5946 EXPORT_SYMBOL(free_netdev); 5953 EXPORT_SYMBOL(free_netdev);
5947 5954
5948 /** 5955 /**
5949 * synchronize_net - Synchronize with packet receive processing 5956 * synchronize_net - Synchronize with packet receive processing
5950 * 5957 *
5951 * Wait for packets currently being received to be done. 5958 * Wait for packets currently being received to be done.
5952 * Does not block later packets from starting. 5959 * Does not block later packets from starting.
5953 */ 5960 */
5954 void synchronize_net(void) 5961 void synchronize_net(void)
5955 { 5962 {
5956 might_sleep(); 5963 might_sleep();
5957 if (rtnl_is_locked()) 5964 if (rtnl_is_locked())
5958 synchronize_rcu_expedited(); 5965 synchronize_rcu_expedited();
5959 else 5966 else
5960 synchronize_rcu(); 5967 synchronize_rcu();
5961 } 5968 }
5962 EXPORT_SYMBOL(synchronize_net); 5969 EXPORT_SYMBOL(synchronize_net);
5963 5970
5964 /** 5971 /**
5965 * unregister_netdevice_queue - remove device from the kernel 5972 * unregister_netdevice_queue - remove device from the kernel
5966 * @dev: device 5973 * @dev: device
5967 * @head: list 5974 * @head: list
5968 * 5975 *
5969 * This function shuts down a device interface and removes it 5976 * This function shuts down a device interface and removes it
5970 * from the kernel tables. 5977 * from the kernel tables.
5971 * If head not NULL, device is queued to be unregistered later. 5978 * If head not NULL, device is queued to be unregistered later.
5972 * 5979 *
5973 * Callers must hold the rtnl semaphore. You may want 5980 * Callers must hold the rtnl semaphore. You may want
5974 * unregister_netdev() instead of this. 5981 * unregister_netdev() instead of this.
5975 */ 5982 */
5976 5983
5977 void unregister_netdevice_queue(struct net_device *dev, struct list_head *head) 5984 void unregister_netdevice_queue(struct net_device *dev, struct list_head *head)
5978 { 5985 {
5979 ASSERT_RTNL(); 5986 ASSERT_RTNL();
5980 5987
5981 if (head) { 5988 if (head) {
5982 list_move_tail(&dev->unreg_list, head); 5989 list_move_tail(&dev->unreg_list, head);
5983 } else { 5990 } else {
5984 rollback_registered(dev); 5991 rollback_registered(dev);
5985 /* Finish processing unregister after unlock */ 5992 /* Finish processing unregister after unlock */
5986 net_set_todo(dev); 5993 net_set_todo(dev);
5987 } 5994 }
5988 } 5995 }
5989 EXPORT_SYMBOL(unregister_netdevice_queue); 5996 EXPORT_SYMBOL(unregister_netdevice_queue);
5990 5997
5991 /** 5998 /**
5992 * unregister_netdevice_many - unregister many devices 5999 * unregister_netdevice_many - unregister many devices
5993 * @head: list of devices 6000 * @head: list of devices
5994 */ 6001 */
5995 void unregister_netdevice_many(struct list_head *head) 6002 void unregister_netdevice_many(struct list_head *head)
5996 { 6003 {
5997 struct net_device *dev; 6004 struct net_device *dev;
5998 6005
5999 if (!list_empty(head)) { 6006 if (!list_empty(head)) {
6000 rollback_registered_many(head); 6007 rollback_registered_many(head);
6001 list_for_each_entry(dev, head, unreg_list) 6008 list_for_each_entry(dev, head, unreg_list)
6002 net_set_todo(dev); 6009 net_set_todo(dev);
6003 } 6010 }
6004 } 6011 }
6005 EXPORT_SYMBOL(unregister_netdevice_many); 6012 EXPORT_SYMBOL(unregister_netdevice_many);
6006 6013
6007 /** 6014 /**
6008 * unregister_netdev - remove device from the kernel 6015 * unregister_netdev - remove device from the kernel
6009 * @dev: device 6016 * @dev: device
6010 * 6017 *
6011 * This function shuts down a device interface and removes it 6018 * This function shuts down a device interface and removes it
6012 * from the kernel tables. 6019 * from the kernel tables.
6013 * 6020 *
6014 * This is just a wrapper for unregister_netdevice that takes 6021 * This is just a wrapper for unregister_netdevice that takes
6015 * the rtnl semaphore. In general you want to use this and not 6022 * the rtnl semaphore. In general you want to use this and not
6016 * unregister_netdevice. 6023 * unregister_netdevice.
6017 */ 6024 */
6018 void unregister_netdev(struct net_device *dev) 6025 void unregister_netdev(struct net_device *dev)
6019 { 6026 {
6020 rtnl_lock(); 6027 rtnl_lock();
6021 unregister_netdevice(dev); 6028 unregister_netdevice(dev);
6022 rtnl_unlock(); 6029 rtnl_unlock();
6023 } 6030 }
6024 EXPORT_SYMBOL(unregister_netdev); 6031 EXPORT_SYMBOL(unregister_netdev);
6025 6032
6026 /** 6033 /**
6027 * dev_change_net_namespace - move device to different nethost namespace 6034 * dev_change_net_namespace - move device to different nethost namespace
6028 * @dev: device 6035 * @dev: device
6029 * @net: network namespace 6036 * @net: network namespace
6030 * @pat: If not NULL name pattern to try if the current device name 6037 * @pat: If not NULL name pattern to try if the current device name
6031 * is already taken in the destination network namespace. 6038 * is already taken in the destination network namespace.
6032 * 6039 *
6033 * This function shuts down a device interface and moves it 6040 * This function shuts down a device interface and moves it
6034 * to a new network namespace. On success 0 is returned, on 6041 * to a new network namespace. On success 0 is returned, on
6035 * a failure a netagive errno code is returned. 6042 * a failure a netagive errno code is returned.
6036 * 6043 *
6037 * Callers must hold the rtnl semaphore. 6044 * Callers must hold the rtnl semaphore.
6038 */ 6045 */
6039 6046
6040 int dev_change_net_namespace(struct net_device *dev, struct net *net, const char *pat) 6047 int dev_change_net_namespace(struct net_device *dev, struct net *net, const char *pat)
6041 { 6048 {
6042 int err; 6049 int err;
6043 6050
6044 ASSERT_RTNL(); 6051 ASSERT_RTNL();
6045 6052
6046 /* Don't allow namespace local devices to be moved. */ 6053 /* Don't allow namespace local devices to be moved. */
6047 err = -EINVAL; 6054 err = -EINVAL;
6048 if (dev->features & NETIF_F_NETNS_LOCAL) 6055 if (dev->features & NETIF_F_NETNS_LOCAL)
6049 goto out; 6056 goto out;
6050 6057
6051 /* Ensure the device has been registrered */ 6058 /* Ensure the device has been registrered */
6052 err = -EINVAL; 6059 err = -EINVAL;
6053 if (dev->reg_state != NETREG_REGISTERED) 6060 if (dev->reg_state != NETREG_REGISTERED)
6054 goto out; 6061 goto out;
6055 6062
6056 /* Get out if there is nothing todo */ 6063 /* Get out if there is nothing todo */
6057 err = 0; 6064 err = 0;
6058 if (net_eq(dev_net(dev), net)) 6065 if (net_eq(dev_net(dev), net))
6059 goto out; 6066 goto out;
6060 6067
6061 /* Pick the destination device name, and ensure 6068 /* Pick the destination device name, and ensure
6062 * we can use it in the destination network namespace. 6069 * we can use it in the destination network namespace.
6063 */ 6070 */
6064 err = -EEXIST; 6071 err = -EEXIST;
6065 if (__dev_get_by_name(net, dev->name)) { 6072 if (__dev_get_by_name(net, dev->name)) {
6066 /* We get here if we can't use the current device name */ 6073 /* We get here if we can't use the current device name */
6067 if (!pat) 6074 if (!pat)
6068 goto out; 6075 goto out;
6069 if (dev_get_valid_name(dev, pat) < 0) 6076 if (dev_get_valid_name(dev, pat) < 0)
6070 goto out; 6077 goto out;
6071 } 6078 }
6072 6079
6073 /* 6080 /*
6074 * And now a mini version of register_netdevice unregister_netdevice. 6081 * And now a mini version of register_netdevice unregister_netdevice.
6075 */ 6082 */
6076 6083
6077 /* If device is running close it first. */ 6084 /* If device is running close it first. */
6078 dev_close(dev); 6085 dev_close(dev);
6079 6086
6080 /* And unlink it from device chain */ 6087 /* And unlink it from device chain */
6081 err = -ENODEV; 6088 err = -ENODEV;
6082 unlist_netdevice(dev); 6089 unlist_netdevice(dev);
6083 6090
6084 synchronize_net(); 6091 synchronize_net();
6085 6092
6086 /* Shutdown queueing discipline. */ 6093 /* Shutdown queueing discipline. */
6087 dev_shutdown(dev); 6094 dev_shutdown(dev);
6088 6095
6089 /* Notify protocols, that we are about to destroy 6096 /* Notify protocols, that we are about to destroy
6090 this device. They should clean all the things. 6097 this device. They should clean all the things.
6091 6098
6092 Note that dev->reg_state stays at NETREG_REGISTERED. 6099 Note that dev->reg_state stays at NETREG_REGISTERED.
6093 This is wanted because this way 8021q and macvlan know 6100 This is wanted because this way 8021q and macvlan know
6094 the device is just moving and can keep their slaves up. 6101 the device is just moving and can keep their slaves up.
6095 */ 6102 */
6096 call_netdevice_notifiers(NETDEV_UNREGISTER, dev); 6103 call_netdevice_notifiers(NETDEV_UNREGISTER, dev);
6097 call_netdevice_notifiers(NETDEV_UNREGISTER_BATCH, dev); 6104 call_netdevice_notifiers(NETDEV_UNREGISTER_BATCH, dev);
6098 6105
6099 /* 6106 /*
6100 * Flush the unicast and multicast chains 6107 * Flush the unicast and multicast chains
6101 */ 6108 */
6102 dev_uc_flush(dev); 6109 dev_uc_flush(dev);
6103 dev_mc_flush(dev); 6110 dev_mc_flush(dev);
6104 6111
6105 /* Actually switch the network namespace */ 6112 /* Actually switch the network namespace */
6106 dev_net_set(dev, net); 6113 dev_net_set(dev, net);
6107 6114
6108 /* If there is an ifindex conflict assign a new one */ 6115 /* If there is an ifindex conflict assign a new one */
6109 if (__dev_get_by_index(net, dev->ifindex)) { 6116 if (__dev_get_by_index(net, dev->ifindex)) {
6110 int iflink = (dev->iflink == dev->ifindex); 6117 int iflink = (dev->iflink == dev->ifindex);
6111 dev->ifindex = dev_new_index(net); 6118 dev->ifindex = dev_new_index(net);
6112 if (iflink) 6119 if (iflink)
6113 dev->iflink = dev->ifindex; 6120 dev->iflink = dev->ifindex;
6114 } 6121 }
6115 6122
6116 /* Fixup kobjects */ 6123 /* Fixup kobjects */
6117 err = device_rename(&dev->dev, dev->name); 6124 err = device_rename(&dev->dev, dev->name);
6118 WARN_ON(err); 6125 WARN_ON(err);
6119 6126
6120 /* Add the device back in the hashes */ 6127 /* Add the device back in the hashes */
6121 list_netdevice(dev); 6128 list_netdevice(dev);
6122 6129
6123 /* Notify protocols, that a new device appeared. */ 6130 /* Notify protocols, that a new device appeared. */
6124 call_netdevice_notifiers(NETDEV_REGISTER, dev); 6131 call_netdevice_notifiers(NETDEV_REGISTER, dev);
6125 6132
6126 /* 6133 /*
6127 * Prevent userspace races by waiting until the network 6134 * Prevent userspace races by waiting until the network
6128 * device is fully setup before sending notifications. 6135 * device is fully setup before sending notifications.
6129 */ 6136 */
6130 rtmsg_ifinfo(RTM_NEWLINK, dev, ~0U); 6137 rtmsg_ifinfo(RTM_NEWLINK, dev, ~0U);
6131 6138
6132 synchronize_net(); 6139 synchronize_net();
6133 err = 0; 6140 err = 0;
6134 out: 6141 out:
6135 return err; 6142 return err;
6136 } 6143 }
6137 EXPORT_SYMBOL_GPL(dev_change_net_namespace); 6144 EXPORT_SYMBOL_GPL(dev_change_net_namespace);
6138 6145
6139 static int dev_cpu_callback(struct notifier_block *nfb, 6146 static int dev_cpu_callback(struct notifier_block *nfb,
6140 unsigned long action, 6147 unsigned long action,
6141 void *ocpu) 6148 void *ocpu)
6142 { 6149 {
6143 struct sk_buff **list_skb; 6150 struct sk_buff **list_skb;
6144 struct sk_buff *skb; 6151 struct sk_buff *skb;
6145 unsigned int cpu, oldcpu = (unsigned long)ocpu; 6152 unsigned int cpu, oldcpu = (unsigned long)ocpu;
6146 struct softnet_data *sd, *oldsd; 6153 struct softnet_data *sd, *oldsd;
6147 6154
6148 if (action != CPU_DEAD && action != CPU_DEAD_FROZEN) 6155 if (action != CPU_DEAD && action != CPU_DEAD_FROZEN)
6149 return NOTIFY_OK; 6156 return NOTIFY_OK;
6150 6157
6151 local_irq_disable(); 6158 local_irq_disable();
6152 cpu = smp_processor_id(); 6159 cpu = smp_processor_id();
6153 sd = &per_cpu(softnet_data, cpu); 6160 sd = &per_cpu(softnet_data, cpu);
6154 oldsd = &per_cpu(softnet_data, oldcpu); 6161 oldsd = &per_cpu(softnet_data, oldcpu);
6155 6162
6156 /* Find end of our completion_queue. */ 6163 /* Find end of our completion_queue. */
6157 list_skb = &sd->completion_queue; 6164 list_skb = &sd->completion_queue;
6158 while (*list_skb) 6165 while (*list_skb)
6159 list_skb = &(*list_skb)->next; 6166 list_skb = &(*list_skb)->next;
6160 /* Append completion queue from offline CPU. */ 6167 /* Append completion queue from offline CPU. */
6161 *list_skb = oldsd->completion_queue; 6168 *list_skb = oldsd->completion_queue;
6162 oldsd->completion_queue = NULL; 6169 oldsd->completion_queue = NULL;
6163 6170
6164 /* Append output queue from offline CPU. */ 6171 /* Append output queue from offline CPU. */
6165 if (oldsd->output_queue) { 6172 if (oldsd->output_queue) {
6166 *sd->output_queue_tailp = oldsd->output_queue; 6173 *sd->output_queue_tailp = oldsd->output_queue;
6167 sd->output_queue_tailp = oldsd->output_queue_tailp; 6174 sd->output_queue_tailp = oldsd->output_queue_tailp;
6168 oldsd->output_queue = NULL; 6175 oldsd->output_queue = NULL;
6169 oldsd->output_queue_tailp = &oldsd->output_queue; 6176 oldsd->output_queue_tailp = &oldsd->output_queue;
6170 } 6177 }
6171 6178
6172 raise_softirq_irqoff(NET_TX_SOFTIRQ); 6179 raise_softirq_irqoff(NET_TX_SOFTIRQ);
6173 local_irq_enable(); 6180 local_irq_enable();
6174 6181
6175 /* Process offline CPU's input_pkt_queue */ 6182 /* Process offline CPU's input_pkt_queue */
6176 while ((skb = __skb_dequeue(&oldsd->process_queue))) { 6183 while ((skb = __skb_dequeue(&oldsd->process_queue))) {
6177 netif_rx(skb); 6184 netif_rx(skb);
6178 input_queue_head_incr(oldsd); 6185 input_queue_head_incr(oldsd);
6179 } 6186 }
6180 while ((skb = __skb_dequeue(&oldsd->input_pkt_queue))) { 6187 while ((skb = __skb_dequeue(&oldsd->input_pkt_queue))) {
6181 netif_rx(skb); 6188 netif_rx(skb);
6182 input_queue_head_incr(oldsd); 6189 input_queue_head_incr(oldsd);
6183 } 6190 }
6184 6191
6185 return NOTIFY_OK; 6192 return NOTIFY_OK;
6186 } 6193 }
6187 6194
6188 6195
6189 /** 6196 /**
6190 * netdev_increment_features - increment feature set by one 6197 * netdev_increment_features - increment feature set by one
6191 * @all: current feature set 6198 * @all: current feature set
6192 * @one: new feature set 6199 * @one: new feature set
6193 * @mask: mask feature set 6200 * @mask: mask feature set
6194 * 6201 *
6195 * Computes a new feature set after adding a device with feature set 6202 * Computes a new feature set after adding a device with feature set
6196 * @one to the master device with current feature set @all. Will not 6203 * @one to the master device with current feature set @all. Will not
6197 * enable anything that is off in @mask. Returns the new feature set. 6204 * enable anything that is off in @mask. Returns the new feature set.
6198 */ 6205 */
6199 u32 netdev_increment_features(u32 all, u32 one, u32 mask) 6206 u32 netdev_increment_features(u32 all, u32 one, u32 mask)
6200 { 6207 {
6201 if (mask & NETIF_F_GEN_CSUM) 6208 if (mask & NETIF_F_GEN_CSUM)
6202 mask |= NETIF_F_ALL_CSUM; 6209 mask |= NETIF_F_ALL_CSUM;
6203 mask |= NETIF_F_VLAN_CHALLENGED; 6210 mask |= NETIF_F_VLAN_CHALLENGED;
6204 6211
6205 all |= one & (NETIF_F_ONE_FOR_ALL|NETIF_F_ALL_CSUM) & mask; 6212 all |= one & (NETIF_F_ONE_FOR_ALL|NETIF_F_ALL_CSUM) & mask;
6206 all &= one | ~NETIF_F_ALL_FOR_ALL; 6213 all &= one | ~NETIF_F_ALL_FOR_ALL;
6207 6214
6208 /* If device needs checksumming, downgrade to it. */ 6215 /* If device needs checksumming, downgrade to it. */
6209 if (all & (NETIF_F_ALL_CSUM & ~NETIF_F_NO_CSUM)) 6216 if (all & (NETIF_F_ALL_CSUM & ~NETIF_F_NO_CSUM))
6210 all &= ~NETIF_F_NO_CSUM; 6217 all &= ~NETIF_F_NO_CSUM;
6211 6218
6212 /* If one device supports hw checksumming, set for all. */ 6219 /* If one device supports hw checksumming, set for all. */
6213 if (all & NETIF_F_GEN_CSUM) 6220 if (all & NETIF_F_GEN_CSUM)
6214 all &= ~(NETIF_F_ALL_CSUM & ~NETIF_F_GEN_CSUM); 6221 all &= ~(NETIF_F_ALL_CSUM & ~NETIF_F_GEN_CSUM);
6215 6222
6216 return all; 6223 return all;
6217 } 6224 }
6218 EXPORT_SYMBOL(netdev_increment_features); 6225 EXPORT_SYMBOL(netdev_increment_features);
6219 6226
6220 static struct hlist_head *netdev_create_hash(void) 6227 static struct hlist_head *netdev_create_hash(void)
6221 { 6228 {
6222 int i; 6229 int i;
6223 struct hlist_head *hash; 6230 struct hlist_head *hash;
6224 6231
6225 hash = kmalloc(sizeof(*hash) * NETDEV_HASHENTRIES, GFP_KERNEL); 6232 hash = kmalloc(sizeof(*hash) * NETDEV_HASHENTRIES, GFP_KERNEL);
6226 if (hash != NULL) 6233 if (hash != NULL)
6227 for (i = 0; i < NETDEV_HASHENTRIES; i++) 6234 for (i = 0; i < NETDEV_HASHENTRIES; i++)
6228 INIT_HLIST_HEAD(&hash[i]); 6235 INIT_HLIST_HEAD(&hash[i]);
6229 6236
6230 return hash; 6237 return hash;
6231 } 6238 }
6232 6239
6233 /* Initialize per network namespace state */ 6240 /* Initialize per network namespace state */
6234 static int __net_init netdev_init(struct net *net) 6241 static int __net_init netdev_init(struct net *net)
6235 { 6242 {
6236 INIT_LIST_HEAD(&net->dev_base_head); 6243 INIT_LIST_HEAD(&net->dev_base_head);
6237 6244
6238 net->dev_name_head = netdev_create_hash(); 6245 net->dev_name_head = netdev_create_hash();
6239 if (net->dev_name_head == NULL) 6246 if (net->dev_name_head == NULL)
6240 goto err_name; 6247 goto err_name;
6241 6248
6242 net->dev_index_head = netdev_create_hash(); 6249 net->dev_index_head = netdev_create_hash();
6243 if (net->dev_index_head == NULL) 6250 if (net->dev_index_head == NULL)
6244 goto err_idx; 6251 goto err_idx;
6245 6252
6246 return 0; 6253 return 0;
6247 6254
6248 err_idx: 6255 err_idx:
6249 kfree(net->dev_name_head); 6256 kfree(net->dev_name_head);
6250 err_name: 6257 err_name:
6251 return -ENOMEM; 6258 return -ENOMEM;
6252 } 6259 }
6253 6260
6254 /** 6261 /**
6255 * netdev_drivername - network driver for the device 6262 * netdev_drivername - network driver for the device
6256 * @dev: network device 6263 * @dev: network device
6257 * @buffer: buffer for resulting name 6264 * @buffer: buffer for resulting name
6258 * @len: size of buffer 6265 * @len: size of buffer
6259 * 6266 *
6260 * Determine network driver for device. 6267 * Determine network driver for device.
6261 */ 6268 */
6262 char *netdev_drivername(const struct net_device *dev, char *buffer, int len) 6269 char *netdev_drivername(const struct net_device *dev, char *buffer, int len)
6263 { 6270 {
6264 const struct device_driver *driver; 6271 const struct device_driver *driver;
6265 const struct device *parent; 6272 const struct device *parent;
6266 6273
6267 if (len <= 0 || !buffer) 6274 if (len <= 0 || !buffer)
6268 return buffer; 6275 return buffer;
6269 buffer[0] = 0; 6276 buffer[0] = 0;
6270 6277
6271 parent = dev->dev.parent; 6278 parent = dev->dev.parent;
6272 6279
6273 if (!parent) 6280 if (!parent)
6274 return buffer; 6281 return buffer;
6275 6282
6276 driver = parent->driver; 6283 driver = parent->driver;
6277 if (driver && driver->name) 6284 if (driver && driver->name)
6278 strlcpy(buffer, driver->name, len); 6285 strlcpy(buffer, driver->name, len);
6279 return buffer; 6286 return buffer;
6280 } 6287 }
6281 6288
6282 static int __netdev_printk(const char *level, const struct net_device *dev, 6289 static int __netdev_printk(const char *level, const struct net_device *dev,
6283 struct va_format *vaf) 6290 struct va_format *vaf)
6284 { 6291 {
6285 int r; 6292 int r;
6286 6293
6287 if (dev && dev->dev.parent) 6294 if (dev && dev->dev.parent)
6288 r = dev_printk(level, dev->dev.parent, "%s: %pV", 6295 r = dev_printk(level, dev->dev.parent, "%s: %pV",
6289 netdev_name(dev), vaf); 6296 netdev_name(dev), vaf);
6290 else if (dev) 6297 else if (dev)
6291 r = printk("%s%s: %pV", level, netdev_name(dev), vaf); 6298 r = printk("%s%s: %pV", level, netdev_name(dev), vaf);
6292 else 6299 else
6293 r = printk("%s(NULL net_device): %pV", level, vaf); 6300 r = printk("%s(NULL net_device): %pV", level, vaf);
6294 6301
6295 return r; 6302 return r;
6296 } 6303 }
6297 6304
6298 int netdev_printk(const char *level, const struct net_device *dev, 6305 int netdev_printk(const char *level, const struct net_device *dev,
6299 const char *format, ...) 6306 const char *format, ...)
6300 { 6307 {
6301 struct va_format vaf; 6308 struct va_format vaf;
6302 va_list args; 6309 va_list args;
6303 int r; 6310 int r;
6304 6311
6305 va_start(args, format); 6312 va_start(args, format);
6306 6313
6307 vaf.fmt = format; 6314 vaf.fmt = format;
6308 vaf.va = &args; 6315 vaf.va = &args;
6309 6316
6310 r = __netdev_printk(level, dev, &vaf); 6317 r = __netdev_printk(level, dev, &vaf);
6311 va_end(args); 6318 va_end(args);
6312 6319
6313 return r; 6320 return r;
6314 } 6321 }
6315 EXPORT_SYMBOL(netdev_printk); 6322 EXPORT_SYMBOL(netdev_printk);
6316 6323
6317 #define define_netdev_printk_level(func, level) \ 6324 #define define_netdev_printk_level(func, level) \
6318 int func(const struct net_device *dev, const char *fmt, ...) \ 6325 int func(const struct net_device *dev, const char *fmt, ...) \
6319 { \ 6326 { \
6320 int r; \ 6327 int r; \
6321 struct va_format vaf; \ 6328 struct va_format vaf; \
6322 va_list args; \ 6329 va_list args; \
6323 \ 6330 \
6324 va_start(args, fmt); \ 6331 va_start(args, fmt); \
6325 \ 6332 \
6326 vaf.fmt = fmt; \ 6333 vaf.fmt = fmt; \
6327 vaf.va = &args; \ 6334 vaf.va = &args; \
6328 \ 6335 \
6329 r = __netdev_printk(level, dev, &vaf); \ 6336 r = __netdev_printk(level, dev, &vaf); \
6330 va_end(args); \ 6337 va_end(args); \
6331 \ 6338 \
6332 return r; \ 6339 return r; \
6333 } \ 6340 } \
6334 EXPORT_SYMBOL(func); 6341 EXPORT_SYMBOL(func);
6335 6342
6336 define_netdev_printk_level(netdev_emerg, KERN_EMERG); 6343 define_netdev_printk_level(netdev_emerg, KERN_EMERG);
6337 define_netdev_printk_level(netdev_alert, KERN_ALERT); 6344 define_netdev_printk_level(netdev_alert, KERN_ALERT);
6338 define_netdev_printk_level(netdev_crit, KERN_CRIT); 6345 define_netdev_printk_level(netdev_crit, KERN_CRIT);
6339 define_netdev_printk_level(netdev_err, KERN_ERR); 6346 define_netdev_printk_level(netdev_err, KERN_ERR);
6340 define_netdev_printk_level(netdev_warn, KERN_WARNING); 6347 define_netdev_printk_level(netdev_warn, KERN_WARNING);
6341 define_netdev_printk_level(netdev_notice, KERN_NOTICE); 6348 define_netdev_printk_level(netdev_notice, KERN_NOTICE);
6342 define_netdev_printk_level(netdev_info, KERN_INFO); 6349 define_netdev_printk_level(netdev_info, KERN_INFO);
6343 6350
6344 static void __net_exit netdev_exit(struct net *net) 6351 static void __net_exit netdev_exit(struct net *net)
6345 { 6352 {
6346 kfree(net->dev_name_head); 6353 kfree(net->dev_name_head);
6347 kfree(net->dev_index_head); 6354 kfree(net->dev_index_head);
6348 } 6355 }
6349 6356
6350 static struct pernet_operations __net_initdata netdev_net_ops = { 6357 static struct pernet_operations __net_initdata netdev_net_ops = {
6351 .init = netdev_init, 6358 .init = netdev_init,
6352 .exit = netdev_exit, 6359 .exit = netdev_exit,
6353 }; 6360 };
6354 6361
6355 static void __net_exit default_device_exit(struct net *net) 6362 static void __net_exit default_device_exit(struct net *net)
6356 { 6363 {
6357 struct net_device *dev, *aux; 6364 struct net_device *dev, *aux;
6358 /* 6365 /*
6359 * Push all migratable network devices back to the 6366 * Push all migratable network devices back to the
6360 * initial network namespace 6367 * initial network namespace
6361 */ 6368 */
6362 rtnl_lock(); 6369 rtnl_lock();
6363 for_each_netdev_safe(net, dev, aux) { 6370 for_each_netdev_safe(net, dev, aux) {
6364 int err; 6371 int err;
6365 char fb_name[IFNAMSIZ]; 6372 char fb_name[IFNAMSIZ];
6366 6373
6367 /* Ignore unmoveable devices (i.e. loopback) */ 6374 /* Ignore unmoveable devices (i.e. loopback) */
6368 if (dev->features & NETIF_F_NETNS_LOCAL) 6375 if (dev->features & NETIF_F_NETNS_LOCAL)
6369 continue; 6376 continue;
6370 6377
6371 /* Leave virtual devices for the generic cleanup */ 6378 /* Leave virtual devices for the generic cleanup */
6372 if (dev->rtnl_link_ops) 6379 if (dev->rtnl_link_ops)
6373 continue; 6380 continue;
6374 6381
6375 /* Push remaining network devices to init_net */ 6382 /* Push remaining network devices to init_net */
6376 snprintf(fb_name, IFNAMSIZ, "dev%d", dev->ifindex); 6383 snprintf(fb_name, IFNAMSIZ, "dev%d", dev->ifindex);
6377 err = dev_change_net_namespace(dev, &init_net, fb_name); 6384 err = dev_change_net_namespace(dev, &init_net, fb_name);
6378 if (err) { 6385 if (err) {
6379 printk(KERN_EMERG "%s: failed to move %s to init_net: %d\n", 6386 printk(KERN_EMERG "%s: failed to move %s to init_net: %d\n",
6380 __func__, dev->name, err); 6387 __func__, dev->name, err);
6381 BUG(); 6388 BUG();
6382 } 6389 }
6383 } 6390 }
6384 rtnl_unlock(); 6391 rtnl_unlock();
6385 } 6392 }
6386 6393
6387 static void __net_exit default_device_exit_batch(struct list_head *net_list) 6394 static void __net_exit default_device_exit_batch(struct list_head *net_list)
6388 { 6395 {
6389 /* At exit all network devices most be removed from a network 6396 /* At exit all network devices most be removed from a network
6390 * namespace. Do this in the reverse order of registration. 6397 * namespace. Do this in the reverse order of registration.
6391 * Do this across as many network namespaces as possible to 6398 * Do this across as many network namespaces as possible to
6392 * improve batching efficiency. 6399 * improve batching efficiency.
6393 */ 6400 */
6394 struct net_device *dev; 6401 struct net_device *dev;
6395 struct net *net; 6402 struct net *net;
6396 LIST_HEAD(dev_kill_list); 6403 LIST_HEAD(dev_kill_list);
6397 6404
6398 rtnl_lock(); 6405 rtnl_lock();
6399 list_for_each_entry(net, net_list, exit_list) { 6406 list_for_each_entry(net, net_list, exit_list) {
6400 for_each_netdev_reverse(net, dev) { 6407 for_each_netdev_reverse(net, dev) {
6401 if (dev->rtnl_link_ops) 6408 if (dev->rtnl_link_ops)
6402 dev->rtnl_link_ops->dellink(dev, &dev_kill_list); 6409 dev->rtnl_link_ops->dellink(dev, &dev_kill_list);
6403 else 6410 else
6404 unregister_netdevice_queue(dev, &dev_kill_list); 6411 unregister_netdevice_queue(dev, &dev_kill_list);
6405 } 6412 }
6406 } 6413 }
6407 unregister_netdevice_many(&dev_kill_list); 6414 unregister_netdevice_many(&dev_kill_list);
6408 list_del(&dev_kill_list); 6415 list_del(&dev_kill_list);
6409 rtnl_unlock(); 6416 rtnl_unlock();
6410 } 6417 }
6411 6418
6412 static struct pernet_operations __net_initdata default_device_ops = { 6419 static struct pernet_operations __net_initdata default_device_ops = {
6413 .exit = default_device_exit, 6420 .exit = default_device_exit,
6414 .exit_batch = default_device_exit_batch, 6421 .exit_batch = default_device_exit_batch,
6415 }; 6422 };
6416 6423
6417 /* 6424 /*
6418 * Initialize the DEV module. At boot time this walks the device list and 6425 * Initialize the DEV module. At boot time this walks the device list and
6419 * unhooks any devices that fail to initialise (normally hardware not 6426 * unhooks any devices that fail to initialise (normally hardware not
6420 * present) and leaves us with a valid list of present and active devices. 6427 * present) and leaves us with a valid list of present and active devices.
6421 * 6428 *
6422 */ 6429 */
6423 6430
6424 /* 6431 /*
6425 * This is called single threaded during boot, so no need 6432 * This is called single threaded during boot, so no need
6426 * to take the rtnl semaphore. 6433 * to take the rtnl semaphore.
6427 */ 6434 */
6428 static int __init net_dev_init(void) 6435 static int __init net_dev_init(void)
6429 { 6436 {
6430 int i, rc = -ENOMEM; 6437 int i, rc = -ENOMEM;
6431 6438
6432 BUG_ON(!dev_boot_phase); 6439 BUG_ON(!dev_boot_phase);
6433 6440
6434 if (dev_proc_init()) 6441 if (dev_proc_init())
6435 goto out; 6442 goto out;
6436 6443
6437 if (netdev_kobject_init()) 6444 if (netdev_kobject_init())
6438 goto out; 6445 goto out;
6439 6446
6440 INIT_LIST_HEAD(&ptype_all); 6447 INIT_LIST_HEAD(&ptype_all);
6441 for (i = 0; i < PTYPE_HASH_SIZE; i++) 6448 for (i = 0; i < PTYPE_HASH_SIZE; i++)
6442 INIT_LIST_HEAD(&ptype_base[i]); 6449 INIT_LIST_HEAD(&ptype_base[i]);
6443 6450
6444 if (register_pernet_subsys(&netdev_net_ops)) 6451 if (register_pernet_subsys(&netdev_net_ops))
6445 goto out; 6452 goto out;
6446 6453
6447 /* 6454 /*
6448 * Initialise the packet receive queues. 6455 * Initialise the packet receive queues.
6449 */ 6456 */
6450 6457
6451 for_each_possible_cpu(i) { 6458 for_each_possible_cpu(i) {
6452 struct softnet_data *sd = &per_cpu(softnet_data, i); 6459 struct softnet_data *sd = &per_cpu(softnet_data, i);
6453 6460
6454 memset(sd, 0, sizeof(*sd)); 6461 memset(sd, 0, sizeof(*sd));
6455 skb_queue_head_init(&sd->input_pkt_queue); 6462 skb_queue_head_init(&sd->input_pkt_queue);
6456 skb_queue_head_init(&sd->process_queue); 6463 skb_queue_head_init(&sd->process_queue);
6457 sd->completion_queue = NULL; 6464 sd->completion_queue = NULL;
6458 INIT_LIST_HEAD(&sd->poll_list); 6465 INIT_LIST_HEAD(&sd->poll_list);
6459 sd->output_queue = NULL; 6466 sd->output_queue = NULL;
6460 sd->output_queue_tailp = &sd->output_queue; 6467 sd->output_queue_tailp = &sd->output_queue;
6461 #ifdef CONFIG_RPS 6468 #ifdef CONFIG_RPS
6462 sd->csd.func = rps_trigger_softirq; 6469 sd->csd.func = rps_trigger_softirq;
6463 sd->csd.info = sd; 6470 sd->csd.info = sd;
6464 sd->csd.flags = 0; 6471 sd->csd.flags = 0;
6465 sd->cpu = i; 6472 sd->cpu = i;
6466 #endif 6473 #endif
6467 6474
6468 sd->backlog.poll = process_backlog; 6475 sd->backlog.poll = process_backlog;
6469 sd->backlog.weight = weight_p; 6476 sd->backlog.weight = weight_p;
6470 sd->backlog.gro_list = NULL; 6477 sd->backlog.gro_list = NULL;
6471 sd->backlog.gro_count = 0; 6478 sd->backlog.gro_count = 0;
6472 } 6479 }
6473 6480
6474 dev_boot_phase = 0; 6481 dev_boot_phase = 0;
6475 6482
6476 /* The loopback device is special if any other network devices 6483 /* The loopback device is special if any other network devices
6477 * is present in a network namespace the loopback device must 6484 * is present in a network namespace the loopback device must
6478 * be present. Since we now dynamically allocate and free the 6485 * be present. Since we now dynamically allocate and free the
6479 * loopback device ensure this invariant is maintained by 6486 * loopback device ensure this invariant is maintained by
6480 * keeping the loopback device as the first device on the 6487 * keeping the loopback device as the first device on the
6481 * list of network devices. Ensuring the loopback devices 6488 * list of network devices. Ensuring the loopback devices
6482 * is the first device that appears and the last network device 6489 * is the first device that appears and the last network device
6483 * that disappears. 6490 * that disappears.
6484 */ 6491 */
6485 if (register_pernet_device(&loopback_net_ops)) 6492 if (register_pernet_device(&loopback_net_ops))
6486 goto out; 6493 goto out;
6487 6494
6488 if (register_pernet_device(&default_device_ops)) 6495 if (register_pernet_device(&default_device_ops))
6489 goto out; 6496 goto out;
6490 6497
6491 open_softirq(NET_TX_SOFTIRQ, net_tx_action); 6498 open_softirq(NET_TX_SOFTIRQ, net_tx_action);
6492 open_softirq(NET_RX_SOFTIRQ, net_rx_action); 6499 open_softirq(NET_RX_SOFTIRQ, net_rx_action);
6493 6500
6494 hotcpu_notifier(dev_cpu_callback, 0); 6501 hotcpu_notifier(dev_cpu_callback, 0);
6495 dst_init(); 6502 dst_init();
6496 dev_mcast_init(); 6503 dev_mcast_init();
6497 rc = 0; 6504 rc = 0;
6498 out: 6505 out:
6499 return rc; 6506 return rc;
6500 } 6507 }
6501 6508
6502 subsys_initcall(net_dev_init); 6509 subsys_initcall(net_dev_init);
6503 6510
6504 static int __init initialize_hashrnd(void) 6511 static int __init initialize_hashrnd(void)
6505 { 6512 {
6506 get_random_bytes(&hashrnd, sizeof(hashrnd)); 6513 get_random_bytes(&hashrnd, sizeof(hashrnd));
6507 return 0; 6514 return 0;
6508 } 6515 }
6509 6516
6510 late_initcall_sync(initialize_hashrnd); 6517 late_initcall_sync(initialize_hashrnd);
6511 6518
6512 6519