Ethernet switch device driver model (switchdev)
  ===============================================
  Copyright (c) 2014 Jiri Pirko <jiri@resnulli.us>
  Copyright (c) 2014-2015 Scott Feldman <sfeldma@gmail.com>
  
  
  The Ethernet switch device driver model (switchdev) is an in-kernel driver
  model for switch devices which offload the forwarding (data) plane from the
  kernel.
  
  Figure 1 is a block diagram showing the components of the switchdev model for
  an example setup using a data-center-class switch ASIC chip.  Other setups
  with SR-IOV or soft switches, such as OVS, are possible.

                               User-space tools
  
         user space                   |
        +-------------------------------------------------------------------+
         kernel                       | Netlink
                                      |
                       +--------------+-------------------------------+
                       |         Network stack                        |
                       |           (Linux)                            |
                       |                                              |
                       +----------------------------------------------+

                             sw1p2     sw1p4     sw1p6

                        sw1p1  +  sw1p3  +  sw1p5  +          eth1
                          +    |    +    |    +    |            +
                          |    |    |    |    |    |            |
                       +--+----+----+----+----+----+---+  +-----+-----+
                       |         Switch driver         |  |    mgmt   |
                       |        (this document)        |  |   driver  |
                       |                               |  |           |
                       +--------------+----------------+  +-----------+
                                      |
         kernel                       | HW bus (eg PCI)
        +-------------------------------------------------------------------+
         hardware                     |
                       +--------------+----------------+
                       |         Switch device (sw1)   |
                       |  +----+                       +--------+
                       |  |    v offloaded data path   | mgmt port
                       |  |    |                       |
                       +--|----|----+----+----+----+---+
                          |    |    |    |    |    |
                          +    +    +    +    +    +
                         p1   p2   p3   p4   p5   p6
  
                               front-panel ports

  
                                      Fig 1.
  
  
  Include Files
  -------------
  
  #include <linux/netdevice.h>
  #include <net/switchdev.h>
  
  
  Configuration
  -------------
  
  Use "depends NET_SWITCHDEV" in driver's Kconfig to ensure switchdev model
  support is built for driver.
  
  
  Switch Ports
  ------------
  
  On switchdev driver initialization, the driver will allocate and register a
  struct net_device (using register_netdev()) for each enumerated physical switch
  port, called the port netdev.  A port netdev is the software representation of
  the physical port and provides a conduit for control traffic to/from the
  controller (the kernel) and the network, as well as an anchor point for higher
  level constructs such as bridges, bonds, VLANs, tunnels, and L3 routers.  Using
  standard netdev tools (iproute2, ethtool, etc), the port netdev can also
  provide to the user access to the physical properties of the switch port such
  as PHY link state and I/O statistics.
  
  There is (currently) no higher-level kernel object for the switch beyond the
  port netdevs.  All of the switchdev driver ops are netdev ops or switchdev ops.
  
  A switch management port is outside the scope of the switchdev driver model.
  Typically, the management port is not participating in offloaded data plane and
  is loaded with a different driver, such as a NIC driver, on the management port
  device.

  Switch ID
  ^^^^^^^^^
  
  The switchdev driver must implement the switchdev op switchdev_port_attr_get
  for SWITCHDEV_ATTR_ID_PORT_PARENT_ID for each port netdev, returning the same
  physical ID for each port of a switch.  The ID must be unique between switches
  on the same system.  The ID does not need to be unique between switches on
  different systems.
  
  The switch ID is used to locate ports on a switch and to know if aggregated
  ports belong to the same switch.

  Port Netdev Naming
  ^^^^^^^^^^^^^^^^^^
  
  Udev rules should be used for port netdev naming, using some unique attribute
  of the port as a key, for example the port MAC address or the port PHYS name.
  Hard-coding of kernel netdev names within the driver is discouraged; let the
  kernel pick the default netdev name, and let udev set the final name based on a
  port attribute.
  
  Using port PHYS name (ndo_get_phys_port_name) for the key is particularly

  useful for dynamically-named ports where the device names its ports based on

  external configuration.  For example, if a physical 40G port is split logically
  into 4 10G ports, resulting in 4 port netdevs, the device can give a unique
  name for each port using port PHYS name.  The udev rule would be:

  SUBSYSTEM=="net", ACTION=="add", ATTR{phys_switch_id}=="<phys_switch_id>", \
  	ATTR{phys_port_name}!="", NAME="swX$attr{phys_port_name}"

  
  Suggested naming convention is "swXpYsZ", where X is the switch name or ID, Y
  is the port name or ID, and Z is the sub-port name or ID.  For example, sw1p1s0
  would be sub-port 0 on port 1 on switch 1.

  Port Features
  ^^^^^^^^^^^^^
  
  NETIF_F_NETNS_LOCAL
  
  If the switchdev driver (and device) only supports offloading of the default
  network namespace (netns), the driver should set this feature flag to prevent
  the port netdev from being moved out of the default netns.  A netns-aware

  driver/device would not set this flag and be responsible for partitioning

  hardware to preserve netns containment.  This means hardware cannot forward
  traffic from a port in one namespace to another port in another namespace.
  
  Port Topology
  ^^^^^^^^^^^^^
  
  The port netdevs representing the physical switch ports can be organized into
  higher-level switching constructs.  The default construct is a standalone
  router port, used to offload L3 forwarding.  Two or more ports can be bonded
  together to form a LAG.  Two or more ports (or LAGs) can be bridged to bridge

  L2 networks.  VLANs can be applied to sub-divide L2 networks.  L2-over-L3

  tunnels can be built on ports.  These constructs are built using standard Linux
  tools such as the bridge driver, the bonding/team drivers, and netlink-based
  tools such as iproute2.
  
  The switchdev driver can know a particular port's position in the topology by
  monitoring NETDEV_CHANGEUPPER notifications.  For example, a port moved into a
  bond will see it's upper master change.  If that bond is moved into a bridge,
  the bond's upper master will change.  And so on.  The driver will track such
  movements to know what position a port is in in the overall topology by
  registering for netdevice events and acting on NETDEV_CHANGEUPPER.
  
  L2 Forwarding Offload
  ---------------------
  
  The idea is to offload the L2 data forwarding (switching) path from the kernel
  to the switchdev device by mirroring bridge FDB entries down to the device.  An
  FDB entry is the {port, MAC, VLAN} tuple forwarding destination.
  
  To offloading L2 bridging, the switchdev driver/device should support:
  
  	- Static FDB entries installed on a bridge port
  	- Notification of learned/forgotten src mac/vlans from device
  	- STP state changes on the port
  	- VLAN flooding of multicast/broadcast and unknown unicast packets
  
  Static FDB Entries
  ^^^^^^^^^^^^^^^^^^
  
  The switchdev driver should implement ndo_fdb_add, ndo_fdb_del and ndo_fdb_dump
  to support static FDB entries installed to the device.  Static bridge FDB
  entries are installed, for example, using iproute2 bridge cmd:
  
  	bridge fdb add ADDR dev DEV [vlan VID] [self]

  The driver should use the helper switchdev_port_fdb_xxx ops for ndo_fdb_xxx

  ops, and handle add/delete/dump of SWITCHDEV_OBJ_ID_PORT_FDB object using

  switchdev_port_obj_xxx ops.

  XXX: what should be done if offloading this rule to hardware fails (for
  example, due to full capacity in hardware tables) ?

  Note: by default, the bridge does not filter on VLAN and only bridges untagged
  traffic.  To enable VLAN support, turn on VLAN filtering:
  
  	echo 1 >/sys/class/net/<bridge>/bridge/vlan_filtering
  
  Notification of Learned/Forgotten Source MAC/VLANs
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  
  The switch device will learn/forget source MAC address/VLAN on ingress packets
  and notify the switch driver of the mac/vlan/port tuples.  The switch driver,
  in turn, will notify the bridge driver using the switchdev notifier call:
  
  	err = call_switchdev_notifiers(val, dev, info);

  Where val is SWITCHDEV_FDB_ADD when learning and SWITCHDEV_FDB_DEL when
  forgetting, and info points to a struct switchdev_notifier_fdb_info.  On
  SWITCHDEV_FDB_ADD, the bridge driver will install the FDB entry into the
  bridge's FDB and mark the entry as NTF_EXT_LEARNED.  The iproute2 bridge
  command will label these entries "offload":

  
  	$ bridge fdb
  	52:54:00:12:35:01 dev sw1p1 master br0 permanent
  	00:02:00:00:02:00 dev sw1p1 master br0 offload
  	00:02:00:00:02:00 dev sw1p1 self
  	52:54:00:12:35:02 dev sw1p2 master br0 permanent
  	00:02:00:00:03:00 dev sw1p2 master br0 offload
  	00:02:00:00:03:00 dev sw1p2 self
  	33:33:00:00:00:01 dev eth0 self permanent
  	01:00:5e:00:00:01 dev eth0 self permanent
  	33:33:ff:00:00:00 dev eth0 self permanent
  	01:80:c2:00:00:0e dev eth0 self permanent
  	33:33:00:00:00:01 dev br0 self permanent
  	01:00:5e:00:00:01 dev br0 self permanent
  	33:33:ff:12:35:01 dev br0 self permanent
  
  Learning on the port should be disabled on the bridge using the bridge command:
  
  	bridge link set dev DEV learning off
  
  Learning on the device port should be enabled, as well as learning_sync:
  
  	bridge link set dev DEV learning on self
  	bridge link set dev DEV learning_sync on self

  Learning_sync attribute enables syncing of the learned/forgotten FDB entry to

  the bridge's FDB.  It's possible, but not optimal, to enable learning on the
  device port and on the bridge port, and disable learning_sync.
  
  To support learning and learning_sync port attributes, the driver implements

  switchdev op switchdev_port_attr_get/set for
  SWITCHDEV_ATTR_PORT_ID_BRIDGE_FLAGS. The driver should initialize the attributes
  to the hardware defaults.

  
  FDB Ageing
  ^^^^^^^^^^

  The bridge will skip ageing FDB entries marked with NTF_EXT_LEARNED and it is
  the responsibility of the port driver/device to age out these entries.  If the
  port device supports ageing, when the FDB entry expires, it will notify the
  driver which in turn will notify the bridge with SWITCHDEV_FDB_DEL.  If the
  device does not support ageing, the driver can simulate ageing using a

  garbage collection timer to monitor FDB entries.  Expired entries will be

  notified to the bridge using SWITCHDEV_FDB_DEL.  See rocker driver for
  example of driver running ageing timer.
  
  To keep an NTF_EXT_LEARNED entry "alive", the driver should refresh the FDB
  entry by calling call_switchdev_notifiers(SWITCHDEV_FDB_ADD, ...).  The

  notification will reset the FDB entry's last-used time to now.  The driver
  should rate limit refresh notifications, for example, no more than once a

  second.  (The last-used time is visible using the bridge -s fdb option).

  
  STP State Change on Port
  ^^^^^^^^^^^^^^^^^^^^^^^^
  
  Internally or with a third-party STP protocol implementation (e.g. mstpd), the
  bridge driver maintains the STP state for ports, and will notify the switch

  driver of STP state change on a port using the switchdev op

  switchdev_attr_port_set for SWITCHDEV_ATTR_PORT_ID_STP_UPDATE.

  
  State is one of BR_STATE_*.  The switch driver can use STP state updates to
  update ingress packet filter list for the port.  For example, if port is
  DISABLED, no packets should pass, but if port moves to BLOCKED, then STP BPDUs
  and other IEEE 01:80:c2:xx:xx:xx link-local multicast packets can pass.
  
  Note that STP BDPUs are untagged and STP state applies to all VLANs on the port
  so packet filters should be applied consistently across untagged and tagged
  VLANs on the port.
  
  Flooding L2 domain
  ^^^^^^^^^^^^^^^^^^
  
  For a given L2 VLAN domain, the switch device should flood multicast/broadcast
  and unknown unicast packets to all ports in domain, if allowed by port's
  current STP state.  The switch driver, knowing which ports are within which

  vlan L2 domain, can program the switch device for flooding.  The packet may
  be sent to the port netdev for processing by the bridge driver.  The

  bridge should not reflood the packet to the same ports the device flooded,
  otherwise there will be duplicate packets on the wire.

  To avoid duplicate packets, the switch driver should mark a packet as already
  forwarded by setting the skb->offload_fwd_mark bit. The bridge driver will mark
  the skb using the ingress bridge port's mark and prevent it from being forwarded
  through any bridge port with the same mark.

  
  It is possible for the switch device to not handle flooding and push the
  packets up to the bridge driver for flooding.  This is not ideal as the number
  of ports scale in the L2 domain as the device is much more efficient at
  flooding packets that software.

  If supported by the device, flood control can be offloaded to it, preventing
  certain netdevs from flooding unicast traffic for which there is no FDB entry.

  IGMP Snooping
  ^^^^^^^^^^^^^

  In order to support IGMP snooping, the port netdevs should trap to the bridge
  driver all IGMP join and leave messages.
  The bridge multicast module will notify port netdevs on every multicast group
  changed whether it is static configured or dynamically joined/leave.
  The hardware implementation should be forwarding all registered multicast
  traffic groups only to the configured ports.

  L3 Routing Offload
  ------------------

  
  Offloading L3 routing requires that device be programmed with FIB entries from
  the kernel, with the device doing the FIB lookup and forwarding.  The device
  does a longest prefix match (LPM) on FIB entries matching route prefix and

  forwards the packet to the matching FIB entry's nexthop(s) egress ports.

  To program the device, the driver has to register a FIB notifier handler
  using register_fib_notifier. The following events are available:
  FIB_EVENT_ENTRY_ADD: used for both adding a new FIB entry to the device,
                       or modifying an existing entry on the device.
  FIB_EVENT_ENTRY_DEL: used for removing a FIB entry
  FIB_EVENT_RULE_ADD, FIB_EVENT_RULE_DEL: used to propagate FIB rule changes

  FIB_EVENT_ENTRY_ADD and FIB_EVENT_ENTRY_DEL events pass:

  	struct fib_entry_notifier_info {
  		struct fib_notifier_info info; /* must be first */

  		u32 dst;
  		int dst_len;
  		struct fib_info *fi;
  		u8 tos;
  		u8 type;

  		u32 tb_id;

  		u32 nlflags;
  	};

  
  to add/modify/delete IPv4 dst/dest_len prefix on table tb_id.  The *fi
  structure holds details on the route and route's nexthops.  *dev is one of the

  port netdevs mentioned in the route's next hop list.

  
  Routes offloaded to the device are labeled with "offload" in the ip route
  listing:
  
  	$ ip route show
  	default via 192.168.0.2 dev eth0
  	11.0.0.0/30 dev sw1p1  proto kernel  scope link  src 11.0.0.2 offload
  	11.0.0.4/30 via 11.0.0.1 dev sw1p1  proto zebra  metric 20 offload
  	11.0.0.8/30 dev sw1p2  proto kernel  scope link  src 11.0.0.10 offload
  	11.0.0.12/30 via 11.0.0.9 dev sw1p2  proto zebra  metric 20 offload
  	12.0.0.2  proto zebra  metric 30 offload
  		nexthop via 11.0.0.1  dev sw1p1 weight 1
  		nexthop via 11.0.0.9  dev sw1p2 weight 1
  	12.0.0.3 via 11.0.0.1 dev sw1p1  proto zebra  metric 20 offload
  	12.0.0.4 via 11.0.0.9 dev sw1p2  proto zebra  metric 20 offload
  	192.168.0.0/24 dev eth0  proto kernel  scope link  src 192.168.0.15

  The "offload" flag is set in case at least one device offloads the FIB entry.

  XXX: add/mod/del IPv6 FIB API

  
  Nexthop Resolution
  ^^^^^^^^^^^^^^^^^^
  
  The FIB entry's nexthop list contains the nexthop tuple (gateway, dev), but for
  the switch device to forward the packet with the correct dst mac address, the
  nexthop gateways must be resolved to the neighbor's mac address.  Neighbor mac
  address discovery comes via the ARP (or ND) process and is available via the
  arp_tbl neighbor table.  To resolve the routes nexthop gateways, the driver
  should trigger the kernel's neighbor resolution process.  See the rocker
  driver's rocker_port_ipv4_resolve() for an example.
  
  The driver can monitor for updates to arp_tbl using the netevent notifier
  NETEVENT_NEIGH_UPDATE.  The device can be programmed with resolved nexthops

  for the routes as arp_tbl updates.  The driver implements ndo_neigh_destroy
  to know when arp_tbl neighbor entries are purged from the port.

  
  Transaction item queue
  ^^^^^^^^^^^^^^^^^^^^^^
  
  For switchdev ops attr_set and obj_add, there is a 2 phase transaction model
  used. First phase is to "prepare" anything needed, including various checks,
  memory allocation, etc. The goal is to handle the stuff that is not unlikely
  to fail here. The second phase is to "commit" the actual changes.

  Switchdev provides an infrastructure for sharing items (for example memory

  allocations) between the two phases.
  
  The object created by a driver in "prepare" phase and it is queued up by:
  switchdev_trans_item_enqueue()
  During the "commit" phase, the driver gets the object by:
  switchdev_trans_item_dequeue()
  
  If a transaction is aborted during "prepare" phase, switchdev code will handle
  cleanup of the queued-up objects.