01 Sep, 2020

1 commit


18 Jul, 2020

2 commits


31 Oct, 2019

1 commit

  • This socket field can be read and written by concurrent cpus.

    Use READ_ONCE() and WRITE_ONCE() annotations to document this,
    and avoid some compiler 'optimizations'.

    KCSAN reported :

    BUG: KCSAN: data-race in tcp_v4_rcv / tcp_v4_rcv

    write to 0xffff88812220763c of 4 bytes by interrupt on cpu 0:
    sk_incoming_cpu_update include/net/sock.h:953 [inline]
    tcp_v4_rcv+0x1b3c/0x1bb0 net/ipv4/tcp_ipv4.c:1934
    ip_protocol_deliver_rcu+0x4d/0x420 net/ipv4/ip_input.c:204
    ip_local_deliver_finish+0x110/0x140 net/ipv4/ip_input.c:231
    NF_HOOK include/linux/netfilter.h:305 [inline]
    NF_HOOK include/linux/netfilter.h:299 [inline]
    ip_local_deliver+0x133/0x210 net/ipv4/ip_input.c:252
    dst_input include/net/dst.h:442 [inline]
    ip_rcv_finish+0x121/0x160 net/ipv4/ip_input.c:413
    NF_HOOK include/linux/netfilter.h:305 [inline]
    NF_HOOK include/linux/netfilter.h:299 [inline]
    ip_rcv+0x18f/0x1a0 net/ipv4/ip_input.c:523
    __netif_receive_skb_one_core+0xa7/0xe0 net/core/dev.c:5010
    __netif_receive_skb+0x37/0xf0 net/core/dev.c:5124
    process_backlog+0x1d3/0x420 net/core/dev.c:5955
    napi_poll net/core/dev.c:6392 [inline]
    net_rx_action+0x3ae/0xa90 net/core/dev.c:6460
    __do_softirq+0x115/0x33f kernel/softirq.c:292
    do_softirq_own_stack+0x2a/0x40 arch/x86/entry/entry_64.S:1082
    do_softirq.part.0+0x6b/0x80 kernel/softirq.c:337
    do_softirq kernel/softirq.c:329 [inline]
    __local_bh_enable_ip+0x76/0x80 kernel/softirq.c:189

    read to 0xffff88812220763c of 4 bytes by interrupt on cpu 1:
    sk_incoming_cpu_update include/net/sock.h:952 [inline]
    tcp_v4_rcv+0x181a/0x1bb0 net/ipv4/tcp_ipv4.c:1934
    ip_protocol_deliver_rcu+0x4d/0x420 net/ipv4/ip_input.c:204
    ip_local_deliver_finish+0x110/0x140 net/ipv4/ip_input.c:231
    NF_HOOK include/linux/netfilter.h:305 [inline]
    NF_HOOK include/linux/netfilter.h:299 [inline]
    ip_local_deliver+0x133/0x210 net/ipv4/ip_input.c:252
    dst_input include/net/dst.h:442 [inline]
    ip_rcv_finish+0x121/0x160 net/ipv4/ip_input.c:413
    NF_HOOK include/linux/netfilter.h:305 [inline]
    NF_HOOK include/linux/netfilter.h:299 [inline]
    ip_rcv+0x18f/0x1a0 net/ipv4/ip_input.c:523
    __netif_receive_skb_one_core+0xa7/0xe0 net/core/dev.c:5010
    __netif_receive_skb+0x37/0xf0 net/core/dev.c:5124
    process_backlog+0x1d3/0x420 net/core/dev.c:5955
    napi_poll net/core/dev.c:6392 [inline]
    net_rx_action+0x3ae/0xa90 net/core/dev.c:6460
    __do_softirq+0x115/0x33f kernel/softirq.c:292
    run_ksoftirqd+0x46/0x60 kernel/softirq.c:603
    smpboot_thread_fn+0x37d/0x4a0 kernel/smpboot.c:165

    Reported by Kernel Concurrency Sanitizer on:
    CPU: 1 PID: 16 Comm: ksoftirqd/1 Not tainted 5.4.0-rc3+ #0
    Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011

    Signed-off-by: Eric Dumazet
    Reported-by: syzbot
    Signed-off-by: David S. Miller

    Eric Dumazet
     

08 Jun, 2019

1 commit


06 Jun, 2019

1 commit


31 May, 2019

1 commit

  • Based on 1 normalized pattern(s):

    this program is free software you can redistribute it and or modify
    it under the terms of the gnu general public license as published by
    the free software foundation either version 2 of the license or at
    your option any later version

    extracted by the scancode license scanner the SPDX license identifier

    GPL-2.0-or-later

    has been chosen to replace the boilerplate/reference in 3029 file(s).

    Signed-off-by: Thomas Gleixner
    Reviewed-by: Allison Randal
    Cc: linux-spdx@vger.kernel.org
    Link: https://lkml.kernel.org/r/20190527070032.746973796@linutronix.de
    Signed-off-by: Greg Kroah-Hartman

    Thomas Gleixner
     

15 Dec, 2018

1 commit

  • A relatively common use case is to have several IPs configured
    on a host, and have different listeners for each of them. We would
    like to add a "catch all" listener on addr_any, to match incoming
    connections not served by any of the listeners bound to a specific
    address.

    However, port-only lookups can match addr_any sockets when sockets
    listening on specific addresses are present if so_reuseport flag
    is set. This patch eliminates lookups into port-only hashtable,
    as lookups by (addr,port) tuple are easily available.

    In addition, compute_score() is tweaked to _not_ match
    addr_any sockets to specific addresses, as hash collisions
    could result in the unwanted behavior described above.

    Tested: the patch compiles; full test in the last patch in this
    patchset. Existing reuseport_* selftests also pass.

    Suggested-by: Eric Dumazet
    Signed-off-by: Peter Oskolkov
    Signed-off-by: David S. Miller

    Peter Oskolkov
     

08 Nov, 2018

1 commit

  • The commit a04a480d4392 ("net: Require exact match for TCP socket
    lookups if dif is l3mdev") only ensures that the correct socket is
    selected for packets in a VRF. However, there is no guarantee that
    the unbound socket will be selected for packets when not in a VRF.
    By checking for a device match in compute_score() also for the case
    when there is no bound device and attaching a score to this, the
    unbound socket is selected. And if a failure is returned when there
    is no device match, this ensures that bound sockets are never selected,
    even if there is no unbound socket.

    Signed-off-by: Mike Manning
    Reviewed-by: David Ahern
    Tested-by: David Ahern
    Signed-off-by: David S. Miller

    Mike Manning
     

11 Aug, 2018

1 commit

  • This patch allows a BPF_PROG_TYPE_SK_REUSEPORT bpf prog to select a
    SO_REUSEPORT sk from a BPF_MAP_TYPE_REUSEPORT_ARRAY introduced in
    the earlier patch. "bpf_run_sk_reuseport()" will return -ECONNREFUSED
    when the BPF_PROG_TYPE_SK_REUSEPORT prog returns SK_DROP.
    The callers, in inet[6]_hashtable.c and ipv[46]/udp.c, are modified to
    handle this case and return NULL immediately instead of continuing the
    sk search from its hashtable.

    It re-uses the existing SO_ATTACH_REUSEPORT_EBPF setsockopt to attach
    BPF_PROG_TYPE_SK_REUSEPORT. The "sk_reuseport_attach_bpf()" will check
    if the attaching bpf prog is in the new SK_REUSEPORT or the existing
    SOCKET_FILTER type and then check different things accordingly.

    One level of "__reuseport_attach_prog()" call is removed. The
    "sk_unhashed() && ..." and "sk->sk_reuseport_cb" tests are pushed
    back to "reuseport_attach_prog()" in sock_reuseport.c. sock_reuseport.c
    seems to have more knowledge on those test requirements than filter.c.
    In "reuseport_attach_prog()", after new_prog is attached to reuse->prog,
    the old_prog (if any) is also directly freed instead of returning the
    old_prog to the caller and asking the caller to free.

    The sysctl_optmem_max check is moved back to the
    "sk_reuseport_attach_filter()" and "sk_reuseport_attach_bpf()".
    As of other bpf prog types, the new BPF_PROG_TYPE_SK_REUSEPORT is only
    bounded by the usual "bpf_prog_charge_memlock()" during load time
    instead of bounded by both bpf_prog_charge_memlock and sysctl_optmem_max.

    Signed-off-by: Martin KaFai Lau
    Acked-by: Alexei Starovoitov
    Signed-off-by: Daniel Borkmann

    Martin KaFai Lau
     

20 Jun, 2018

1 commit

  • Similar to 69678bcd4d2d ("udp: fix SO_BINDTODEVICE"), TCP socket lookups
    need to fail if dev_match is not true. Currently, a packet to a given port
    can match a socket bound to device when it should not. In the VRF case,
    this causes the lookup to hit a VRF socket and not a global socket
    resulting in a response trying to go through the VRF when it should not.

    Fixes: 3fa6f616a7a4d ("net: ipv4: add second dif to inet socket lookups")
    Fixes: 4297a0ef08572 ("net: ipv6: add second dif to inet6 socket lookups")
    Reported-by: Lou Berger
    Diagnosed-by: Renato Westphal
    Tested-by: Renato Westphal
    Signed-off-by: David Ahern
    Signed-off-by: David S. Miller

    David Ahern
     

03 Dec, 2017

1 commit

  • The current listener hashtable is hashed by port only.
    When a process is listening at many IP addresses with the same port (e.g.
    [IP1]:443, [IP2]:443... [IPN]:443), the inet[6]_lookup_listener()
    performance is degraded to a link list. It is prone to syn attack.

    UDP had a similar issue and a second hashtable was added to resolve it.

    This patch adds a second hashtable for the listener's sockets.
    The second hashtable is hashed by port and address.

    It cannot reuse the existing skc_portaddr_node which is shared
    with skc_bind_node. TCP listener needs to use skc_bind_node.
    Instead, this patch adds a hlist_node 'icsk_listen_portaddr_node' to
    the inet_connection_sock which the listener (like TCP) also belongs to.

    The new portaddr hashtable may need two lookup (First by IP:PORT.
    Second by INADDR_ANY:PORT if the IP:PORT is a not found). Hence,
    it implements a similar cut off as UDP such that it will only consult the
    new portaddr hashtable if the current port-only hashtable has >10
    sk in the link-list.

    lhash2 and lhash2_mask are added to 'struct inet_hashinfo'. I take
    this chance to plug a 4 bytes hole. It is done by first moving
    the existing bind_bucket_cachep up and then add the new
    (int lhash2_mask, *lhash2) after the existing bhash_size.

    Signed-off-by: Martin KaFai Lau
    Reviewed-by: Eric Dumazet
    Signed-off-by: David S. Miller

    Martin KaFai Lau
     

30 Nov, 2017

1 commit

  • Since commit e32ea7e74727 ("soreuseport: fast reuseport UDP socket
    selection") and commit c125e80b8868 ("soreuseport: fast reuseport
    TCP socket selection") the relevant reuseport socket matching the current
    packet is selected by the reuseport_select_sock() call. The only
    exceptions are invalid BPF filters/filters returning out-of-range
    indices.
    In the latter case the code implicitly falls back to using the hash
    demultiplexing, but instead of selecting the socket inside the
    reuseport_select_sock() function, it relies on the hash selection
    logic introduced with the early soreuseport implementation.

    With this patch, in case of a BPF filter returning a bad socket
    index value, we fall back to hash-based selection inside the
    reuseport_select_sock() body, so that we can drop some duplicate
    code in the ipv4 and ipv6 stack.

    This also allows faster lookup in the above scenario and will allow
    us to avoid computing the hash value for successful, BPF based
    demultiplexing - in a later patch.

    Signed-off-by: Paolo Abeni
    Acked-by: Craig Gallek
    Signed-off-by: David S. Miller

    Paolo Abeni
     

08 Aug, 2017

1 commit

  • Add a second device index, sdif, to inet6 socket lookups. sdif is the
    index for ingress devices enslaved to an l3mdev. It allows the lookups
    to consider the enslaved device as well as the L3 domain when searching
    for a socket.

    TCP moves the data in the cb. Prior to tcp_v4_rcv (e.g., early demux) the
    ingress index is obtained from IPCB using inet_sdif and after tcp_v4_rcv
    tcp_v4_sdif is used.

    Signed-off-by: David Ahern
    Signed-off-by: David S. Miller

    David Ahern
     

01 Jul, 2017

1 commit

  • refcount_t type and corresponding API should be
    used instead of atomic_t when the variable is used as
    a reference counter. This allows to avoid accidental
    refcounter overflows that might lead to use-after-free
    situations.

    This patch uses refcount_inc_not_zero() instead of
    atomic_inc_not_zero_hint() due to absense of a _hint()
    version of refcount API. If the hint() version must
    be used, we might need to revisit API.

    Signed-off-by: Elena Reshetova
    Signed-off-by: Hans Liljestrand
    Signed-off-by: Kees Cook
    Signed-off-by: David Windsor
    Signed-off-by: David S. Miller

    Reshetova, Elena
     

19 Jan, 2017

1 commit

  • We pass these per-protocol equal functions around in various places, but
    we can just have one function that checks the sk->sk_family and then do
    the right comparison function. I've also changed the ipv4 version to
    not cast to inet_sock since it is unneeded.

    Signed-off-by: Josef Bacik
    Signed-off-by: David S. Miller

    Josef Bacik
     

30 Oct, 2016

1 commit

  • As part of a series to implement faster SO_REUSEPORT lookups,
    commit 086c653f5862 ("sock: struct proto hash function may error")
    added return values to protocol hash functions and
    commit 496611d7b5ea ("inet: create IPv6-equivalent inet_hash function")
    implemented a new hash function for IPv6. However, the latter does
    not respect the former's convention.

    This properly propagates the hash errors in the IPv6 case.

    Fixes: 496611d7b5ea ("inet: create IPv6-equivalent inet_hash function")
    Reported-by: Soheil Hassas Yeganeh
    Signed-off-by: Craig Gallek
    Acked-by: Soheil Hassas Yeganeh
    Signed-off-by: David S. Miller

    Craig Gallek
     

17 Oct, 2016

1 commit

  • Currently, socket lookups for l3mdev (vrf) use cases can match a socket
    that is bound to a port but not a device (ie., a global socket). If the
    sysctl tcp_l3mdev_accept is not set this leads to ack packets going out
    based on the main table even though the packet came in from an L3 domain.
    The end result is that the connection does not establish creating
    confusion for users since the service is running and a socket shows in
    ss output. Fix by requiring an exact dif to sk_bound_dev_if match if the
    skb came through an interface enslaved to an l3mdev device and the
    tcp_l3mdev_accept is not set.

    skb's through an l3mdev interface are marked by setting a flag in
    inet{6}_skb_parm. The IPv6 variant is already set; this patch adds the
    flag for IPv4. Using an skb flag avoids a device lookup on the dif. The
    flag is set in the VRF driver using the IP{6}CB macros. For IPv4, the
    inet_skb_parm struct is moved in the cb per commit 971f10eca186, so the
    match function in the TCP stack needs to use TCP_SKB_CB. For IPv6, the
    move is done after the socket lookup, so IP6CB is used.

    The flags field in inet_skb_parm struct needs to be increased to add
    another flag. There is currently a 1-byte hole following the flags,
    so it can be expanded to u16 without increasing the size of the struct.

    Fixes: 193125dbd8eb ("net: Introduce VRF device driver")
    Signed-off-by: David Ahern
    Signed-off-by: David S. Miller

    David Ahern
     

28 Apr, 2016

1 commit


10 Apr, 2016

1 commit

  • A stupid refactoring bug in inet6_lookup_listener() needs to be fixed
    in order to get proper SO_REUSEPORT behavior.

    Fixes: 3b24d854cb35 ("tcp/dccp: do not touch listener sk_refcnt under synflood")
    Signed-off-by: Eric Dumazet
    Reported-by: Maciej Żenczykowski
    Signed-off-by: David S. Miller

    Eric Dumazet
     

05 Apr, 2016

3 commits

  • When a SYNFLOOD targets a non SO_REUSEPORT listener, multiple
    cpus contend on sk->sk_refcnt and sk->sk_wmem_alloc changes.

    By letting listeners use SOCK_RCU_FREE infrastructure,
    we can relax TCP_LISTEN lookup rules and avoid touching sk_refcnt

    Note that we still use SLAB_DESTROY_BY_RCU rules for other sockets,
    only listeners are impacted by this change.

    Peak performance under SYNFLOOD is increased by ~33% :

    On my test machine, I could process 3.2 Mpps instead of 2.4 Mpps

    Most consuming functions are now skb_set_owner_w() and sock_wfree()
    contending on sk->sk_wmem_alloc when cooking SYNACK and freeing them.

    Signed-off-by: Eric Dumazet
    Signed-off-by: David S. Miller

    Eric Dumazet
     
  • RX packet processing holds rcu_read_lock(), so we can remove
    pairs of rcu_read_lock()/rcu_read_unlock() in lookup functions
    if inet_diag also holds rcu before calling them.

    This is needed anyway as __inet_lookup_listener() and
    inet6_lookup_listener() will soon no longer increment
    refcount on the found listener.

    Signed-off-by: Eric Dumazet
    Signed-off-by: David S. Miller

    Eric Dumazet
     
  • Since linux 2.6.29, lookups only use rcu locking.

    Signed-off-by: Eric Dumazet
    Signed-off-by: David S. Miller

    Eric Dumazet
     

11 Feb, 2016

3 commits

  • This change extends the fast SO_REUSEPORT socket lookup implemented
    for UDP to TCP. Listener sockets with SO_REUSEPORT and the same
    receive address are additionally added to an array for faster
    random access. This means that only a single socket from the group
    must be found in the listener list before any socket in the group can
    be used to receive a packet. Previously, every socket in the group
    needed to be considered before handing off the incoming packet.

    This feature also exposes the ability to use a BPF program when
    selecting a socket from a reuseport group.

    Signed-off-by: Craig Gallek
    Signed-off-by: David S. Miller

    Craig Gallek
     
  • This is a preliminary step to allow fast socket lookup of SO_REUSEPORT
    groups. Doing so with a BPF filter will require access to the
    skb in question. This change plumbs the skb (and offset to payload
    data) through the call stack to the listening socket lookup
    implementations where it will be used in a following patch.

    Signed-off-by: Craig Gallek
    Signed-off-by: David S. Miller

    Craig Gallek
     
  • In order to support fast lookups for TCP sockets with SO_REUSEPORT,
    the function that adds sockets to the listening hash set needs
    to be able to check receive address equality. Since this equality
    check is different for IPv4 and IPv6, we will need two different
    socket hashing functions.

    This patch adds inet6_hash identical to the existing inet_hash function
    and updates the appropriate references. A following patch will
    differentiate the two by passing different comparison functions to
    __inet_hash.

    Additionally, in order to use the IPv6 address equality function from
    inet6_hashtables (which is compiled as a built-in object when IPv6 is
    enabled) it also needs to be in a built-in object file as well. This
    moves ipv6_rcv_saddr_equal into inet_hashtables to accomplish this.

    Signed-off-by: Craig Gallek
    Signed-off-by: David S. Miller

    Craig Gallek
     

13 Oct, 2015

1 commit

  • SO_INCOMING_CPU as added in commit 2c8c56e15df3 was a getsockopt() command
    to fetch incoming cpu handling a particular TCP flow after accept()

    This commits adds setsockopt() support and extends SO_REUSEPORT selection
    logic : If a TCP listener or UDP socket has this option set, a packet is
    delivered to this socket only if CPU handling the packet matches the specified
    one.

    This allows to build very efficient TCP servers, using one listener per
    RX queue, as the associated TCP listener should only accept flows handled
    in softirq by the same cpu.
    This provides optimal NUMA behavior and keep cpu caches hot.

    Note that __inet_lookup_listener() still has to iterate over the list of
    all listeners. Following patch puts sk_refcnt in a different cache line
    to let this iteration hit only shared and read mostly cache lines.

    Signed-off-by: Eric Dumazet
    Signed-off-by: David S. Miller

    Eric Dumazet
     

10 Jul, 2015

2 commits

  • inet_twsk_deschedule() calls are followed by inet_twsk_put().

    Only particular case is in inet_twsk_purge() but there is no point
    to defer the inet_twsk_put() after re-enabling BH.

    Lets rename inet_twsk_deschedule() to inet_twsk_deschedule_put()
    and move the inet_twsk_put() inside.

    Signed-off-by: Eric Dumazet
    Signed-off-by: David S. Miller

    Eric Dumazet
     
  • timewait sockets have a complex refcounting logic.
    Once we realize it should be similar to established and
    syn_recv sockets, we can use sk_nulls_del_node_init_rcu()
    and remove inet_twsk_unhash()

    In particular, deferred inet_twsk_put() added in commit
    13475a30b66cd ("tcp: connect() race with timewait reuse")
    looks unecessary : When removing a timewait socket from
    ehash or bhash, caller must own a reference on the socket
    anyway.

    Signed-off-by: Eric Dumazet
    Signed-off-by: David S. Miller

    Eric Dumazet
     

28 May, 2015

1 commit


14 Apr, 2015

1 commit

  • Using a timer wheel for timewait sockets was nice ~15 years ago when
    memory was expensive and machines had a single processor.

    This does not scale, code is ugly and source of huge latencies
    (Typically 30 ms have been seen, cpus spinning on death_lock spinlock.)

    We can afford to use an extra 64 bytes per timewait sock and spread
    timewait load to all cpus to have better behavior.

    Tested:

    On following test, /proc/sys/net/ipv4/tcp_tw_recycle is set to 1
    on the target (lpaa24)

    Before patch :

    lpaa23:~# ./super_netperf 200 -H lpaa24 -t TCP_CC -l 60 -- -p0,0
    419594

    lpaa23:~# ./super_netperf 200 -H lpaa24 -t TCP_CC -l 60 -- -p0,0
    437171

    While test is running, we can observe 25 or even 33 ms latencies.

    lpaa24:~# ping -c 1000 -i 0.02 -qn lpaa23
    ...
    1000 packets transmitted, 1000 received, 0% packet loss, time 20601ms
    rtt min/avg/max/mdev = 0.020/0.217/25.771/1.535 ms, pipe 2

    lpaa24:~# ping -c 1000 -i 0.02 -qn lpaa23
    ...
    1000 packets transmitted, 1000 received, 0% packet loss, time 20702ms
    rtt min/avg/max/mdev = 0.019/0.183/33.761/1.441 ms, pipe 2

    After patch :

    About 90% increase of throughput :

    lpaa23:~# ./super_netperf 200 -H lpaa24 -t TCP_CC -l 60 -- -p0,0
    810442

    lpaa23:~# ./super_netperf 200 -H lpaa24 -t TCP_CC -l 60 -- -p0,0
    800992

    And latencies are kept to minimal values during this load, even
    if network utilization is 90% higher :

    lpaa24:~# ping -c 1000 -i 0.02 -qn lpaa23
    ...
    1000 packets transmitted, 1000 received, 0% packet loss, time 19991ms
    rtt min/avg/max/mdev = 0.023/0.064/0.360/0.042 ms

    Signed-off-by: Eric Dumazet
    Signed-off-by: David S. Miller

    Eric Dumazet
     

19 Mar, 2015

4 commits


25 Aug, 2014

2 commits

  • This patch makes no changes to the logic of the code but simply addresses
    coding style issues as detected by checkpatch.

    Both objdump and diff -w show no differences.

    This patch removes some blank lines between the end of a function
    definition and the EXPORT_SYMBOL_GPL macro in order to prevent
    checkpatch warning that EXPORT_SYMBOL must immediately follow
    a function.

    Signed-off-by: Ian Morris
    Signed-off-by: David S. Miller

    Ian Morris
     
  • This patch makes no changes to the logic of the code but simply addresses
    coding style issues as detected by checkpatch.

    Both objdump and diff -w show no differences.

    A number of items are addressed in this patch:
    * Multiple spaces converted to tabs
    * Spaces before tabs removed.
    * Spaces in pointer typing cleansed (char *)foo etc.
    * Remove space after sizeof
    * Ensure spacing around comparators such as if statements.

    Signed-off-by: Ian Morris
    Signed-off-by: David S. Miller

    Ian Morris
     

24 Aug, 2014

1 commit


20 Oct, 2013

2 commits