Blame view
Documentation/networking/timestamping.txt
18.7 KB
8fe2f761c net-timestamp: ex... |
1 2 3 4 |
1. Control Interfaces The interfaces for receiving network packages timestamps are: |
cb9eff097 net: new user spa... |
5 6 |
* SO_TIMESTAMP |
8fe2f761c net-timestamp: ex... |
7 8 9 |
Generates a timestamp for each incoming packet in (not necessarily monotonic) system time. Reports the timestamp via recvmsg() in a control message as struct timeval (usec resolution). |
cb9eff097 net: new user spa... |
10 11 |
* SO_TIMESTAMPNS |
8fe2f761c net-timestamp: ex... |
12 13 |
Same timestamping mechanism as SO_TIMESTAMP, but reports the timestamp as struct timespec (nsec resolution). |
cb9eff097 net: new user spa... |
14 15 |
* IP_MULTICAST_LOOP + SO_TIMESTAMP[NS] |
8fe2f761c net-timestamp: ex... |
16 17 |
Only for multicast:approximate transmit timestamp obtained by reading the looped packet receive timestamp. |
cb9eff097 net: new user spa... |
18 |
|
8fe2f761c net-timestamp: ex... |
19 20 21 22 |
* SO_TIMESTAMPING Generates timestamps on reception, transmission or both. Supports multiple timestamp sources, including hardware. Supports generating timestamps for stream sockets. |
cb9eff097 net: new user spa... |
23 |
|
cb9eff097 net: new user spa... |
24 |
|
8fe2f761c net-timestamp: ex... |
25 |
1.1 SO_TIMESTAMP: |
adca47678 net: Improve SO_T... |
26 |
|
8fe2f761c net-timestamp: ex... |
27 28 29 30 |
This socket option enables timestamping of datagrams on the reception path. Because the destination socket, if any, is not known early in the network stack, the feature has to be enabled for all packets. The same is true for all early receive timestamp options. |
adca47678 net: Improve SO_T... |
31 |
|
8fe2f761c net-timestamp: ex... |
32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 |
For interface details, see `man 7 socket`. 1.2 SO_TIMESTAMPNS: This option is identical to SO_TIMESTAMP except for the returned data type. Its struct timespec allows for higher resolution (ns) timestamps than the timeval of SO_TIMESTAMP (ms). 1.3 SO_TIMESTAMPING: Supports multiple types of timestamp requests. As a result, this socket option takes a bitmap of flags, not a boolean. In err = setsockopt(fd, SOL_SOCKET, SO_TIMESTAMPING, (void *) val, &val); val is an integer with any of the following bits set. Setting other bit returns EINVAL and does not change the current state. |
adca47678 net: Improve SO_T... |
51 |
|
adca47678 net: Improve SO_T... |
52 |
|
8fe2f761c net-timestamp: ex... |
53 |
1.3.1 Timestamp Generation |
adca47678 net: Improve SO_T... |
54 |
|
8fe2f761c net-timestamp: ex... |
55 56 57 58 59 60 61 62 63 |
Some bits are requests to the stack to try to generate timestamps. Any combination of them is valid. Changes to these bits apply to newly created packets, not to packets already in the stack. As a result, it is possible to selectively request timestamps for a subset of packets (e.g., for sampling) by embedding an send() call within two setsockopt calls, one to enable timestamp generation and one to disable it. Timestamps may also be generated for reasons other than being requested by a particular socket, such as when receive timestamping is enabled system wide, as explained earlier. |
adca47678 net: Improve SO_T... |
64 |
|
8fe2f761c net-timestamp: ex... |
65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 |
SOF_TIMESTAMPING_RX_HARDWARE: Request rx timestamps generated by the network adapter. SOF_TIMESTAMPING_RX_SOFTWARE: Request rx timestamps when data enters the kernel. These timestamps are generated just after a device driver hands a packet to the kernel receive stack. SOF_TIMESTAMPING_TX_HARDWARE: Request tx timestamps generated by the network adapter. SOF_TIMESTAMPING_TX_SOFTWARE: Request tx timestamps when data leaves the kernel. These timestamps are generated in the device driver as close as possible, but always prior to, passing the packet to the network interface. Hence, they require driver support and may not be available for all devices. SOF_TIMESTAMPING_TX_SCHED: Request tx timestamps prior to entering the packet scheduler. Kernel transmit latency is, if long, often dominated by queuing delay. The difference between this timestamp and one taken at SOF_TIMESTAMPING_TX_SOFTWARE will expose this latency independent of protocol processing. The latency incurred in protocol processing, if any, can be computed by subtracting a userspace timestamp taken immediately before send() from this timestamp. On machines with virtual devices where a transmitted packet travels through multiple devices and, hence, multiple packet schedulers, a timestamp is generated at each layer. This allows for fine grained measurement of queuing delay. SOF_TIMESTAMPING_TX_ACK: Request tx timestamps when all data in the send buffer has been acknowledged. This only makes sense for reliable protocols. It is currently only implemented for TCP. For that protocol, it may over-report measurement, because the timestamp is generated when all data up to and including the buffer at send() was acknowledged: the cumulative acknowledgment. The mechanism ignores SACK and FACK. 1.3.2 Timestamp Reporting The other three bits control which timestamps will be reported in a generated control message. Changes to the bits take immediate effect at the timestamp reporting locations in the stack. Timestamps are only reported for packets that also have the relevant timestamp generation request set. SOF_TIMESTAMPING_SOFTWARE: Report any software timestamps when available. SOF_TIMESTAMPING_SYS_HARDWARE: This option is deprecated and ignored. SOF_TIMESTAMPING_RAW_HARDWARE: Report hardware timestamps as generated by SOF_TIMESTAMPING_TX_HARDWARE when available. 1.3.3 Timestamp Options |
829ae9d61 net-timestamp: al... |
124 |
The interface supports the options |
8fe2f761c net-timestamp: ex... |
125 126 127 128 129 130 131 |
SOF_TIMESTAMPING_OPT_ID: Generate a unique identifier along with each packet. A process can have multiple concurrent timestamping requests outstanding. Packets can be reordered in the transmit path, for instance in the packet scheduler. In that case timestamps will be queued onto the error |
cbd3aad5c net-timestamp: ex... |
132 133 134 135 136 137 138 139 140 141 142 143 144 145 |
queue out of order from the original send() calls. It is not always possible to uniquely match timestamps to the original send() calls based on timestamp order or payload inspection alone, then. This option associates each packet at send() with a unique identifier and returns that along with the timestamp. The identifier is derived from a per-socket u32 counter (that wraps). For datagram sockets, the counter increments with each sent packet. For stream sockets, it increments with every byte. The counter starts at zero. It is initialized the first time that the socket option is enabled. It is reset each time the option is enabled after having been disabled. Resetting the counter does not change the identifiers of existing packets in the system. |
8fe2f761c net-timestamp: ex... |
146 147 148 |
This option is implemented only for transmit timestamps. There, the timestamp is always looped along with a struct sock_extended_err. |
138a7f492 net-timestamp: Fi... |
149 |
The option modifies field ee_data to pass an id that is unique |
8fe2f761c net-timestamp: ex... |
150 |
among all possibly concurrently outstanding timestamp requests for |
cbd3aad5c net-timestamp: ex... |
151 |
that socket. |
8fe2f761c net-timestamp: ex... |
152 |
|
829ae9d61 net-timestamp: al... |
153 154 155 156 157 158 159 160 |
SOF_TIMESTAMPING_OPT_CMSG: Support recv() cmsg for all timestamped packets. Control messages are already supported unconditionally on all packets with receive timestamps and on IPv6 packets with transmit timestamp. This option extends them to IPv4 packets with transmit timestamp. One use case is to correlate packets with their egress device, by enabling socket option IP_PKTINFO simultaneously. |
49ca0d8bf net-timestamp: no... |
161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 |
SOF_TIMESTAMPING_OPT_TSONLY: Applies to transmit timestamps only. Makes the kernel return the timestamp as a cmsg alongside an empty packet, as opposed to alongside the original packet. This reduces the amount of memory charged to the socket's receive budget (SO_RCVBUF) and delivers the timestamp even if sysctl net.core.tstamp_allow_data is 0. This option disables SOF_TIMESTAMPING_OPT_CMSG. New applications are encouraged to pass SOF_TIMESTAMPING_OPT_ID to disambiguate timestamps and SOF_TIMESTAMPING_OPT_TSONLY to operate regardless of the setting of sysctl net.core.tstamp_allow_data. An exception is when a process needs additional cmsg data, for instance SOL_IP/IP_PKTINFO to detect the egress network interface. Then pass option SOF_TIMESTAMPING_OPT_CMSG. This option depends on having access to the contents of the original packet, so cannot be combined with SOF_TIMESTAMPING_OPT_TSONLY. |
8fe2f761c net-timestamp: ex... |
180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 |
1.4 Bytestream Timestamps The SO_TIMESTAMPING interface supports timestamping of bytes in a bytestream. Each request is interpreted as a request for when the entire contents of the buffer has passed a timestamping point. That is, for streams option SOF_TIMESTAMPING_TX_SOFTWARE will record when all bytes have reached the device driver, regardless of how many packets the data has been converted into. In general, bytestreams have no natural delimiters and therefore correlating a timestamp with data is non-trivial. A range of bytes may be split across segments, any segments may be merged (possibly coalescing sections of previously segmented buffers associated with independent send() calls). Segments can be reordered and the same byte range can coexist in multiple segments for protocols that implement retransmissions. It is essential that all timestamps implement the same semantics, regardless of these possible transformations, as otherwise they are incomparable. Handling "rare" corner cases differently from the simple case (a 1:1 mapping from buffer to skb) is insufficient because performance debugging often needs to focus on such outliers. In practice, timestamps can be correlated with segments of a bytestream consistently, if both semantics of the timestamp and the timing of measurement are chosen correctly. This challenge is no different from deciding on a strategy for IP fragmentation. There, the definition is that only the first fragment is timestamped. For bytestreams, we chose that a timestamp is generated only when all bytes have passed a point. SOF_TIMESTAMPING_TX_ACK as defined is easy to implement and reason about. An implementation that has to take into account SACK would be more complex due to possible transmission holes and out of order arrival. On the host, TCP can also break the simple 1:1 mapping from buffer to skbuff as a result of Nagle, cork, autocork, segmentation and GSO. The implementation ensures correctness in all cases by tracking the individual last byte passed to send(), even if it is no longer the last byte after an skbuff extend or merge operation. It stores the relevant sequence number in skb_shinfo(skb)->tskey. Because an skbuff has only one such field, only one timestamp can be generated. In rare cases, a timestamp request can be missed if two requests are collapsed onto the same skb. A process can detect this situation by enabling SOF_TIMESTAMPING_OPT_ID and comparing the byte offset at send time with the value returned for each timestamp. It can prevent the situation by always flushing the TCP stack in between requests, for instance by enabling TCP_NODELAY and disabling TCP_CORK and autocork. These precautions ensure that the timestamp is generated only when all bytes have passed a timestamp point, assuming that the network stack itself does not reorder the segments. The stack indeed tries to avoid reordering. The one exception is under administrator control: it is possible to construct a packet scheduler configuration that delays segments from the same stream differently. Such a setup would be unusual. 2 Data Interfaces Timestamps are read using the ancillary data feature of recvmsg(). See `man 3 cmsg` for details of this interface. The socket manual page (`man 7 socket`) describes how timestamps generated with SO_TIMESTAMP and SO_TIMESTAMPNS records can be retrieved. 2.1 SCM_TIMESTAMPING records These timestamps are returned in a control message with cmsg_level SOL_SOCKET, cmsg_type SCM_TIMESTAMPING, and payload of type |
69298698c net: corrected do... |
251 252 |
struct scm_timestamping { |
8fe2f761c net-timestamp: ex... |
253 |
struct timespec ts[3]; |
69298698c net: corrected do... |
254 |
}; |
cb9eff097 net: new user spa... |
255 |
|
8fe2f761c net-timestamp: ex... |
256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 |
The structure can return up to three timestamps. This is a legacy feature. Only one field is non-zero at any time. Most timestamps are passed in ts[0]. Hardware timestamps are passed in ts[2]. ts[1] used to hold hardware timestamps converted to system time. Instead, expose the hardware clock device on the NIC directly as a HW PTP clock source, to allow time conversion in userspace and optionally synchronize system time with a userspace PTP stack such as linuxptp. For the PTP clock API, see Documentation/ptp/ptp.txt. 2.1.1 Transmit timestamps with MSG_ERRQUEUE For transmit timestamps the outgoing packet is looped back to the socket's error queue with the send timestamp(s) attached. A process receives the timestamps by calling recvmsg() with flag MSG_ERRQUEUE set and with a msg_control buffer sufficiently large to receive the relevant metadata structures. The recvmsg call returns the original outgoing data packet with two ancillary messages attached. A message of cm_level SOL_IP(V6) and cm_type IP(V6)_RECVERR embeds a struct sock_extended_err. This defines the error type. For timestamps, the ee_errno field is ENOMSG. The other ancillary message will have cm_level SOL_SOCKET and cm_type SCM_TIMESTAMPING. This embeds the struct scm_timestamping. 2.1.1.2 Timestamp types The semantics of the three struct timespec are defined by field ee_info in the extended error structure. It contains a value of type SCM_TSTAMP_* to define the actual timestamp passed in scm_timestamping. The SCM_TSTAMP_* types are 1:1 matches to the SOF_TIMESTAMPING_* control fields discussed previously, with one exception. For legacy reasons, SCM_TSTAMP_SND is equal to zero and can be set for both SOF_TIMESTAMPING_TX_HARDWARE and SOF_TIMESTAMPING_TX_SOFTWARE. It is the first if ts[2] is non-zero, the second otherwise, in which case the timestamp is stored in ts[0]. 2.1.1.3 Fragmentation Fragmentation of outgoing datagrams is rare, but is possible, e.g., by explicitly disabling PMTU discovery. If an outgoing packet is fragmented, then only the first fragment is timestamped and returned to the sending socket. 2.1.1.4 Packet Payload The calling application is often not interested in receiving the whole packet payload that it passed to the stack originally: the socket error queue mechanism is just a method to piggyback the timestamp on. In this case, the application can choose to read datagrams with a smaller buffer, possibly even of length 0. The payload is truncated accordingly. Until the process calls recvmsg() on the error queue, however, the full packet is queued, taking up budget from SO_RCVBUF. 2.1.1.5 Blocking Read Reading from the error queue is always a non-blocking operation. To block waiting on a timestamp, use poll or select. poll() will return POLLERR in pollfd.revents if any data is ready on the error queue. There is no need to pass this flag in pollfd.events. This flag is ignored on request. See also `man 2 poll`. 2.1.2 Receive timestamps On reception, there is no reason to read from the socket error queue. The SCM_TIMESTAMPING ancillary data is sent along with the packet data on a normal recvmsg(). Since this is not a socket error, it is not accompanied by a message SOL_IP(V6)/IP(V6)_RECVERROR. In this case, the meaning of the three fields in struct scm_timestamping is implicitly defined. ts[0] holds a software timestamp if set, ts[1] is again deprecated and ts[2] holds a hardware timestamp if set. 3. Hardware Timestamping configuration: SIOCSHWTSTAMP and SIOCGHWTSTAMP |
cb9eff097 net: new user spa... |
337 338 |
Hardware time stamping must also be initialized for each device driver |
69298698c net: corrected do... |
339 340 |
that is expected to do hardware time stamping. The parameter is defined in /include/linux/net_tstamp.h as: |
cb9eff097 net: new user spa... |
341 342 |
struct hwtstamp_config { |
69298698c net: corrected do... |
343 344 345 |
int flags; /* no flags defined right now, must be zero */ int tx_type; /* HWTSTAMP_TX_* */ int rx_filter; /* HWTSTAMP_FILTER_* */ |
cb9eff097 net: new user spa... |
346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 |
}; Desired behavior is passed into the kernel and to a specific device by calling ioctl(SIOCSHWTSTAMP) with a pointer to a struct ifreq whose ifr_data points to a struct hwtstamp_config. The tx_type and rx_filter are hints to the driver what it is expected to do. If the requested fine-grained filtering for incoming packets is not supported, the driver may time stamp more than just the requested types of packets. A driver which supports hardware time stamping shall update the struct with the actual, possibly more permissive configuration. If the requested packets cannot be time stamped, then nothing should be changed and ERANGE shall be returned (in contrast to EINVAL, which indicates that SIOCSHWTSTAMP is not supported at all). Only a processes with admin rights may change the configuration. User space is responsible to ensure that multiple processes don't interfere with each other and that the settings are reset. |
fd468c74b net_tstamp: Add S... |
365 366 367 |
Any process can read the actual configuration by passing this structure to ioctl(SIOCGHWTSTAMP) in the same way. However, this has not been implemented in all drivers. |
cb9eff097 net: new user spa... |
368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 |
/* possible values for hwtstamp_config->tx_type */ enum { /* * no outgoing packet will need hardware time stamping; * should a packet arrive which asks for it, no hardware * time stamping will be done */ HWTSTAMP_TX_OFF, /* * enables hardware time stamping for outgoing packets; * the sender of the packet decides which are to be * time stamped by setting SOF_TIMESTAMPING_TX_SOFTWARE * before sending the packet */ HWTSTAMP_TX_ON, }; /* possible values for hwtstamp_config->rx_filter */ enum { /* time stamp no incoming packet at all */ HWTSTAMP_FILTER_NONE, /* time stamp any incoming packet */ HWTSTAMP_FILTER_ALL, |
69298698c net: corrected do... |
393 394 |
/* return value: time stamp all packets requested plus some others */ HWTSTAMP_FILTER_SOME, |
cb9eff097 net: new user spa... |
395 396 397 |
/* PTP v1, UDP, any kind of event packet */ HWTSTAMP_FILTER_PTP_V1_L4_EVENT, |
69298698c net: corrected do... |
398 399 400 |
/* for the complete list of values, please check * the include file /include/linux/net_tstamp.h */ |
cb9eff097 net: new user spa... |
401 |
}; |
8fe2f761c net-timestamp: ex... |
402 |
3.1 Hardware Timestamping Implementation: Device Drivers |
cb9eff097 net: new user spa... |
403 404 |
A driver which supports hardware time stamping must support the |
69298698c net: corrected do... |
405 |
SIOCSHWTSTAMP ioctl and update the supplied struct hwtstamp_config with |
fd468c74b net_tstamp: Add S... |
406 407 |
the actual values as described in the section on SIOCSHWTSTAMP. It should also support SIOCGHWTSTAMP. |
69298698c net: corrected do... |
408 409 410 411 412 413 414 415 416 417 |
Time stamps for received packets must be stored in the skb. To get a pointer to the shared time stamp structure of the skb call skb_hwtstamps(). Then set the time stamps in the structure: struct skb_shared_hwtstamps { /* hardware time stamp transformed into duration * since arbitrary point in time */ ktime_t hwtstamp; |
69298698c net: corrected do... |
418 |
}; |
cb9eff097 net: new user spa... |
419 420 |
Time stamps for outgoing packets are to be generated as follows: |
2244d07bf net: simplify fla... |
421 422 423 |
- In hard_start_xmit(), check if (skb_shinfo(skb)->tx_flags & SKBTX_HW_TSTAMP) is set no-zero. If yes, then the driver is expected to do hardware time stamping. |
cb9eff097 net: new user spa... |
424 |
- If this is possible for the skb and requested, then declare |
2244d07bf net: simplify fla... |
425 426 427 428 429 430 431 432 433 |
that the driver is doing the time stamping by setting the flag SKBTX_IN_PROGRESS in skb_shinfo(skb)->tx_flags , e.g. with skb_shinfo(skb)->tx_flags |= SKBTX_IN_PROGRESS; You might want to keep a pointer to the associated skb for the next step and not free the skb. A driver not supporting hardware time stamping doesn't do that. A driver must never touch sk_buff::tstamp! It is used to store software generated time stamps by the network subsystem. |
59cb89e6b doc: update drive... |
434 435 436 |
- Driver should call skb_tx_timestamp() as close to passing sk_buff to hardware as possible. skb_tx_timestamp() provides a software time stamp if requested and hardware timestamping is not possible (SKBTX_IN_PROGRESS not set). |
cb9eff097 net: new user spa... |
437 438 439 |
- As soon as the driver has sent the packet and/or obtained a hardware time stamp for it, it passes the time stamp back by calling skb_hwtstamp_tx() with the original skb, the raw |
69298698c net: corrected do... |
440 441 442 443 444 445 446 |
hardware time stamp. skb_hwtstamp_tx() clones the original skb and adds the timestamps, therefore the original skb has to be freed now. If obtaining the hardware time stamp somehow fails, then the driver should not fall back to software time stamping. The rationale is that this would occur at a later time in the processing pipeline than other software time stamping and therefore could lead to unexpected deltas between time stamps. |