Let’s talk about deterministic networks

Low latency in the network is particularly important for many emerging services and applications, such as drones, industrial automation, and self-driving cars. International standards organizations are currently developing new technologies to meet the requirements of these deterministic applications.

IEEE 802.1 is working to support deterministic Ethernet services in its Time Sensitive Networking (TSN) Task Group. 3GPP is working to deliver deterministic 5G to support ultra-reliable and low-latency communications (URLLC) use cases. IETF is working to deliver deterministic services over IP routers and wireless networks in the Deterministic Networking (DetNet) and RAW working groups.

Deterministic networks provide deterministic latency on a per-deterministic flow basis. Data traffic for each deterministic flow is delivered within deterministic bounded latency and low latency variation constraints. Deterministic networks aim to achieve zero data loss for all allowed deterministic flows, and may deny or deprioritize certain flows to ensure delivery of high priority flows. Deterministic networks support a wide range of applications, each of which can have different QoS requirements.

In traditional networks, achieving lower latency means dropping more packets (or requiring a lot of overprovisioning). In the case of deterministic services, the goal is to address the long tail and provide bounded latency, see Figure 1.

Figure 1: Traditional services and deterministic services

IEEE 802.1 Time-Sensitive Networking (TSN)

standardization

The IEEE 802.1 Working Group (WG) focuses on standards and practices in the following areas: (1) 802 LAN/MAN architecture, (2) interworking between 802 LANs, MANs, and other WANs, (3) 802 Security, (4) 802 overall network management, and protocol layers above the MAC and LLC layers.

The Time Sensitive Networking (TSN) Task Group (TG) of the IEEE 802.1 Working Group is responsible for deterministic services in IEEE 802 networks, including:

• Guaranteed packet delivery
• Low packet loss rate
• Bounded Low Latency
• Low packet delay variation

The TSN Task Group evolved from the Audio Video Bridging (AVB) Task Group.

TSN standards and projects are divided into three groups:

1) Basic technologies (such as 802.1CB, 802.1Qbv, etc.)

2) Configuration (such as 802.1Qcp, 802.1Qcc, etc.)

3) Profiles (e.g., 802.1BA, 802.1CM, IEC/IEEE 60802, etc.)

Figure 2: IEEE 802.1 TSN components

TSN Features

IEEE 802.1 defines a TSN flow as a unidirectional data flow from a Talker to one or more Listeners. During the forwarding process of the bridge, QoS functions are applied to the frames of the TSN flow, such as filtering and policing, shaping, and queuing.

IEEE 802.1 TSN TG defines a wide range of TSN capabilities. This article discusses only some of them. The primary medium for TSN is IEEE 802.3 Ethernet. Work is also underway involving wireless, such as the 5G – TSN integration work in 3GPP.

Scheduled Traffic (802.1Qbv) reduces delay variations in known time frames. This is achieved through time-based control and bridge queue programming. Each queue is equipped with time-gates (time-gated queues), and the queue can only be served when the "gate" is open. The gate open/closed state changes according to a periodically repeated schedule. This feature requires end-to-end time synchronization.

Frame Preemption (802.3br and 802.1Qbu) enables so-called fast frames (i.e. critical traffic) to pause the transmission of preemptible frames (i.e. non-critical traffic). As a result, the delay variation of fast traffic is reduced and the available bandwidth for preemptible traffic is increased. Frame preemption is a link-local per-hop function, that is, not multi-hop.

Per-Stream Filtering and Policing (802.1Qci) provides protection against traffic violating its bandwidth allocation, malfunctioning, participating in attacks, etc. Filtering and policing decisions can be made on a per-stream, per-priority, etc. basis.

Asynchronous Traffic Shaping (ATS) (802.1Qcr) provides zero congestion penalty and does not require time synchronization. The essence of the ATS feature is to smooth the traffic pattern by reshaping at each hop so that urgent traffic takes precedence over less urgent or elastic traffic. ATS uses strict priority queues.

Frame Replication and Elimination for Reliability (FRER) (802.1CB) is designed to avoid frame losses due to equipment failures. It is a per-frame 1+1 (or 1+n) redundancy feature. No failure detection or switching mechanisms are required. FRER sends frames on two (or more) maximally disjoint paths, then combines the streams and removes extra frames.

Explicit Trees by IS-IS Path Control & Reservation (802.1Qca, RFC 7813) adds non-shortest path or explicit path forwarding, providing IS-IS control beyond the shortest path tree (SPT). The protocol has not changed, only a few new sub-TLVs are defined, and existing sub-TLVs are reused as much as possible. The concept is a hybrid software-defined network (SDN) approach, where IS-IS provides basic functions such as topology discovery and default paths, and one or more controllers control the explicit trees.

Stream Reservation Protocol (SRP) Enhancements and Performance Improvements (802.1Qcc): Provides time-sensitive network (TSN) configuration-related properties. 802.1Qcc describes three models for TSN user and network configuration (fully distributed, centralized network/distributed user, and fully centralized models). Each model specification defines the logical flow of user/network configuration information between different entities in the network.

The future of TSN

TSN standardization is still in progress. The IEC/IEEE 60802 TSN Industrial Automation Specification is a joint project of IEC SC65C/WG18 and IEEE 802. This joint effort will provide a dual-label standard that is both an International Electrotechnical Commission (IEC) and an IEEE standard.

OPC UA is built on TSN, DetNet and 5G. Several OPC UA work items related to TSN are in progress. One of them is the FLC (Field Level Communications) working group, which is mainly based on the IEC/IEEE 60802 specification and related assessment specifications.

3GPP supports deterministic transmission (URLLC)

standardization

The three major application scenarios of 5G include enhanced mobile broadband (eMMB), massive machine type communication (mMTC) and ultra-reliable and low-latency communication (uRLLC). Among them, URLLC makes 5G the best candidate to support wireless deterministic and time-sensitive communication applications.

5G R15 introduced several features, with one-way latency of message transmission as low as 1 millisecond and reliability as high as 99.999%. R16 added more URLLC features to support one-way latency as low as 0.5 milliseconds and reliability as high as 99.9999%.

URLLC Features

At the beginning of R15 research, a work project was established to study delay reduction technologies such as subcarrier spacing, flexible frame structure, and short time slot scheduling. As of R16, 3GPP has completed the performance evaluation of URLLC use cases, the enhancement of each channel in the physical layer, and the research and standardization of technologies such as URLLC and eMBB uplink multiplexing, but there are still many optimization tasks expected to be left for R17 research.

5G defines robust transmission modes to improve reliability for both data and control radio channels. Multi-antenna transmission, use of multiple carriers, and packet duplication on independent radio links all further improve reliability.

Time synchronization has been embedded into cellular radio systems as an essential component of their operation. Devices are time-calibrated by the base station to compensate for their different propagation delays. The wireless network components themselves are also time-synchronized. This is an excellent basis for providing synchronization for time-critical applications.

In addition to 5G RAN functions, the 5G system also provides core network (CN) solutions for Ethernet networking and URLLC. 5G CN supports local Ethernet protocol data unit (PDU) sessions. For user plane redundancy at the 5G system level, 5G supports the establishment of redundant user plane paths through the 5G system including RAN, CN, and transport network. Redundant paths are achieved by using a single user equipment (UE) with RAN dual connection function in the terminal device or by using multiple UEs in the terminal device. In addition, 5G can also provide virtual networks (5G-VN) and LAN groups to allocate resources to members of a specific group.

All these new URLLC features of 5G provide a good design and a solid foundation for using 5G in deterministic scenarios, even as a standalone solution or as part of a deterministic network.

Figure 3: System architecture view, 5GS shown as a deterministic node (here a TSN bridge)

Figure 3 shows the 5G system architecture, where the 5G system is considered as a TSN bridge. A new translation function (called DS-TT and NW-TT) is specified to save and forward user plane packets to eliminate jitter, where the 5G system (5GS) is integrated as a bridge to connect TSN networks. The 5GS includes the TSN Translator (TT) function to adapt the 5GS to the TSN domain of the user plane and control plane.

Future prospects of URLLC

5G URLLC capabilities are a good match for TSN and deterministic networking capabilities. Therefore, the three technologies can be integrated to provide end-to-end deterministic connectivity, i.e., between input/output devices and their controllers. The integration already includes data plane support for the necessary basic bridging/routing functions and TSN/DetNet add-ons, but the control and management planes require further standardization work.

IETF Deterministic Networking (DetNet)

standardization

IETF DetNet WG (Working Group) belongs to the Routing Area (RTG), mainly studying routing protocols and signaling protocols. It focuses on deterministic data paths running on Layer 2 bridges and Layer 3 routing segments, which can provide limitations on latency, loss, and packet jitter and have high reliability. The scope of DetNet WG includes: overall architecture, data plane specifications, data flow information model, and related YANG models.

There is a close collaboration between the IETF DetNet WG and the IEEE 802.1 TSN TG.

DetNet operates at the IP/MPLS layer, with the initial scope being to achieve deterministic assurance for networks under a single management control or within a closed management group.

The solution document specifies the procedures and behaviors required for nodes supporting DetNet, and its specifications focus on interoperable implementations. The following two data planes are defined:

• IP: Use IP and transport protocol header information to support DetNet [RFC 8939]
• MPLS: Using Labels to Support DetNet [RFC 8964

The forwarding feature is achieved by allocating network resources (such as link bandwidth and buffer space) to DetNet flows and protecting data packets. Unused reserved resources can be used for the transmission of non-DetNet data flows, enabling co-network transmission of business flows of different priorities.

The following defines the forwarding parameters from the source to the destination layer:

• Minimum and maximum end-to-end delay: timely delivery, and bounded jitter (packet delay variation) derived from these constraints
• Packet loss rate: Lost during transmission, extremely low packet loss value can be applied
• Upper bound on out-of-order packet delivery: Some deterministic network applications cannot tolerate any out-of-order delivery

One difference with deterministic networking (similar to TSN) is that it only looks at the worst-case values ​​for end-to-end latency, latency variation, and out-of-orderness. Average or typical values ​​are not important because they do not impact the ability of the real-time system to perform its tasks.

Deterministic Networking Features:

• Congestion protection
• Service Guarantee
• Explicit Routing

Congestion protection means allocating resources such as buffer space or link bandwidth along the path of a DetNet flow.

Congestion protection eliminates the congestion-related penalties by using properly designed queues, so that packets are not dropped due to lack of buffer storage. It also serves as a tool to reduce delay variations, for example, allowing the convergence of sensitive non-IP networks onto a common IP network infrastructure. Many features of congestion protection require time synchronization of deterministic network nodes, however, time synchronization is outside the scope of the deterministic network discussion as it does not affect interoperability. Time synchronization should be provided by an appropriate solution, for example, by lower layers.

Service protection addresses packet errors and equipment failures, for example, packet duplication and elimination (to prevent failures), packet encoding (to prevent packet errors), and reordering (to ensure in-order delivery), which can be used to ensure service protection. The PREOF defined by deterministic networks are: packet duplication function (PRF: sending copies of the same packet with sequencing information on multiple paths), redundancy elimination function (PEF: discarding duplicates based on the sequencing information and history of received packets), and packet sequencing function (POF: restoring the original packet order, because out-of-order delivery affects the buffering capacity of the destination to correctly process the received data). Packet duplication and elimination do not react and correct for failures, these functions are completely passive. Packet encoding (also known as network coding) encodes information into multiple transmission units, sends them using multiple paths, and combines these units at the other end.

Explicit routing can be used to address the effects of convergence (i.e., temporary disruption) of routing or bridging protocols.

Deterministic networking functionality is implemented in two adjacent sublayers of the protocol stack:

1) DetNet service sublayer: provides DetNet services (e.g., service protection) to higher layers in the protocol stack and applications

2) DetNet forwarding sublayer: supports DetNet services in the underlying network (e.g., by providing explicit routing and congestion protection) to DetNet flows

Figure 4: DetNet data plane protocol stack

The layer 3 equivalent of a TSN flow is called a DetNet flow. A DetNet flow is a sequence of packets that uniquely conforms to a flow identifier and to which deterministic network services will be provided. It includes any deterministic network headers added to support the DetNet service and the forwarding sublayer.

Deterministic network-related mechanisms require two properties:

• Flow-ID: Identifies the flow to which the packet belongs
• Sequence Numbers: Identify duplicate packets and reorder packets

The future of deterministic networking

Standardization of deterministic networks is still in progress. Close collaboration between IETF DetNet and IEEE TSN will continue to ensure interoperability and simplify the implementation of deterministic functions applicable to Layer 2 and Layer 3. For example, IEEE P802.1CBdb (FRER Extended Stream Identification Functions) focuses on extending the fields used for stream identification functions to arbitrary mask matching, which is critical for combined network scenarios combining TSN and DetNet. Control and management plane related work is the next focus of the DetNet WG.

in conclusion

In the past, packet-based networks were designed to carry all but very latency-sensitive/real-time application traffic. Over time, leveraging advances in deterministic technology, packet-based networks have evolved to incorporate support for demanding applications.

TSN, DetNet, and 5G URLLC can meet the networking requirements of deterministic applications and provide ultra-reliable, low-latency connections through converged networks. TSN and DetNet (for wired) and 5G (for wireless) technologies are perfect partners in deterministic transmission networks. A certain degree of overall integration of these technologies is required to provide end-to-end connections that meet deterministic requirements.

For example, time synchronization on the wireless 5G domain and the wired TSN/DetNet domain is required because a common reference time is essential for deterministic endpoints regardless of the network technology connecting them. Providing limited low latency may also require integration between TSN, DetNet, and 5G, depending on the deterministic tools used in the deployment. End-to-end ultra-reliability adjusts to the characteristics of the necessary disjoint forwarding paths. The first step to support overall integration is done using an SDN-based approach, and the TSN, DetNet, and URLLC foundation technologies are ready, and their combined deployment is imminent.