A white-label vendor's perspective: The evolution of minimalist switch NOS

Traditional switch operating systems (NOS) are a relatively closed field to the public. With the rapid growth of white-label switches, NOSs have been open sourced, and NOS developers have expanded from only equipment vendor engineers to Internet, carriers, and cloud computing practitioners.

The role of NOS is to run the services in the network on the switch according to the will of the manager. Therefore, NOS first needs to provide an interface to the manager or controller; then it needs to run protocol operations and interact with other switches in the network on the protocol plane; thirdly, it needs a hardware interface to adapt to the switching chip, fan power supply and other on-board hardware. As shown in the figure below, we can split NOS into three core functional modules and an infrastructure module.

Management interface, including traditional CLI, SNMP, WEB functions. SDN introduced Openflow, NET-CONF, OPEN Config, Restful API functions, etc.

Protocol application modules, including the second-layer protocol modules STP, LLDP, M-LAG, the third-layer protocol modules OSPF, BGP, VRRP, etc., as well as application modules such as DHCP and NTP, and Openflow Agent in the SDN era, including OVS;

The hardware interface includes management interfaces for docking switch chips, power supplies, and fans;

A NOS that has the above three core functions is considered a suitable NOS. However, to see whether a NOS is good or not, we must also look at the "infrastructure", which is the skeleton and muscle of NOS. The robustness and ductility of NOS are determined by it. The evolution of NOS is actually the evolution history of this infrastructure.

We divide the process into two stages. The first stage is the competition period among giant manufacturers such as Cisco, Juniper, and Arista. During this period, core technologies were concentrated in the hands of equipment manufacturers, and it was a stage of technology accumulation. The second stage is the era of new open source forces such as OPS, Sonic, and OPX. Internet manufacturers who like to play subversion participated with new SDN needs.

Battle between giants

Original architecture

The IOS architecture introduced in "Inside Cisco IOS Software Architecture" is shown in the figure above. The framework also includes a software forwarding module, which shows that it is a very early version. By analyzing Cisco IOS through the blue box, we can see that IOS meets the three elements of NOS: management interface, protocol application module, and hardware interface. However, the basic architecture is still relatively primitive, and the management interface and protocol application module are not separated. This architecture is more about solving the problem of whether there is a problem or not, and the focus at that time was more on the business module.

Modular architecture

The JunOS architecture introduced in "JunOS OS for Dummies" is shown in the figure above, which includes management interfaces, protocol application modules, hardware interfaces, and proposes the concept of modular architecture.

The modular architecture of Junos OS allows individual control plane processes to run in their own module (also sometimes called a daemon). Each module has specified processing resources and runs in its own protected memory space, avoiding the processing conflicts that can occur in other platforms.

This is also a relatively early architecture, but through this architecture we can clearly see that the management interface is separated from other modules, and there is already some separation of control and forwarding in it. However, this evolution is not revolutionary, but more like an improvement from food and clothing to a well-off life.

database schema

The architecture diagram of Arista's EOS is shown above. The most amazing thing about EOS is its database architecture. SysDB is a Key-Value in-memory database. The core highlight of Arista is that it can natively solve process-level failures. The process is as follows:

It can be seen that when a process fails, the switching chip continues to forward, and after the process recovers, it re-obtains the status from SysDB and continues to run. This function not only ensures the stable operation of the business, but also enables the single-process upgrade function. I remember that a typical Arista DEMO was to kill the running STP and perform a single-process upgrade. Because the STP status is stored in SysDB, the business layer can be unaware of the recovery of STP.

The evolution of database architecture is a major change, which subverts the traditional architecture of defining data structure and then communicating messages between processes. Developers can use the same ideas as developing general software for development, and the data of NOS is visualized, which greatly reduces the difficulty of locating and solving problems.

Cisco also implements HA through a Key-Value memory database on NX-OS, as shown below:

Cisco's NX-OS clearly separates management and protocol applications (in no particular order), implements a lightweight Key-Value in-memory database, and completes HA Infrastructure. It is also admirable that Cisco continues to evolve despite the burden of a huge protocol stack.

New Forces

With the rapid development of SDN, the white-label industry has spawned a number of open source NOS. These emerging NOS stand on the shoulders of giants and are based on database architecture. OPS chose OVSDB, Sonic and OPX chose Redis. OVSDB and Redis are both Key-Value in-memory databases.

But this did not satisfy the new forces. SDN requires faster, more flexible, larger scale, and better scalability. The NOS development cycle of the giants was still too long, and the upgrade was still a bit inconvenient. Internet developers who were used to using dynamic languages ​​said they could not accept it. The most painful demand of Internet users in data centers for NOS is how to complete version iteration without traffic perception, and how to upgrade versions more conveniently and efficiently.

Database architecture + container architecture

Microsoft, which is engaged in public cloud, naturally thought of reconstructing NOS through virtualization and applied container technology to NOS. Container technology can be simply understood as: from the perspective of Linux operating system, container is a process, and container itself is a lightweight virtual machine. Sonic runs various processes, such as BGP, in containers, natively solving the modularization problem raised by JunOS. More importantly, it cooperates with the database architecture, and changes the upgrade of a single process to the upgrade of the container, which cleverly uses mature and common technologies to solve traditional problems.

The entire infrastructure development history is summarized as follows:

After generations of evolution, NOS is no longer a specialized operating system with a complex structure that requires hackers to locate problems and a productization cycle similar to that of chips. Modern NOS has evolved in architecture to use general databases, container virtualization technology, and support high-speed iteration. In a sense, can equipment vendors also call themselves Internet companies?

About the author: Cheng Wei, System Engineer Manager, Suzhou Shengke