Ten techniques for API protocol design

In this digital age, our daily lives are filled with interactions between various applications and systems. Whether it is social media, online shopping or smart home devices, they all need to use APIs (application programming interfaces) to achieve data transmission and communication. However, behind these seemingly simple operations, there are complex protocols hidden.

API protocols contain a set of rules and standards that define how different systems communicate and share data. They act as a bridge between different applications, enabling them to understand and communicate with each other. The design and implementation of API protocols need to take into account factors such as security, reliability, and efficiency to ensure accurate data transmission and normal operation of the system.

To gain a deeper understanding of the world of APIs, here are 10 common API protocol designs.

1. REST

REST is the most popular API development technology in modern web development. It provides a stateless architecture for data transfer. The client request contains all the details required to fulfill the request, and the server does not retain the client's state.

picture

In a RESTful API, each resource can be identified and accessed through a unique URL. Clients can perform various operations such as getting resources, creating new resources, updating existing resources, or deleting resources by sending HTTP requests. The design of RESTful API follows some basic principles, such as resource expression, client-server architecture, statelessness, and caching. REST API supports local HTTP cache headers and uses HTTP methods (POST, GET, PUT, PATCH, and DELETE) to manipulate data. Anyone can easily start using REST, it is simple, and the learning curve is smooth. It also has good readability and maintainability because it uses standard HTTP methods and status codes to represent different operation results.

However, RESTful API also has some limitations. Due to its stateless nature, each request needs to contain all necessary information, which may result in a large amount of data transfer. As the application expands, the number of endpoints increases dramatically, and it is not easy to update the database schema or data structure. In addition, for complex business logic, RESTful API may not be flexible enough and requires additional architecture and design to meet the needs.

If you don’t have any specific requirements, REST is the best choice. For example, if you are a novice developer, then using REST is a perfect match as it has a shallow learning curve. Also, it has a large ecosystem where you can easily find solutions to any problem. Also, it is better to use REST when dealing with many requests and limited bandwidth. In such cases, you can use its caching support to improve performance.

2. 𝗚𝗿𝗮𝗽𝗵𝗤𝗟

GraphQL is a data query language introduced in 2015. It allows developers to pinpoint and fetch the exact data they need. Compared to REST, GraphQL is a client-driven approach where the client decides what data is needed, how to fetch it, and in what format. It also solves the problem of over-fetching and under-fetching, as the client can pinpoint the data it needs.

GraphQL is seen as a new way of thinking for APIs. GraphQL is both a query language for APIs and a runtime environment that satisfies your data queries. GraphQL provides a complete and easy-to-understand description of the data in the API, and also makes it easier for the API to evolve over time. GitHub is one of the largest companies using GraphQL. In 2016, GitHub switched from REST to GraphQL, which greatly promoted GitHub's rapid growth. There is already a lot of information on the Internet about GraphQL, so I won't describe it too much here. For details, please refer to GraphQL.org.

picture

In GraphQL, the type system is used to describe the capabilities of the GraphQL Server and to determine whether a query is valid. The type system also describes the input types of query parameters and checks the validity of parameter values ​​in the GraphQL Runtime. A GraphQL service is created by defining types and fields on types, and then providing resolution functions for each field on each type. For example, in Github GraphQL Server, the viewer field is used to describe the information of the currently logged in user, and the viewer type is User. Once a GraphQL Server is started, it can accept GraphQL queries, validate and execute the query. The GraphQL Server first checks the query to ensure that it only references defined types and fields, and then runs the specified resolution function to generate the result.

Compared with ordinary REST API, the strong type system is one of the most attractive features of GraphQL. The GraphQL type system is its foundation and everyone must comply with it, which plays an important role in forming a unified understanding of API interface descriptions. In GraphQL, the GraphQL Runtime determines the capabilities that the GraphQL Server can provide, and the consumer has the final say on what data to obtain. The client (consumer) can participate more actively in the entire data interaction process on a more equal footing. With the help of GraphQL's type system, the client can get what it needs more freely according to its own needs without being restricted by the server.

GraphQL not only changes the discourse power of both parties in communication, but also allows the client to accurately predict the response of the server. For API version control, GraphQL draws on the @deprecated annotation in other languages.

The downsides of GraphQL are that queries can be complex and it lacks built-in caching support. Learning GraphQL is somewhat challenging compared to REST, and it does not support file uploads by default.

Even so, careful analysis is still required when determining whether to use GraphQL technology, and GraphQL should not be adopted for the sake of novelty. GraphQL is an excellent choice for querying databases with many records. You can use GraphQL to eliminate extra reads of data and retrieve only necessary data in a specific format to improve application performance. In addition, GraphQL is well suited for situations where data needs to be aggregated from multiple resources. GraphQL can also be used when you don't fully understand how the client will use the API. With GraphQL, there is no need to define a strict contract in advance. Instead, the API can be built incrementally based on client feedback.

3. 𝗴𝗥𝗣𝗖 (google 𝗥𝗲𝗺𝗼𝘁𝗲 𝗣𝗿𝗼𝗰𝗲𝗱𝘂𝗿𝗲 𝗖𝗮𝗹𝗹𝘀)

gRPC is an open source remote procedure call (RPC) framework introduced by Google in 2016, using Protocol Buffers (protobuf) as the interface description language. It is a lightweight solution and provides maximum performance using minimal resources.

picture

gRPC follows a contract-based communication approach. It requires both the client and the server to have a contract before they can start communicating. GRPC uses Protobuf (a declarative language) to create contracts, which generates compatible code for the client and server in the selected language. gRPC provides multi-language support, including but not limited to C++, Java, Python, Go, Node.js, etc. This allows developers to build mutually compatible services and clients in different languages.

gRPC supports four communication methods:

• Simple request/response: The client makes a single request to the server, and the server sends a single response.
• Client streaming communication: The client sends a series of requests to the server, then sends a message to notify the server that the stream has ended, and finally, the server sends a response.
• Server streaming communication: The client makes a single request to the server. The server then sends a stream of messages to the client.
• Bidirectional streaming communication: gRPC supports bidirectional streaming, allowing multiple messages to be sent simultaneously between the client and the server. This bidirectional communication mechanism makes gRPC very suitable for real-time applications and streaming data processing.

gRPC uses HTTP/2 as the underlying transport protocol, which brings higher performance and efficiency. HTTP/2 supports features such as multiplexing, header compression, and binary transmission, which improves the speed of communication and resource utilization. It serializes data in binary format, supports full-duplex communication, and is also capable of load balancing. gRPC has a rich ecosystem, including libraries and tools that support various languages. It also integrates with popular open source projects such as Kubernetes, Envoy, Istio, etc. to provide a more comprehensive solution. gRPC

Overall, gRPC is a powerful and efficient RPC framework suitable for building distributed systems, microservice architectures, and applications that support real-time communication. Its design philosophy focuses on performance, scalability, and cross-language support, making it widely used in modern application development.

4.𝗪𝗲𝗯𝗵𝗼𝗼𝗸𝘀

Webhooks are a powerful technology that enables instant updates and notifications between systems. By using HTTP callback mechanisms, Webhooks can ensure that data between systems remains in sync.

picture

When using Webhooks, an application (service provider) usually provides a registration interface for another application (service consumer) to register the events it is interested in. After successful registration, the service provider will send an HTTP request to the callback address provided by the service consumer when the relevant event occurs to trigger the corresponding action.

The working principle of Webhook is very simple. When an event occurs, such as a user submitting a form, publishing a new article, or updating a database, the server will send an HTTP POST request to a predefined URL. This URL can be an API endpoint of a third-party application or a server you have built yourself. After receiving the request, the server will execute the corresponding logic and return the result to the caller through an HTTP response.

In this way, Webhook enables real-time communication and data synchronization between systems. It eliminates the need for polling and periodic requests, reducing network traffic and latency. At the same time, Webhook is also highly scalable and flexible, and can adapt to a variety of different application scenarios. Whether developing e-commerce websites, social applications or IoT devices, Webhook is a very useful tool.

5. Event sending on the server side——𝗦𝗦𝗘(𝗦𝗲𝗿𝘃𝗲𝗿-𝗦𝗲𝗻𝘁 𝗘𝘃𝗲𝗻𝘁𝘀 )

SSE is an HTTP-based communication protocol that allows the server to push real-time updated data to the client. Unlike traditional polling or long polling, SSE achieves two-way communication of data by establishing a persistent connection. Once the connection is established, the server can push data to the client through the connection without the client initiating a request again. For example, the client first sends an HTTP GET request to the server to establish a persistent connection. The server then keeps the connection open and pushes new data to the client at any time. The client can display or process the data in real time by parsing the event stream sent by the server. For Web-based applications, browser support for SSE is as follows:

picture

Since SSE is based on the HTTP protocol, it is compatible with various programming languages ​​and platforms. Whether it is JavaScript, Python or Java, SSE can be used through the corresponding library or framework. In addition, SSE also has good scalability and performance advantages, and is suitable for processing large amounts of real-time data updates.

Using Server-Sent Events (SSE), you can experience the convenience of real-time data updates. This lightweight protocol is very suitable for transmitting dynamic content and instant information. Whether developing real-time chat applications, news aggregation websites or monitoring dashboards, SSE is a very useful tool.

6. EDI (𝗘𝗹𝗲𝗰𝘁𝗿𝗼𝗻𝗶𝗰 𝗗𝗮𝘁𝗮 𝗜𝗻𝘁𝗲𝗿𝗰𝗵𝗮𝗻𝗴𝗲)

𝗘𝗹𝗲𝗰𝘁𝗿𝗼𝗻𝗶𝗰 𝗗𝗮𝘁𝗮 𝗜𝗻𝘁𝗲𝗿𝗰𝗵𝗮𝗻𝗴𝗲 (𝗘𝗗𝗜) is a standard for exchanging business documents. It enables smoother exchange of business information between different systems by standardizing the structure and content of documents, that is, standardizing the format of API communication content. By using the 𝗘𝗗𝗜 standard, enterprises can simplify business processes, increase automation, and reduce transaction costs.

picture

EDI can achieve interoperability through API. EDI converts business documents between enterprises into standard data formats, which are then converted into data formats required by other applications. EDI can integrate business documents between enterprises with data within the enterprise, while API can integrate data between different applications, thereby achieving data sharing and circulation. EDI can automatically process business documents, and API can automatically process data exchange and communication between applications, thereby automating business processes.

For information security, EDI can use security measures such as encryption and digital certificates, while API can use security measures such as access control and identity authentication to ensure information security. At the same time, I can achieve data mining and analysis through data analysis. EDI can aggregate and analyze business documents between enterprises, and use API to aggregate and analyze data between different applications, thereby achieving data mining and analysis.

7. Event-driven design for APIs

"Event-Driven Architecture (EDA)" refers to event-driven architecture. In the field of API, it means event-driven architecture designed for API.

The event-driven architecture emphasizes communication and collaboration between components in the system through events. In this architecture, components can be independent services, modules, or entire systems. Events are things that happen in the system, which may be state changes, user actions, external triggers, etc. When an event occurs, components in the system can publish (or broadcast) the event, and other components interested in the event can subscribe to these events and respond.

picture

Event-Driven Architecture applies event-driven thinking to API design and interaction. This architectural style is very useful in dealing with asynchronous, distributed, and real-time applications. This architecture emphasizes loose coupling between API components through event publishing and subscription mechanisms. API components can be producers (components that publish events) or consumers (components that subscribe to and respond to events). DA makes API communication asynchronous, allowing components to continue execution without directly waiting for a response. This helps improve system performance and scalability.

The event-driven architecture is suitable for scenarios that require real-time response, such as real-time data updates, notification push, etc. Through the event distribution mechanism, the system can respond to changes more immediately. The event-driven feature facilitates communication and collaboration between microservices. Generally, the API gateway can act as a distributor of events, responsible for sending events to the corresponding subscribers. This helps to centrally manage the flow and processing of events. By using events to drive API interactions, the system can better adapt to dynamic changes and asynchronous communication needs between different components.

8. 𝗪𝗲𝗯𝘀𝗼𝗰𝗸𝗲𝘁𝘀

WebSocket enables two-way communication between the client and the server, providing a real-time and efficient way for both parties to interact. Through WebSocket, a persistent connection can be established between the client and the server, allowing both parties to send and receive data at any time. This two-way communication mechanism greatly expands the traditional request-response model, providing developers with a more flexible and instantaneous means of interaction.

The WebSocket protocol allows both parties to communicate in real time through a single socket by creating a persistent connection between the client and the server. Unlike the traditional HTTP request-response model, WebSocket does not need to establish a new connection for each communication, thus reducing communication overhead and latency. This is very beneficial for scenarios such as real-time applications, online games, and chat applications.

picture

In WebSocket, the communication between the client and the server is event-based. Once the connection is established, either party can send messages to the other party asynchronously, and the other party can receive and respond immediately. This real-time two-way communication mechanism greatly improves the interactivity of the application, allowing users to enjoy a smoother application experience without perceiving delays.

In general, the introduction of WebSocket has enabled Web applications to make significant progress in processing real-time data, push notifications, and establishing interactivity. Through WebSocket, two-way communication between clients and servers has become an indispensable part of modern Web applications, providing developers with more creative and real-time possibilities.

9. Simple Object Access Protocol (SOAP)

SOAP is a Web service communication protocol that defines the format of Web service messages. SOAP encoding is used to tell the SOAP runtime environment how to convert data structures such as Java into SOAP XML.

picture

The readability and extensibility of XML enable SOAP to flexibly adapt to different application scenarios. Common Web service specifications include:

• Web Services Security (WS Security): Standardizes the way messages are secured and transmitted using unique identifiers called "tokens".
• WS-Reliable Messaging: Standardizes error handling for messages transmitted across unreliable IT infrastructures.
• Web Services Addressing (WS Addressing): Packaging routing information as metadata in the SOAP header, rather than maintaining such information deep in the network.
• Web Services Description Language (WSDL): describes the functionality of a Web service and where the service begins and ends its work.

SOAP is protocol independent and can run on various network protocols, such as HTTP, SMTP, etc. The most common use of SOAP is on HTTP, encapsulating SOAP messages in the HTTP protocol for transmission. SOAP and WSDL indicate the communication between a Web service and its clients. SOAP supports multiple message interaction modes, including one-way messages, request-response mode, and asynchronous messages. This makes it suitable for different application scenarios, from simple data queries to complex business processes.

The transmission of SOAP messages can use security protocols, such as HTTPS, to ensure confidentiality and integrity during transmission over the network. In addition, SOAP can also be used in conjunction with other security standards (such as WS-Security) to provide more advanced security support.

10. 𝗠𝗲𝘀𝘀𝗮𝗴𝗲 𝗤𝘂𝗲𝘂𝗶𝗻𝗴 𝗧𝗲𝗹𝗲𝗺𝗲𝘁𝗿𝘆 𝗧𝗿𝗮𝗻𝘀𝗽𝗼𝗿𝘁 (𝗠𝗤𝗧𝗧)

MQTT is a lightweight, open message queue transport protocol designed for inter-device communication in low-bandwidth, high-latency or unstable network environments. Its design focuses on resource efficiency, making it ideal for devices and applications running in constrained environments. Its protocol header is small and has low communication overhead, making it suitable for embedded systems and mobile devices.

The "MQ" in "MQTT" is derived from IBM's MQ (then called MQSeries) product line, where MQ stands for "Message Queue". Despite the name, however, the protocol does not use message queues; instead, it provides publish-subscribe messaging: a device publishes a message on a specific topic, and all devices subscribed to that topic receive the message. Its main applications include sending messages to control outputs, and reading and publishing data from sensor nodes.

picture

MQTT provides different Quality of Service (QoS) levels to meet the needs of different application scenarios. QoS levels include at most once, at least once, and exactly once.

MQTT supports persistent sessions. Clients can choose to create persistent sessions, which allows the server to retain its subscription information after the client disconnects. This helps ensure that the client can receive previously missed messages when it reconnects.
MQTT supports basic authentication and transport layer security, but typically needs to be used in conjunction with other security mechanisms, such as TLS/SSL.

In summary, MQTT is a flexible, lightweight, easy-to-implement, reliable and efficient protocol, especially suitable for IoT and embedded systems that require real-time, reliable communication.