How should the system's facade API gateway be designed?

As the popularity of his e-commerce website grew, the system welcomed some "uninvited guests". In the early hours of the morning, the number of searches for products and user interface calls in the system would increase sharply, and then return to normal after a period of time.

The characteristic of such search requests is that they come from a specific set of devices. When I introduced a throttling strategy for device IDs on the search service, the peak search requests in the early morning were controlled. However, not long after, the user service began to receive a large number of crawler requests for user information, and the product service also began to receive crawler requests for product information. Therefore, I was forced to add the same throttling strategy to both services. But this raises a problem: the same strategy needs to be implemented repeatedly in multiple services, and code reuse cannot be achieved. If other services also face similar problems, then the code needs to be copied, which is obviously not a good practice.

As a Java programmer, my first reaction is to separate the current limiting function into a reusable JAR package for the three services to reference. But the problem is that our e-commerce team not only uses Java, but also uses multiple programming languages ​​such as PHP and Go to develop services. For services in different programming languages, it is not possible to directly use the Java JAR package. This is why the API gateway is introduced. The API gateway can provide a cross-language solution, allowing services in different programming languages ​​to easily use the same current limiting strategy.

The role of API Gateway

API Gateway is not a single open source component, but an architectural pattern. It integrates some shared functions of some services and deploys them independently as an independent layer to solve service governance problems. You can think of it as the front door of the system, responsible for unified management of the system's traffic. From my point of view, API Gateways can be divided into two main types: ingress gateways and egress gateways.

• Ingress Gateway: The ingress gateway is located inside the system and is used to handle the traffic of external requests entering the system. It can perform some important tasks, such as routing requests to the correct service, identity verification, authorization, current limiting, request conversion, and response conversion. The ingress gateway helps manage the visibility of the system to the outside world and provides a consistent interface.
• Egress Gateway: The egress gateway is located inside the system and is used to handle traffic from internal services to external systems. It can perform tasks such as load balancing, caching, retries, error handling, etc. to ensure the reliability and performance of communication between internal services. Egress gateways are usually used to isolate internal services and external systems while providing a consistent interface.

These two types of API gateways can help manage the boundaries of the system and provide solutions in terms of traffic control, security, performance optimization, etc. to ensure the stability and scalability of the system.

The entry gateway is a type of gateway we often use. It is deployed between the load balancing server and the application server and has several main functions.

• The API gateway provides a single entry address for the client. It can dynamically route requests to different business services based on the client's request and perform necessary protocol conversion. In your system, different microservices may provide services to the outside world with different protocols: some are HTTP services, some have been transformed into RPC services, and some may still be Web services. The role of the API gateway is to shield the deployment addresses and protocol details of these services from the client, thereby facilitating client calls. The client only needs to interact with the API gateway, without directly communicating with services of various different protocols, making the overall calling process simpler.

• On the other hand, in the API gateway, we can embed some service governance strategies, such as service circuit breaking, degradation, flow control and diversion, etc.
• Furthermore, the implementation of client authentication and authorization can also be placed in the API gateway.
• In addition, the API gateway can also do some things related to blacklists and whitelists, such as blacklists and whitelists for dimensions such as device ID, user IP, and user ID.
• Finally, some logging can also be done in the API gateway, such as recording the access log of HTTP requests. When using a distributed tracing system, the requestId that marks a request can also be generated in the gateway.

picture

The function of the egress gateway is relatively simple, but it also plays an important role in system development. Our applications often need to rely on external third-party systems, such as third-party account login or payment services. In this case, we can deploy an egress gateway between the application server and the external system to uniformly handle tasks such as authentication, authorization, auditing, and access control of external APIs. This can help us better manage and protect communications with external systems and ensure system security and compliance.

picture

How to implement API Gateway

After understanding the role of the API gateway, the next step is to focus on the key aspects that need to be paid attention to when implementing the API gateway, as well as to understand some common open source API gateway solutions. This will enable you to make decisions more easily in actual work, whether considering developing your own API gateway or using an existing open source implementation.

When developing an API gateway, the first thing to consider is its performance. This is very important because the API entry gateway needs to handle all traffic from the client. For example, if the processing time of the business service is 10 milliseconds, and the processing time of the API gateway is 1 millisecond, then the response time of each interface will increase by 10%, which will have a huge impact on performance. The key to performance usually lies in the choice of I/O model. Here is just an example to illustrate the impact of the I/O model on performance.

In the 1.0 version of Netflix's open source API gateway Zuul, the synchronous blocking I/O model is adopted. The overall system can be regarded as a Servlet, which receives user requests, then executes configured authentication, protocol conversion and other logic in the gateway, and finally calls the backend service to obtain data and return it to the user.

In Zuul 2.0, the Netflix team made significant improvements. They transformed the original Servlet into a Netty server, using an I/O multiplexing model to handle incoming I/O requests. At the same time, they also improved the previous synchronous blocking method of calling backend services and adopted a non-blocking Netty client call. After these improvements, the Netflix team tested and found that the performance was improved by about 20%.

The functions of API gateway can be divided into two types: one is pre-defined and configurable, such as blacklist and whitelist settings, dynamic interface routing, etc.; the other needs to be defined and implemented according to specific business. Therefore, in the design of API gateway, it is very important to pay attention to scalability. This means that you can add custom logic to the execution link at any time, and you can also remove unnecessary logic at any time to achieve the so-called hot-swap function. This flexibility allows the behavior of API gateway to be customized according to different business needs, making it more versatile and adaptable.

Typically, we can define each operation as a filter and then string these filters together using the chain of responsibility pattern. The chain of responsibility pattern allows us to dynamically organize these filters and decouple them. This means that we can add or remove filters at any time without affecting other filters. This pattern allows us to flexibly customize the behavior of the API gateway and build filter chains according to specific needs without worrying about the mutual impact between filters.

Zuul uses the chain of responsibility pattern to process requests. In Zuul 1, it defines three types of filters: pre routing filter, routing filter, and after routing filter. Each filter defines the order of execution, and when the filters are registered, they are inserted into the filter chain in order. This means that when Zuul receives a request, it executes the filters inserted in the filter chain in order. This approach makes it easy to apply various filters during request processing to achieve different types of operations.

picture

In addition, it is important to note that in order to improve the gateway's ability to process requests in parallel, a thread pool is usually used to process multiple requests simultaneously. However, this also brings a problem: if the commodity service responds slowly, the thread calling the commodity service is blocked and cannot be released. Over time, the threads in the thread pool may be occupied by the commodity service, thus affecting other services. Therefore, we need to consider adopting thread isolation or protection strategies for different services. From my point of view, there are two main ideas to deal with this problem:

If your backend services are not split much, you can use different thread pools for different services, so that the failure of the product service will not affect the payment service and user service;

In the thread pool, you can protect threads for different services or even different interfaces. For example, if the maximum number of threads in the thread pool is 1000, you can set a maximum quota for each service.

Typically, the execution time of a service should be very fast, usually in the millisecond level. After using a thread, it should be quickly released back to the thread pool for subsequent requests. At the same time, the number of executing threads running in the system should be kept low and not increase significantly, so that the thread quota of the service or interface does not affect normal execution.

However, if a failure occurs, the response time of a certain interface or service will slow down, which may cause a significant increase in the number of threads. However, due to the thread quota limit, this will not have a negative impact on other interfaces or services. This approach helps isolate the failure and ensure that the failure does not spread to the entire system.

You can also combine these two methods in actual applications, for example, use different thread pools for different services, and set quotas for different interfaces within the thread pool.

How to introduce API Gateway into your system

At present, our e-commerce system has undergone a service-oriented transformation, in which a thin Web layer is introduced between the service layer and the client. This Web layer has two main tasks: First, it is responsible for aggregating the service layer interface data. For example, the interface of the product details page may aggregate data from multiple service interfaces, including product information, user information, store information, and user reviews. Second, the Web layer needs to convert HTTP requests into RPC requests and impose some restrictions on the front-end traffic, such as adding a blacklist of device IDs for certain requests.

Therefore, when making the transformation, we can first separate the API gateway from the Web layer, move the functions such as protocol conversion, current limiting, blacklist and whitelist to the API gateway for processing, and form an independent entry gateway layer.

There are usually two solutions: one is to set up a group of independent gateways to be responsible for traffic aggregation and timeout control, which we usually call traffic gateways;

The other is to extract the interface aggregation operation into an independent service layer. In this way, the service layer can be roughly divided into the atomic service layer and the aggregation service layer.

In my opinion, interface data aggregation belongs to business operations. Instead of implementing it in a general gateway layer, it is better to implement it in a service layer that is closer to the business. Therefore, I prefer the second solution. This approach helps to better divide responsibilities and make the system more modular and maintainable.

picture

At the same time, we can deploy the egress gateway service between the system and the third-party payment service and the login service. Originally, you would complete the encryption and signing of the data required by the third-party payment interface in the separated payment service, and then call the third-party payment interface to complete the payment request. Now, you put the encryption and signing of the data in the egress gateway, so that the payment service only needs to call the unified payment interface of the egress gateway.

After introducing the API Gateway, our system architecture became as follows:

picture

Emphasis points:

API gateways can be divided into two categories: ingress gateways and egress gateways. Ingress gateways have multiple functions, including isolating clients and microservices, providing protocol conversion, implementing security policies, performing authentication, limiting current, and implementing circuit breakers. The main function of the egress gateway is to provide a unified exit for calling third-party services. It can perform unified authentication, authorization, auditing, and access control tasks to ensure communication security and compliance with external systems.

The key to API gateways is the realization of performance and scalability. To improve the performance of the gateway, you can use a multi-channel I/O multiplexing model and a thread pool to concurrently process requests. To improve the scalability of the gateway, you can use the chain of responsibility pattern to organize and manage filters so that you can easily add, remove, or modify filters to meet different needs.

The thread pool in the API gateway can isolate and protect different interfaces or services, which can improve the availability of the gateway;

The API gateway can replace the Web layer in the original system, move the protocol conversion, authentication, current limiting and other functions in the Web layer into the API gateway, and sink the service aggregation logic to the service layer.

API gateways not only provide a convenient way to call APIs, but also separate some service governance functions to achieve better reusability. Although there may be some sacrifices in performance, these performance losses are usually acceptable when using mature open source API gateway components. Therefore, when your microservice system becomes more and more complex, you can consider using the API gateway as the facade of the entire system to simplify the system architecture and provide better maintainability.