Security monitoring and observability capability building based on system operation experience

Observability refers to the ability to measure the working status of a network system through its operational data output (such as logs, indicators, and tracking). Initially, observability was defined as an external monitoring attribute of a system and was first proposed in the field of industrial control and production. Managers use observability to view various information data hidden in the system. For example, the operators of a water treatment plant cannot directly know the water flow state, flow pattern, and water quality in the water pipe from the outside. At this time, by adding observability tools (such as flow meters, sensors, etc.) to the pipeline and connecting these tools to the dashboard, the operators can fully understand the water flow in the pipeline and make timely adjustments based on the situation, thereby significantly improving the operational reliability of related systems.

As the complexity of modern network infrastructure continues to increase, various potential faults have become more difficult to identify, and the importance of improving the observability of network systems for security protection has become increasingly prominent. Its value can be reflected in the following aspects:

1. Help the security team plan and develop network security protection processes;

2. Help operation and maintenance personnel discover and solve network security problems in advance;

3. Improve the efficiency and accuracy of detecting and responding to network security incidents;

4. Improve the service life and operation stability of network equipment systems.

The difference between observability and traditional security monitoring

Before the term “observability” came into being, we often used the concept of “monitoring”, but there is a big difference between the two.

Traditional monitoring modes can be used to trace back when something went wrong, but observability allows security teams to know the causes of security incidents. Monitoring tracks the overall health of network security devices and summarizes operational data about system performance based on access speed, connectivity, downtime, etc. Observability provides fine-grained analysis and contextual insights into failure modes to delve into the "what" and "why" of abnormal application operations and further identify security risks. Monitoring only provides answers to known problems or failures, while software systems with observability can satisfy developers' exploration of unknown network security risks.

It can be said that the traditional monitoring mode is like looking at the operation of the security system from an outsider's perspective, while observability is a self-examination of the security system, standing from the perspective of the developer to explore how the network security equipment should properly display its own status. On the other hand, monitoring can also be said to be a subset of observability. In addition to providing more traditional monitoring measurements, observability usually also includes the ability to observe system application experience and business processes.

In general, observability is a service that can provide value to all parties. It can not only monitor enterprise network security applications and platforms, but also monitor the enterprise's business processes, development team efficiency and even the execution of various business plans. In addition to cloud environments or data centers, observability can also be extended to the enterprise's digital business applications, providing enterprises with all-round observation and perception from function requests to customer satisfaction.

Building observability capabilities

Security monitoring and observability capabilities based on digital technology application experience can be achieved through the Observability River technology, which can help users understand what types of network operation data can be collected and need to be paid attention to, and start from simple data streams and continue to accumulate until a comprehensive observability solution data set is formed for users to use. The key points of building the Observability River are:

01 Start with basic security monitoring

All important information related to the reliable operation of the enterprise network platform, such as user access and audit logs, the number of failed logins of suspicious accounts, etc., may contain the attack characteristics of attackers, and this information needs to be paid special attention.

02 Monitoring network security equipment, cloud platform indicators and logs

The observability long-term technical solution requires adding log information and operating parameters about the device central processing unit (CPU), memory bar (RAM) and storage system, as well as cloud platform indicators about the execution speed of Lambda functions, etc.

03 Understand the operational performance of network security equipment

Strengthen performance monitoring of network security device applications, including function execution speed, database search time, and even tracking data based on OpenTelemetry or similar products. (OpenTelemetry is an observability project of the Cloud Native Computing Foundation, which aims to provide standardized solutions in the field of observability, solve standardization issues such as data models, collection, processing, and export of observation data, and provide neutral services.)

04 Strengthen the collection of employee behavior information

The two functions of Real User Monitoring (RUM) and Synthetic User Monitoring (SUM) can help enterprises further understand the user's behavior path and the actual use of the product, and can improve the user experience based on these behavior analyses. Synthetic user monitoring refers to submitting a page that needs to be audited in a simulated scenario, running the page through a series of tools and rules, and evaluating the performance indicators of the page after running to obtain relevant audit reports. Real user monitoring, on the other hand, is the analysis and monitoring of the data information accessed by users on real pages.

05 Ability to process data

The purpose of collecting data is to process it correctly and visualize the long river of data. Graphical interpretation is the best way to effectively visualize data. For example, when users visualize employee behavior information and key sensitive data of the enterprise, the dashboard will clearly show the potential security risks and security shortcomings of the enterprise, which is of great significance to the stable and safe development of digital business work.

06 Add alarm function

When a security incident occurs or there is a potential security risk, the enterprise's internal network security team should be notified in a timely manner and make appropriate responses based on the contextual data statistics and alarm results.

How to choose the right observability solution?

Here are some key questions to consider when choosing an observability security solution:

1. Does the enterprise have a team capable of supporting observability security solutions?

Before building an observability security solution, enterprises must first understand whether they have the technical level and ability to run the service. A network security team responsible for observability solutions needs to invest a lot of financial costs and management energy in terms of establishment, operation, work and development. To some extent, directly purchasing a mature third-party security monitoring system will be more cost-effective. ​

2. Are users willing to have monitoring data exist outside of their “own environment”?

When running an observability solution, you generally have three options:

Within the firewall: This is the safest option both in terms of user perception and in theory. It ensures that data is always within the boundaries of the network environment, and there is less risk of monitoring data (and details of infrastructure layout) leaking into the public domain. This is also the solution chosen by most users.

Software as a Service (SaaS): Software as a Service (SaaS) is rapidly becoming the main way for enterprises to purchase monitoring solutions, especially in cloud environments. It transfers difficult tasks such as alarms and operation and maintenance to third parties, reducing the technical threshold required of internal enterprise personnel. However, in this way, the enterprise's monitoring data may be stored on multiple platforms, increasing the risk of leakage.

Cloud monitoring: Cloud providers such as Amazon and Google often provide fully managed monitoring platforms (such as CloudWatch), as well as managed services (such as AWS Managed Grafana and AWS OpenSearch Service) to help enterprises build observability capabilities. The pros and cons of cloud monitoring may be between the above two options, and the cost is lower than the complete SaaS product.

3. What needs does the company itself want to observe and analyze?

Many companies are looking for the most feature-rich and market-popular monitoring platforms and observability security solutions. However, in actual applications, less than half of the product functions may be actually utilized. Such high investment is not reasonable with actual needs.

4. Observability solutions support programming languages

When enterprises are shopping for network observability solutions, they need to pay attention to which development language environments these tools are more suitable for. For example, AppDynamics is a very popular application performance monitoring tool in enterprise organizations. If the enterprise employees are very good at Java or C#, then it may be fine. But if the employees are only good at Python or Rust, the results will be unsatisfactory because it has poor support for these languages.