[Footprints] In addition to creating a highly available application environment, what else does FreeWheel's operation and maintenance do?

[51CTO.com original article] Perhaps in the eyes of many people, many practices of FreeWheel are unexpected: the company's business, sales, and marketing are all in Europe and the United States, but the technical R&D team is mainly in China; in the IT workplace where female programmers are as scarce as pandas, FreeWheel's Beijing R&D center with nearly 300 employees, about 40% of the employees are women; companies all advertise that they are eager to recruit talents, but how many companies are like FreeWheel that would specially build a new office in New York for the engineer they like in order to retain him?

If FreeWheel's external "mysterious willfulness" attracts the attention of many job seekers, then what attracts reporters is the company's internal IT architecture and operation and maintenance. This multinational company with offices in Europe, the United States, and China, whose advertising platform is used by 90% of mainstream TV media and operators in the United States, how to ensure efficient collaboration? FreeWheel has been established for ten years. From the cumulative annual advertising broadcast volume of 1 million when it was first established to nearly 1 billion advertisements per day, how does the operation and maintenance department ensure a stable application environment for the product? As a high-tech company that is very sensitive to emerging technologies, how to choose the most suitable technology products for itself?

[[187197]]

In the past month, the reporter interviewed FreeWheel co-founder and beautiful CTO Diane Yu and Vice President of Operations and Maintenance Vito Leung. Through their sharing, the series of questions mentioned above were answered and a real FreeWheel was restored. In the reporter's opinion, this company is full of vitality, has no historical baggage, and is not short of code experts. Their clear positioning of products, rational judgment and innovation of new technologies, and prejudgment and calmness in IT planning are all worth learning from. The reporter also hopes that through this article, more references can be provided to multinational companies, high-tech companies, and innovative companies that are facing IT operation and maintenance confusion.

A research and development center that goes against the grain

People familiar with FreeWheel know that the R&D center of this multinational company has been located in Beijing since its inception. Some people attribute this to the fact that Diane was born and raised in Beijing and has a strong attachment to her hometown. In fact, this is not entirely true.

[[187198]]

Diane worked in the United States for nine years and met many Chinese programmers. She discovered early on that Chinese engineers have solid basic skills, work hard and have outstanding abilities, but often suffer from language problems. Another disadvantage is that Chinese IT talents are distributed in different departments of different companies. There is no formed team, and it is impossible to support and help each other, making it difficult to improve together. At that time, she was thinking, why can't we recruit the best engineers in China to form a research and development team? Later, she met the other two founders of FreeWheel and proposed the idea of ​​establishing a research and development center in Beijing, which was quickly accepted.

FreeWheel R&D center recruits mostly top students from top universities such as Tsinghua University, Peking University, Chinese Academy of Sciences, and Harbin Institute of Technology. At the beginning of the team formation, in addition to the obvious disadvantage in language, the difference in thinking between China and foreign countries also took a while to adjust. Many tiny details that Diane had never thought of before could also cause misunderstandings. For example, when the R&D team sends emails, the writing of communication time is often marked according to the Chinese custom of "year-month-day", while the standard custom of the United States for time is "month-day-year", so often when the emails are sent from China, the American team is confused and can't figure out the meeting time.

But soon, after a painful "running-in period", the Chinese R&D team exploded with amazing R&D strength. On the one hand, the team is very thoughtful and has strong R&D capabilities, and can quickly respond to the needs of the US product department. On the other hand, one-third of the FreeWheel R&D team has experience of "sitting in" and rotating in the United States or Europe, and has close contact with product applications and customer service, and has a clearer understanding of the focus and direction of R&D. Of course, another benefit that needs no further explanation is the rapid improvement of English communication skills. Facts have proved that her decision was right. Now her partners have told customers or investors on various occasions that the reason why FreeWheel can get to where it is today is inseparable from Diane's decision to set up a R&D center in Beijing. "There was a very nervous stage, but I am very happy that the facts proved that I was right."

Trial and error at the lowest cost

FreeWheel divides the operation and maintenance team of more than 60 people into several small teams, including those responsible for network, basic operation and maintenance, and product application operation and maintenance, etc. Managing the operation and maintenance team is one of Vito's important responsibilities. The entire operation and maintenance team mainly undertakes three things: first, learning and drawing on external emerging technologies; second, keeping pace with the company's product development and providing support at any time; third, communicating and coordinating with different departments to meet their needs.

[[187199]]

These three things are easy to say, but not easy to put into practice. Take the first thing for example. Vito needs to solve various challenges that FreeWheel encounters in the process of IT development, which requires him to find the most effective solution with the least trial and error cost. He gave the reporter two examples:

★ The road to database selection

In the Internet advertising industry, accurate advertising based on user information and historical interest behavior has become a basic requirement. In order to meet this requirement, it is necessary to build a user database system that supports high concurrency, low latency, scalability, and high availability. This is a very big technical challenge faced by many real-time advertising systems.

FreeWheel's user data has grown from tens of thousands of records and tens of GB in the beginning to 600 million records and TB in size at present. Up to 100 million records are updated every day, requiring millisecond-level data access performance across data centers to ensure the real-time delivery of digital advertising.

Vito told 51CTO reporters that based on the above business needs, FreeWheel has made many attempts, explorations and improvements in product selection, programming interfaces, software design, operation and maintenance of user databases.

In the initial stage, the amount of data was small. Based on access performance considerations, they first tried memcached, a very popular open source software product in the industry, to achieve full memory access, which achieved very good results. As the amount of data continued to increase, full memory storage could not meet the needs. Next, R&D and operation and maintenance colleagues began to evaluate leveldb and made some special customizations based on FreeWheel's business needs, thereby achieving persistent storage of data on disk and getting rid of the limitation of memory capacity.

However, problems and challenges followed. From the perspective of operation and maintenance, many problems could not be solved well, such as difficulty in achieving high availability, high cost of adding nodes, large delay across data centers, etc. At this time, FreeWheel began to actively seek and try more software products and solutions, and finally chose a product like Aerospike. It has outstanding advantages in API implementation, data access performance, namespace definition, low-latency data synchronization, SSD hard disk access optimization, high availability implementation, and operation and maintenance friendliness, which not only greatly improved the response speed of FreeWheel's advertising delivery system, but also controlled the average delay of synchronization across data centers to milliseconds (ms).

★ Evolution of network file systems

At FreeWheel, the operation and maintenance team uses the NFS (Network File System) solution to achieve data sharing between multiple systems and servers. NFS is a widely used and mature shared file system under the Linux/Unix operating system, which can share resources between computers through the TCP/IP protocol. Driven by the operation and maintenance team, the application of NFS in FreeWheel has gone through several stages.

In the initial business stage, they only used one NFS server to provide all data sharing services for front-end/back-end products, including advertising creative files, user data reports, advertising logs, etc.

Later, with the continuous upgrading of FreeWheel products and the expansion of business models, the amount of data and the read and write throughput increased, and a single NFS server could no longer meet the needs. Therefore, the new solution was to split the existing data resources according to business logic, distribute them to multiple NFS servers, and isolate data resources from the perspective of business logic. At the same time, this also required colleagues in the product and development departments to adjust the application design to adapt to this improvement.

After basically solving the capacity and performance problems, the operation and maintenance team further improved the high availability and scalability of multiple NFS servers. After research and comparison, they finally chose Redhat Cluster Suite as the solution, which has been realized from 2-node mutual backup to 4-node many-to-many mutual backup, and finally to the current 7-node many-to-many mutual backup architecture, thus providing strong support for the system in terms of shared resource read and write performance, service availability, system redundancy, horizontal expansion capability, and other aspects.

Collaboration pain points: How to synchronize the United States, Europe, and China?

[[187200]]

During the interview, Vito repeatedly said that as an operation and maintenance team that needs to work collaboratively in multiple locations around the world, what bothers him most is not product and business issues, but how to make operation and maintenance teams in different regions have consistent goals and priorities.

FreeWheel's offices in the United States, Europe, and China have different main functions. Some offices are more inclined to communicate with users, while others are more inclined to engineering. Therefore, the operation and maintenance teams in different offices face and need to solve different problems. Vito explained in more detail: If a team has a lot of contact with customers in their daily work and needs to meet some specific needs of customers, then how to handle customer needs better and faster will be the problem that the team needs to focus on; and if the office usually contacts mainly the engineering team, then how to better serve the engineering team will naturally become a priority issue.

Therefore, as a global operation and maintenance team, how to put the needs of various regions together to determine the priority, and as a whole, share a backlog (work list) has become a major challenge for FreeWheel's operation and maintenance work. ***The solution to this problem is the "Global Operation Project Management" process. Simply put, the operation and maintenance team leaders in various regions and the company's IT architects need to communicate regularly about all projects and list priorities to ensure that everyone is consistent.

In terms of collaboration, as the company grows, in order to improve the quality of customer service, FreeWheel's SLA (Service Level Agreement) has become more stringent, the process has become more mature, and temporary needs have become less and less. Vito said that instead of SOP (standard operating procedure) and hardware demand application process, communication and cooperation between teams have become smoother.

2017, Embrace DevOps

As business needs change, FreeWheel has grown from a simple system with only two rack servers (ui↔adserver↔db) to thousands of servers across multiple computer rooms, and covers a complex system architecture with multiple layers such as cache, reporting, forecasting, and nosql. In the past few years, FreeWheel has adopted private cloud solutions, and Vito told reporters that he has recently begun to study the direction of hybrid cloud, using both public and private clouds.

Vito revealed that FreeWheel's next development focus will be on DevOps. He believes that there is a big difference between operations in the United States and China in this regard. In the United States, most operations engineers must have both operations (system + network) capabilities and development capabilities. In China, traditional operations engineers are still more focused on operations. "With the advancement of China's technology industry, the operations field has begun to require operations engineers to have more development thinking in addition to operations thinking."

Another direction of Vito's thinking is how to support faster and faster version iterations. Obviously, this is not just a matter of speed. What is more important is how to maintain the high quality and stability of the production environment system while being fast. This will involve research and investment in both the technology itself and the improvement of the product architecture.

Interview Postscript:

After interviewing FreeWheel, the reporter actually had many feelings beyond professional experience. There are actually many inevitabilities behind the success of this company: rigorous market research, rational technical judgment, accurate market positioning, efficient three-region coordination, and sufficient attention to product application and development... Many of their practices seem to run counter to conventional practices, but when you think about it carefully, they are "reasonable".

As domestic enterprises are going global, the reporter also suggests that other enterprises can refer to FreeWheel's rational thinking and choose the development path that suits them best instead of the most well-known one.

FreeWheel was founded in 2007 and is headquartered in Silicon Valley, USA. It is a wholly foreign-owned company specializing in providing key solutions such as Internet video advertising, monitoring, prediction, and value-added. The founders are Douglas Knopper, Jon Heller, and Diane Yu.

The company has been developing for ten years. Currently, about 80% of the digital video advertising business of traditional TV media and operators in the United States uses FreeWheel's services. Most of the top 10 video websites ranked by ComScore are the company's customers or partners. Starting from 2017, FreeWheel will focus on developing the European market and upgrade its market share based on the approximately 50% market share it has already occupied.

[51CTO original article, please indicate the original author and source as 51CTO.com when reprinting on partner sites]