How does user-mode Tcpdump capture kernel network packets?

[[422515]]

This article is reprinted from the WeChat public account "Kaida Neigong Xiuxian", written by Zhang Yanfei allen. To reprint this article, please contact the WeChat public account "Kaida Neigong Xiuxian".

Hello everyone, I am Fei Ge!

Today let’s talk about tcpdump, which is often used in our work.

In the process of sending and receiving network packets, most of the work is done in kernel mode. So the question is, how does the tcpdump program that we often use in user mode capture kernel mode packets? Some students know that tcpdump is based on libpcap, so what is the working principle of libpcap? If you are asked to write a packet capture program, do you have any ideas?

According to Fei Ge's style, I won't stop until I get to the bottom of the principle. So I conducted an in-depth analysis of the relevant source code. Through this article, you will thoroughly understand the following questions.

How does tcpdump work?

• Can tcpdump capture packets filtered by netfilter?
• How would you start if you were asked to write a packet capture program yourself?
• With the help of these questions, let’s start today’s journey of exploration!

1. Network packet receiving process

In the article Illustrated Linux Network Packet Receiving Process, we introduced in detail how the network packet reaches the user process from the network card. This process can be simply represented by the following diagram.

Find the tcpdump packet capture point

We found the packet capture entry of tcpdump in the code of the network device layer. In the function __netif_receive_skb_core, the protocols on ptype_all will be traversed. Remember that we mentioned above that tcpdump registered a virtual protocol on ptype_all. Now it can be executed. Let's look at the function:

 //file: net/core/dev.c
 static   int __netif_receive_skb_core(struct sk_buff *skb, bool pfmemalloc)
 {
 ......
 //Traverse ptype_all (tcpdump hangs the virtual protocol here)
    list_for_each_entry_rcu(ptype, &ptype_all, list) {
        if (!ptype->dev || ptype->dev == skb->dev) {
 if (pt_prev)
                ret = deliver_skb(skb, pt_prev, orig_dev);
 pt_prev = ptype;
 }
 }
 }

In the above function, ptype_all is traversed and deliver_skb is used to call the callback function in the protocol.

 //file: net/core/dev.c
 static inline int deliver_skb(...)
 {
 return pt_prev->func(skb, skb->dev, pt_prev, orig_dev);
 }

For tcpdump, it will enter packet_rcv (we will talk about why it enters this function later). This function is in the net/packet/af_packet.c file.

 //file: net/packet/af_packet.c
 static   int packet_rcv(struct sk_buff *skb, ...)
 {
 __skb_queue_tail(&sk->sk_receive_queue, skb);
 ......
 }

It can be seen that packet_rcv puts the received skb into the receiving queue of the current packet socket. In this way, the captured packet can be obtained when recvfrom is called later!!

Find the netfilter filter point again

To explain the problem mentioned at the beginning, let's take a closer look at the protocol layer. In ip_rcv, we found a netfilter-related execution logic.

 //file: net/ipv4/ip_input.c
 int ip_rcv(...)
 {
 ......
 return NF_HOOK(NFPROTO_IPV4, NF_INET_PRE_ROUTING, skb, dev, NULL ,
 ip_rcv_finish);
 }

If you use NF_HOOK as a keyword to search, you can find many netfilter filter points, but all of them are located at the IP protocol layer.

In the process of receiving packets, the data packets first pass through the network device layer and then reach the protocol layer.

Then we have the answer to one of the questions in the opening chapter. If we set the netfilter rule, in the process of receiving the packet, tcpdump working at the network device layer will start working first. Before netfilter filters, tcpdump will capture the packet!

Therefore, in the process of receiving packets, netfilter filtering does not affect tcpdump's packet capture!

2. Network packet sending process

Let's look at the network packet sending process. In the article "25 pictures, 10,000 words, disassembling the Linux network packet sending process", we described the network packet sending process in detail. The sending process can be summarized into a simple picture.

Find the netfilter filter point

During the sending process, it is also filtered by various netfilter rules at the IP layer.

 //file: net/ipv4/ip_output.c
 int ip_local_out(struct sk_buff *skb)
 {
 //Execute netfilter filtering
 err = __ip_local_out(skb);
 } 
 
 int __ip_local_out(struct sk_buff *skb)
 {
 ......
 return nf_hook(NFPROTO_IPV4, NF_INET_LOCAL_OUT, skb, NULL ,
         skb_dst(skb)->dev, dst_output);
 }

In this file, you can also see several netfilter filtering logics.

Find the tcpdump packet capture point

When the sending process is completed at the protocol layer and reaches the network device layer, there is also a tcpdump packet capture point.

 //file: net/core/dev.c
 int dev_hard_start_xmit(struct sk_buff *skb, struct net_device *dev,
 struct netdev_queue *txq)
 {
 ...
 if (!list_empty(&ptype_all))
  dev_queue_xmit_nit(skb, dev);
 } 
 
 static void dev_queue_xmit_nit(struct sk_buff *skb, struct net_device *dev)
 {
 list_for_each_entry_rcu(ptype, &ptype_all, list) {
  if ((ptype->dev == dev || !ptype->dev) &&
      (!skb_loop_sk(ptype, skb))) {
 if (pt_prev) {
    deliver_skb(skb2, pt_prev, skb->dev);
 pt_prev = ptype;
 continue ;
 }
 ......
 }
 }
 }

In the above code, we can see that in dev_queue_xmit_nit, the protocols in ptype_all are traversed and deliver_skb is called in sequence. This will execute the virtual protocol that tcpdump is hanging on.

In the process of sending network packets, the process is exactly the opposite of the receiving process, that is, the protocol layer processes it first and the network device layer processes it later.

If netfilter sets filtering rules, the packets will be directly filtered out at the protocol layer, and tcpdump working at the lower network device layer will no longer be able to capture the network packets.

3. TCPDUMP startup

In the previous two sections, we mentioned that the kernel captures packets by traversing ptype_all. Now let's take a look at how user-mode tcpdump mounts the protocol to the internal ptype_all.

We use the strace command to capture the system call of the tcpdump command, and there is a line of socket system call in the display result. The source of Tcpdump's secret is hidden in this line of calling the socket function.

 # strace tcpdump -i eth0
 socket(AF_PACKET, SOCK_RAW, 768)
 ......

The first parameter of the socket system call indicates the address cluster or protocol cluster to which the created socket belongs, and the value starts with AF or PF. In Linux, many protocol clusters are supported, and all definitions can be found in include/linux/socket.h. Here, a packet type socket is created.

Protocol family and address family: Each protocol family has its corresponding address family. For example, the protocol family definition of IPV4 is called PF_INET, and its address family definition is AF_INET. They correspond one to one, and the values ​​are exactly the same, so they are often mixed.

 //file: include/linux/socket.h
 #define AF_UNSPEC 0
 #define AF_UNIX 1 /* Unix domain sockets */
 #define AF_LOCAL 1 /* POSIX name   for AF_UNIX */
 #define AF_INET 2 /* Internet IP Protocol */
 #define AF_INET6 10 /* IP version 6 */
 #define AF_PACKET 17 /* Packet family */
 ......

In addition, the third parameter 768 above represents ETH_P_ALL, socket.htons(ETH_P_ALL) = 768.

Let's take a look at what happens during the creation of this packet type socket and find the socket creation source code.

 //file: net/socket.c
 SYSCALL_DEFINE3(socket, int , family, int , type, int , protocol)
 {
 ......
 retval = sock_create(family, type, protocol, &sock);
 } 
 
 int __sock_create(struct net *net, int family, int type, ...)
 {
 ......
 pf = rcu_dereference(net_families[family]);
 err = pf-> create (net, sock, protocol, kern);
 }

In __sock_create, the specified protocol is obtained from net_families and its create method is called to complete the creation.

net_families is an array. In addition to the commonly used PF_INET (ipv4), it also supports many other protocol families, such as PF_UNIX, PF_INET6 (ipv6), PF_PACKET, etc. Each protocol family can be found in a specific position of the net_families array. In this family type, the member function create points to the corresponding creation function of the protocol family.

According to the above figure, we can see that for packet type socket, pf->create actually calls packet_create function. Let's go into this function to find out, which is the key to understand how tcpdump works!

 //file: packet/af_packet.c
 static   int packet_create(struct net *net, struct socket *sock, int protocol,
 int kern)
 {
 ...
 po = pkt_sk(sk);
 po->prot_hook.func = packet_rcv; 
 
 //Registration hook
 if (proto) {
 po->prot_hook.type = proto;
 register_prot_hook(sk);
 }
 } 
 
 static void register_prot_hook(struct sock *sk)
 {
 struct packet_sock *po = pkt_sk(sk);
 dev_add_pack(&po->prot_hook);
 }

Set the callback function to packet_rcv in packet_create, and then complete the registration through register_prot_hook => dev_add_pack. After registration, a virtual protocol is added to the global protocol ptype_all linked list.

Let's take a look at how dev_add_pack registers the protocol to ptype_all. Looking back at the socket function call we saw at the beginning, the third parameter proto passed in is ETH_P_ALL. So dev_add_pack actually adds the hook function to ptype_all in the end. The code is as follows.

 //file: net/core/dev.c
 void dev_add_pack(struct packet_type *pt)
 {
 struct list_head *head = ptype_head(pt);
 list_add_rcu(&pt->list, head);
 } 
 
 static inline struct list_head *ptype_head(const struct packet_type *pt)
 {
 if (pt->type == htons(ETH_P_ALL))
 return &ptype_all;
 else  
 return &ptype_base[ntohs(pt->type) & PTYPE_HASH_MASK];
 }

We use ETH_P_ALL as an example throughout this article, but sometimes there are other cases. In other cases, the protocol may be registered in ptype_base instead of ptype_all. Similarly, the protocol in ptype_base will also be executed during the sending and receiving process.

Summary: The internal logic of tcpdump when it starts is actually very simple, it just registers a virtual protocol in ptype_all.

IV. Conclusion

Now let’s go back to the issues mentioned at the beginning.

1. How does tcpdump work?

The user-mode tcpdump command uses the socket system call to hook into the ptype_all function used in the kernel source code. Whether in the process of receiving or sending network packets, the protocol in ptype_all will be traversed at the network device layer and the callbacks will be executed. The tcpdump command works based on this underlying principle.

2. Can tcpdump capture packets filtered by netfilter?

Regarding this question, we can look at the receiving and sending processes separately. In the process of receiving network packets, tcpdump can capture the packets that hit the netfilter filtering rules because it is close to the water tower and gets the moon first.

But in the sending process, the opposite is true. The network packet first passes through the protocol layer. If it is filtered out by netfilter at this time, the underlying tcpdump will not see anything before it can see it.

3. How to write a packet capture program yourself

If you want to write a packet capture program similar to tcpdump, you can use packet socket. I wrote a simple demo in C to capture packets and parse the source and destination IP.

Source code address: https://github.com/yanfeizhang/coder-kung-fu/blob/main/tests/network/test04/main.c

Compile it and note that root privileges are required to run it.

 # gcc -o main main.c
 # ./main

The running results are previewed as follows.

Finally, please watch it again and forward it!