Does Localhost necessarily mean Localhost?

[[405743]]

We often use the localhost domain name when testing or communicating locally, but does accessing localhost necessarily correspond to our local address?

background

On a sunny afternoon, I suddenly received feedback from an operation and maintenance colleague, saying that one of our service calls suddenly reported an error. The key point is that this service has not been updated for half a year. After asking, I learned that there have been no changes to the infrastructure recently. This is very confusing.

After checking the logs, we found that this service called an unknown IP address. This address can still be pinged, but we clearly configured localhost, so why does this address appear? Shouldn't localhost point to 127.0.0.1? We used dig and nslookup and found that localhost is indeed 127.0.0.1.

We modified the application configuration to make this call directly call 127.0.0.1. It turned out that the service was normal at this time. Then we captured the packet on the machine and found that localhost actually went through the domain name resolution! And the localhost domain name is also registered in our intranet, and the resolved address is the unknown address found at the beginning.

summary

So we subconsciously think that the domain name resolution process should be like this, first look for the /etc/hosts file, localhost is found (the default is 127.0.0.1) and then returned

After investigation, I found that the actual process was as follows: I first did a DNS query, but the DNS was not found, so I checked the /etc/hosts file.

Until one day, we added a localhost domain name resolution to our intranet domain name resolution, and the query was successfully returned directly.

Reproduction

Let's use a simple code to reproduce it first, and simply request localhost.

 package main 
 
 import (
 "fmt"  
 "net/http"  
 ) 
 
 func main() {
 client := &http.Client{}
 _, err := client.Get( "http://localhost:8080" )
 fmt.Println(err)
 }

Then we use the GODEBUG="netdns=go+2" environment variable to execute the program. With this environment variable, when the program is running, it will output whether to execute the DNS query first or query from the /etc/hosts file first.

 GODEBUG= "netdns=go+2" go run main.go
 go package net: GODEBUG setting forcing use of Go's resolver
 go package net: hostLookupOrder(localhost) = files,dns
 Get "http://localhost:8080" : dial tcp [::1]:8080: connect : connection refused

The files and dns shown above mean that the query is first from the /etc/hosts file, and then the dns result is queried. However, the running result of our service at that time was dns and files. Where did this problem occur? It is related to the Go version and the local environment.

We used Docker to simulate the online environment. We also use Docker online.

 FROM golang:1.15 as builder 
 
 WORKDIR /app 
 
 COPY main.go main.go
 COPY run.sh run.sh 
 
 ENV CGO_ENABLED=0
 ENV GOOS=linux 
 
 RUN go build main.go 
 
 FROM alpine:3 
 
 WORKDIR /app 
 
 COPY --from=builder /app /app  
 COPY run.sh run.sh 
 
 RUN chmod +x run.sh 
 
 ENV GODEBUG= "netdns=go+2"  
 ENV CGO_ENABLED=0
 ENV GOOS=linux 
 
 CMD /app/run.sh

The result of running this container is as follows. You can see that it has become dns, files. Why is this the case?

 go package net: built with netgo build tag; using Go's DNS resolver
 go package net: hostLookupOrder(localhost) = dns,files
 Get "http://localhost:8080" : dial tcp 127.0.0.1:8080: connect : connection refused

Troubleshooting

src/net/dnsclient_unix.go

Go defines the following DNS resolution sequences, where files means querying the /etc/hosts file and dns means executing a DNS query.

 // hostLookupOrder specifies the order   of LookupHost lookup strategies.
 // It is basically a simplified representation of nsswitch.conf.
 // "files" means /etc/hosts.
 type hostLookupOrder int   
 
 const (
 // hostLookupCgo means defer to cgo.
 hostLookupCgo hostLookupOrder = iota
 hostLookupFilesDNS // files first  
 hostLookupDNSFiles // dns first  
 hostLookupFiles // only files
 hostLookupDNS // only DNS
 )

You can see in src/net/conf.go

Go will first determine the query order based on some initial conditions, and then look for the hosts configuration item in the /etc/nsswitch.conf file. If it does not exist, it will use some fallback logic. The problem this time is in this fallback logic

 func (c *conf) hostLookupOrder(r *Resolver, hostname string) (ret hostLookupOrder) {
 // ... omitted 
 
 nss := c.nss
 srcs := nss.sources[ "hosts" ]
 // If /etc/nsswitch.conf doesn 't exist or doesn't specify any  
 // sources for   "hosts" , assume Go's DNS will work fine.
 if os.IsNotExist(nss.err) || (nss.err == nil && len(srcs) == 0) {
 if c.goos == "solaris" {
 // illumos defaults to   "nis [NOTFOUND=return] files"  
 return fallbackOrder
 }
 if c.goos == "linux" {
 // glibc says the default   is   "dns [!UNAVAIL=return] files"  
   // https://www.gnu.org/software/libc/manual/html_node/Notes-on-NSS-Configuration-File.html .
 return hostLookupDNSFiles
 }
 return hostLookupFilesDNS
 }
 if nss.err != nil {
 // We failed to parse or   open nsswitch.conf, so
  // conservatively assume we should use cgo if it's
 // available.
 return fallbackOrder
 }
 }

From the above code, we can find that if the current system is Linux and the /etc/nsswitch.conf file does not exist, the order of dns, files will be directly returned. This is based on the implementation of glibc[^2]

This problem is usually not a problem on virtual machines, because most operating systems have this configuration file by default. However, after containerization, we generally like to use a relatively small base image such as alpine linux. The /etc/nsswitch.conf file does not exist in alpine, so there may be problems.

The above logic cannot be reproduced in 1.16 because 1.16 has modified this logic, mainly by deleting the linux judgment branch. If you are interested, you can see this modification record[^3] and this issue[^4]

Summarize

The biggest feeling is that empiricism kills people. Many times, due to our knowledge points, there may be some things that go against our common sense. At this time, we need to make bold assumptions and carefully verify them.

As a fix for this problem, we directly deleted the resolution of localhost. After reviewing the issue, I would like to give you some immature suggestions.

• Don’t register a localhost domain name for your company’s intranet
• The maintenance of the base image is very important. It is recommended that you unify the base image so that you can not only save some disk space, but also make some unified changes. For example, in this case, you can directly add the /etc/nsswitch.conf file to the base image to prevent other businesses from falling into the pit.
• If there is no special version dependency (most applications don't have it), it is recommended to upgrade the Go version to 1.16 to save a lot of trouble.
• DNS resolution does not necessarily query the hosts file first. In addition to this default situation, you can also manually modify the /etc/nsswitch.conf file to adjust the resolution order. If you are interested, you can try it.
• This article also tried to use figma to make a few small animations, which I think are pretty good. I will try it again when I have time to write an article later (Cao Da, please stop being so stubborn, I can’t learn anymore)

References

[^1]: Go 1.14 standard library source code: https://github.com/golang/go/blob/go1.14/src/net/conf.go

[^2]: glibc implementation https://www.gnu.org/software/libc/manual/html_node/Notes-on-NSS-Configuration-File.html

[^3]: Change log: https://github.com/golang/go/commit/c80022204e8fc36ec487888d471de27a5ea47e17#diff-a7c29e18c1a96d08fed3e81f367d079d14c53ea85d739e7460b21fb29a063128

[^4]: https://github.com/golang/go/issues/35305

Original blog: https://lailin.xyz/