HTTPS protocols: TLS, SSL, SNI, ALPN, NPN

HTTPS is now widely used. While it brings security, it also introduces more complex concepts to the Web, including a series of network protocols that have never been seen before. Now Harttle starts from the principle of HTTPS and tries to interpret these protocols involved in HTTPS in the most popular way.

HTTPS Overview

HTTPS is HTTP built on secure communication, using transport layer encryption (TLS or SSL). Its purpose is to protect user privacy (such as preventing HTTP content from being intercepted by network nodes) and data integrity (such as operators inserting advertisements), that is, end-to-end encryption to prevent man-in-the-middle attacks.

TLS/SSL is a protocol above the transport layer and below the application layer, so the content of the HTTP protocol is not affected. These encryptions use asymmetric encryption algorithms, so an official is needed to publish the public key, which is the key infrastructure (CA). Therefore, each browser will have some CA root certificates built in, and these CAs can further authorize other domain names, so that your browser can authenticate the domain name you are visiting.

If you want your own service to support HTTPS, you can register your own domain name with a CA. There are some free CAs such as GoDaddy, Let's Encrypt, CloudFlare, etc.

HTTPS interaction example

The following Wireshark log records a GET request sent to https://github.com/harttle, showing the interaction process of several major protocols:

• TCP. The first three lines complete a pair of SYN/ACK (commonly known as the three-way handshake), and the TCP connection has been successfully established.
• TLS. Lines 4-5 begin the TLS handshake, establishing this encryption layer.
• TLS has many extended protocols such as SNI, NPN, ALPN, etc. (see below), which occur in the ClientHello and ServerHello stages of TLS.

TLS/SSL

TLS is formerly known as SSL, which runs on top of TCP in the TCP/IP protocol stack and is used to exchange keys and form an encryption layer (Record Layer). TLS is the core protocol of HTTPS, and the main difference between HTTPS interaction and HTTP interaction lies in this layer:

Before starting to transmit ciphertext, it is necessary to exchange keys, verify the server certificate and other preparations. Therefore, TLS also has a handshake phase. The main steps are: the client sends ClientHello, the server sends ServerHello, the server continues to send Certificate, and then send KeyExchange messages to each other, and finally send ChangeCipherSpec to notify the other party that the subsequent transmission is ciphertext. For specific interactions and protocol fields, please refer to RFC 5246 (TLSv1.2) and RFC 6176 (TLSv2.0).

TLS is a widely used encryption protocol in the TCP/IP protocol stack. RFC 4366 - TLS Extensions is defined to support protocol extensions of common mechanisms. TLS has been adopted by mail services, web services, FTP, etc. Here is a list of extended protocols.

This article focuses on extensions related to Web services (HTTPS), such as SNI, NPN, and ALPN. These protocols add new features to TLS by extending the TLS ClientHello/ServerHello messages. To this end, let's first look at the structure of the ClientHello message (ServerHello is similar):

 struct {
    ProtocolVersion client_version;
 Random random;
 SessionID session_id;
    CipherSuite cipher_suites<2..2^16-2>;
    CompressionMethod compression_methods<1..2^8-1>;
 select (extensions_present) {
 case   false :
 struct {};
 case   true :
            Extension extensions<0..2^16-1>;
 };
 } ClientHello;

Note the last field, there can be up to 65536 extensions, where an extension is defined as a two-byte ExtensionType and the corresponding opaque data. The following SNI, NPN, and ALPN are all one of them.

SNI

SNI (Server Name Indication) specifies the host name to connect to during the TLS handshake. The SNI protocol is designed to support multiple domain names for the same IP (and port).

Because during the TLS handshake, the server needs to send a certificate to the client, and for this it needs to know the domain name requested by the client (because the certificates for different domain names may be different). At this time, some students will ask, isn’t the host name to be connected the same as the Host when initiating HTTP? This is a misunderstanding of the HTTPS mechanism. When the TLS Handshake occurs, the HTTP interaction has not yet begun, so naturally the HTTP header has not yet reached the server. The SNI protocol is defined in RFC 6066:

 struct {
 NameType name_type;
 select (name_type) {
 case host_name: HostName;
 } name ;
 } ServerName; 
 
 enum {
 host_name(0), (255)
 } NameType; 
 
 opaque HostName<1..2^16-1>;
 struct {
    ServerName server_name_list<1..2^16-1>
 } ServerNameList;

Let's look at the example at the beginning of this article, the SNI Extension field in the ClientHello sent to github.com in line 4:

 Extension Header || Extension Payload (SNI)
 ------------------------------------------------------------------------------------------------------------------  
 ExtensionType Length || PayloadLength Type ServerLength ServerName
 ------------------------------------------------------------------------------------------------------------------  
 00 00 00 0f 00 0d 00 00 0a 67 69 74 68 75 62 2e 63 6f 6d
 sni(0) 15 || 13 host_name 10 github.com

ALPN/NPN

ALPN (Application-Layer Protocol Negotiation) is also an extension of the TLS layer, used to negotiate the protocol used by the application layer. Its predecessor was NPN, which was originally used to support the Google SPDY protocol (now standardized as HTTP/2). Problems with TLS client and server versions have caused a lot of pain in the switching process from SPDY->HTTP/2 and NPN->ALPN:

• The day Google Chrome disables HTTP/2
• Let’s start with enabling HTTP/2 and making the website inaccessible

Therefore, it has become a consensus of today's Web platform to promote Web infrastructure in a standard-first way. We are not talking about those (quasi) browser manufacturers who are still engaged in workshop-style production. Any implementation that blocks the development of the Web platform (even standards, such as XHTML, OSI...) will be eliminated sooner or later.

Back to the topic, ALPN is defined in RFC 7301 - Transport Layer Security (TLS) Application-Layer Protocol Negotiation Extension.

 enum {
    application_layer_protocol_negotiation(16), (65535)
 } ExtensionType; 
 
 opaque ProtocolName<1..2^8-1>; 
 
 struct {
    ProtocolName protocol_name_list<2..2^16-1>
 } ProtocolNameList;

Let's look at the example at the beginning of this article, the ALPN Extension field in the ClientHello sent to github.com in line 4:

 Extension Header || Extention Payload (ALPN)
 ------------------------------------------------------------------------------------------------------------------  
 ExtensionType Length || PayloadLength StringLength Protocol StringLength Protocol
 ------------------------------------------------------------------------------------------------------------------  
 00 10 00 0e 00 0c 02 68 32 08 68 74 74 70 2f 31 2e 31
 alpn(16) 14 || 12 2 h2 8 http/1.1

The message body of Extension contains multiple strings (protocol_name_list), which represent all application layer protocols supported by the client. In the above example, there are two protocols: h2 and http/1.1. Clients that support SPDY will have an additional spdy/2. You need to select one of them in the ServerHello given by the server. In the example in this article, the ALPN field of ServerHello is:

 00 10 00 0b 00 09 08 68 74 74 70 2f 31 2e 31
                     h t t p / 1 . 1

In this way, the Server and Client have reached a consensus using the ALPN protocol and will communicate using the HTTP/1.1 protocol after the handshake is completed.

References and Acknowledgements

Starting from TLS, the key layer of HTTPS, a typical HTTPS interaction process is introduced. Combined with the byte sequence given by the packet capture, the principles and main contents of TLS, SNI, ALPN and other protocols are introduced in turn.