An article to help you understand HTTPS

I encountered some problems when working on HTTPS-related requirements some time ago. Today I have time to sort out the relevant knowledge of HTTPS, and I hope it will be helpful to you.

picture

What is HTTPS

HTTPS, or HTTP Security, is built on the SSL/TSL protocol. TSL is an upgraded version of the SSL protocol. TLS 1.0 is usually marked as SSL 3.1, TLS 1.1 is SSL 3.2, and TLS 1.2 is SSL 3.3. They can be understood as the same set of protocols. Its main functions are as follows:

Prevent eavesdropping. All information is transmitted in encrypted form and cannot be eavesdropped by third parties.

Prevent tampering. It has a verification mechanism. Once tampered, both parties will find it immediately.

Prevent impersonation. Equip an identity certificate to prevent identity impersonation. This article will try to explain what HTTPS is from the perspective of an Android user.

TLS Verification Process

TLS Verification Process

The interaction process is as follows:

1. The client sends a request to the server. The request content includes:

1. The supported protocol versions, such as TLS 1.0.
2. A client-generated random number that is later used to generate a "session key".
3. Supported encryption methods, such as RSA public key encryption.
4. Supported compression methods.

2. The server responds to the client with the following:
3. Confirm the version of the encrypted communication protocol used, such as TLS 1.0. If the browser and server support different versions, the server will close the encrypted communication.
4. A server-generated random number that is later used to generate a "session key".
5. Confirm the encryption method used, such as RSA public key encryption.
6. Server certificate.
7. After receiving the server's response, the client first verifies the server's certificate. If the certificate is not issued by a trusted authority, or the domain name in the certificate is inconsistent with the actual domain name, or the certificate has expired, a warning will be displayed to the visitor, allowing him to choose whether to continue communication.
   If there is no problem with the certificate, the client will extract the server's public key from the certificate and then send the following three pieces of information to the server.
8. A random number. The random number is encrypted with the server's public key to prevent eavesdropping.
9. The coding change notification indicates that subsequent information will be sent using the encryption method and key agreed upon by both parties.
10. The client handshake end notification indicates that the client handshake phase has ended. This item is also the hash value of all the content sent previously, which is used for server verification.
11. After receiving the third random number pre-master key from the client, the server calculates and generates the "session key" for this session. Then, it sends the following information to the client.
12. The coding change notification indicates that subsequent information will be sent using the encryption method and key agreed upon by both parties.
13. The server handshake end notification indicates that the server handshake phase has ended. This item is also the hash value of all the content sent previously, which is used for client verification.

As for why three random numbers must be used to generate the "session key", dog250 explained it very well:

"Both the client and the server need random numbers so that the generated key will not be the same every time. Since the certificate in the SSL protocol is static, it is necessary to introduce a random factor to ensure the randomness of the negotiated key.

For the RSA key exchange algorithm, the pre-master-key itself is a random number. Together with the random number in the hello message, the three random numbers are used by a key exporter to eventually derive a symmetric key.

The existence of pre-master is that the SSL protocol does not trust every host to generate completely random numbers. If the random numbers are not random, then the pre-master secret may be guessed, and it is not appropriate to use only the pre-master secret as the key. Therefore, a new random factor must be introduced. Then the key generated by the client, server and pre-master secret together will not be easy to guess. One pseudo-random number may not be random at all, but three pseudo-random numbers are very close to random. For every additional degree of freedom, the randomness increases by more than one. "

HTTPS related terms

HTTPS involves many concepts, such as X509, .pem, .crt, etc. After understanding the interaction process of HTTPS, let's sort out these concepts first.

• X509 - This is a certificate standard that mainly defines what should be included in the certificate. For details, please refer to RFC5280. SSL uses this certificate standard.
• PEM - Privacy Enhanced Mail, open it to see the text format, it starts with "-----BEGIN..." and ends with "-----END...", and the content is BASE64 encoded. View the information of the PEM format certificate:

 openssl x509 -in certificate.pem -text -noout

Apache and *NIX servers prefer this encoding format.

• DER - Distinguished Encoding Rules, when opened, it is in binary format and unreadable. View the information of the DER format certificate:

 openssl x509 -in certificate.der -inform der -text -noout

Java and Windows servers prefer this encoding format.

The above are the certificate standards and two commonly used encoding formats. Now let's take a look at the file extensions.

• CRT - CRT should be the three letters of certificate, which actually means certificate. It is commonly used in *NIX systems. It may be PEM encoded or DER encoded. Most of them should be PEM encoded. I believe you already know how to identify them.
• CER - certificate, or certificate, commonly seen in Windows systems. Similarly, it may be PEM encoding or DER encoding, and most of them should be DER encoding.
• KEY - usually used to store a public key or private key, not an X.509 certificate, the encoding is the same, it may be PEM or DER. How to view KEY:

 openssl rsa -in mykey.key -text -noout

If it is in DER format, it should be like this: openssl rsa -in mykey.key -text -noout -inform der

• CSR - Certificate Signing Request, that is, certificate signing request, this is not a certificate, but an application for a signed certificate from an authoritative certificate authority. Its core content is a public key (of course, it also comes with some other information). When generating this application, a private key will also be generated. The private key should be kept by yourself. Friends who have made iOS APPs should know how to apply for a developer certificate from Apple. How to check: (If it is in DER format, add -inform der as usual, I won’t write it here)

 openssl req -noout -text -in my.csr

• PFX/P12 - predecessor of PKCS#12. For *nix servers, CRT and KEY are usually stored in different files, but IIS on Windows stores them in a PFX file (so this file contains both the certificate and the private key). Is this unsafe? Probably not. PFX usually has an "extraction password". If you want to read the contents out, it will ask you to provide the extraction password. PFX uses DER encoding. How to convert PFX to PEM encoding?

 openssl pkcs12 -in for-iis.pfx -out for-iis.pem -nodes

At this time, you will be prompted to enter the extraction code. for-iis.pem is a readable text. The command to generate pfx is similar to this: openssl pkcs12 -export -in certificate.crt -inkey privateKey.key -out certificate.pfx -certfile CACert.crt

CACert.crt is the root certificate of the CA (certificate authority), and if available, it is also brought in through the -certfile parameter. So it seems that PFX is actually a certificate keystore.

• JKS - Java Key Storage, which is a Java patent and has little to do with OpenSSL. Using a Java tool called "keytool", you can convert PFX to JKS. Of course, keytool can also generate JKS directly, but I won't go into details here.
• BKS - from BouncyCastleProvider, it also uses TripleDES to protect the key in the keystore, which can prevent the certificate library from being accidentally modified (changing a bit in the keyentry of the Keystore will cause an error), BKS can interoperate with JKS. The following command can convert crt to bks, that is, bks is actually the certificate file mentioned above, and so is jks.

 keytool -importcert -trustcacerts -keystore e:\key.bks -file e:\server.crt -storetype BKS -provider org.bouncycastle.jce.provider.BouncyCastleProvider

bks generation reference: Generation of bks format certificates used in SSL communication

HTTPS on Java

Using HTTPS on Java is very simple. A typical HTTPS method is as follows:

 URL url = new URL("https://google.com"); HttpsURLConnection urlConnection = url.openConnection(); InputStream in = urlConnection.getInputStream();

The default SSLSocketFactory is used at this time, which is consistent with the SSLContext used in the next code segment

 private synchronized SSLSocketFactory getDefaultSSLSocketFactory() { try { SSLContext sslContext = SSLContext.getInstance("TLS"); sslContext.init(null, null, null); return defaultSslSocketFactory = sslContext.getSocketFactory(); } catch (GeneralSecurityException e) { throw new AssertionError(); // The system has no TLS. Just give up. } }

The most important method in this code is sslContext.init(null, null, null);, which has three parameters:

• KeyManager[], its own certificate, is used to verify whether the server is trustworthy. If it is a one-way verification, pass empty, indicating that the server does not need to be verified.
• TrustManager[] verifies whether the server certificate is credible. If it is empty, Android's own certificate library is used. If the server certificate is in the default library, the verification passes.

Sometimes the CA is not trustworthy. In order to obtain higher security, we need to specify the anchor point of the newcomer ourselves. The following code can be used:

 // 取到证书的输入流InputStream is = new FileInputStream("anchor.crt"); CertificateFactory cf = CertificateFactory.getInstance("X.509"); Certificate ca = cf.generateCertificate(is); // 创建Keystore 包含我们的证书String keyStoreType = KeyStore.getDefaultType(); KeyStore keyStore = KeyStore.getInstance(keyStoreType); keyStore.load(null); keyStore.setCertificateEntry("anchor", ca); // 创建一个TrustManager 仅把Keystore 中的证书作为信任的锚点String algorithm = TrustManagerFactory.getDefaultAlgorithm(); TrustManagerFactory trustManagerFactory = TrustManagerFactory.getInstance(algorithm); trustManagerFactory.init(keyStore); TrustManager[] trustManagers = trustManagerFactory.getTrustManagers(); // 用TrustManager 初始化一个SSLContext SSLContext sslContext = SSLContext.getInstance("TLS"); sslContext.init(null, trustManagers, null); return sslContext.getSocketFactory();

In this case, only anchor.crt and the certificate it issues will be trusted.

• SecureRandom, which is the strategy for generating random numbers, is usually passed directly into new SecureRandom() as a parameter.

Note that one-way authentication is used here, that is, only the client needs to verify the server's certificate, and the client caches the certificate locally, so the three parameters of the sslContext.init() method here are empty.

When the default SSLSocketFactory verifies the server's certificate (that is, when TrustManager[] is passed empty), it trusts more than 100 root certificates built into the device.

Since there is one-way authentication, there is also two-way authentication, that is, not only the client needs to verify the server, but the server also needs to verify the client. In this case, the client needs to send its own certificate to the server for verification. In this case, you only need to pass the certificate's KeyManager as the first parameter to SSLContext in init().