How to replace the Query field in the URL?

[[420519]]

When we write a crawler, we may need to generate a new URL based on the current URL in the crawler. For example, the following pseudo code:

 import re
 current_url = 'https://www.kingname.info/archives/page/2/'  
 current_page = re.search( '/(\d+)' , current_url). group (1)
 next_page = int (current_page) + 1
 next_url = re.sub( '\d+' , str(next_page), current_url)
 make_request(next_url)

The running effect is shown in the figure below:

But sometimes, the page turning parameter is not necessarily a number. For example, on some websites, visit a URL: https://xxx.com/articlelist?category=technology&after=asdrtJKSAZFD

When you access this URL, it returns a JSON string, and this JSON contains the following fields:

 ...
 "paging" : {
 "cursors" : {
 "before" : "MTA3NDU0NDExNDEzNTgz" ,
 "after" : "MTE4OTc5MjU0NDQ4NTkwMgZDZD"  
 }, 
         
 }
 ...

This situation is more common in information flow websites. It can only scroll down infinitely to view the next page, and cannot jump directly by page number. The parameter after the next page is returned every time a request is made. When you want to access the next page, use this parameter to replace the parameter after after= in the current URL.

In this way, replacing the parameters in the URL is not a simple task. Because the URL may have 4 situations:

• First page, no after parameter: https://xxx.com/articlelist?category=technology
• The first page has the after parameter name but no value: https://xxx.com/articlelist?category=technology&after=
• On subsequent pages, there is no content after the after parameter value: https://xxx.com/articlelist?category=technology&after=asdrtJKSAZFD
• On the subsequent pages, there is content after the aster parameter value: https://xxx.com/articlelist?category=technology&after=asdrtJKSAZFD&other=abc

You can try to cover these 4 situations and generate the URL of the next page using regular expressions.

In fact, we don't need to use regular expressions. Python's built-in urllib module already provides a solution to this problem. Let's take a look at a piece of code:

 from urllib.parse import urlparse, urlunparse, parse_qs, urlencode 
 
 
 def replace_field(url, name , value):
 parse = urlparse(url)
 query = parse.query
    query_pair = parse_qs(query)
    query_pair[ name ] = value
    new_query = urlencode(query_pair, doseq= True )
    new_parse = parse._replace(query=new_query)
    next_page = urlunparse(new_parse)
 return next_page 
 
 url_list = [
 'https://xxx.com/articlelist?category=technology' ,
 'https://xxx.com/articlelist?category=technology&after=' ,
 'https://xxx.com/articlelist?category=technology&after=asdrtJKSAZFD' ,
 'https://xxx.com/articlelist?category=technology&after=asdrtJKSAZFD&other=abc'  
 ] 
 
 for url in url_list:
    next_page = replace_field(url, 'after' , '0000000' )
 print(next_page)

The running effect is shown in the figure below:

As can be seen from the figure, in these four cases, we can successfully add the parameter after = 0000000 for the next page. There is no need to consider how regular expressions can adapt to all situations.

Among them, urlparse and urlunparse are a pair of opposite functions. The former converts the URL into a ParseResult object, and the latter converts the ParseResult object back to a URL string.

The .query property of the ParseResult object is a string, which is the content after the question mark in the URL, in the following format:

parse_qs and urlencode are also a pair of opposite functions. The former converts the string output by .query into a dictionary, while the latter converts the field into a string in the form of .query:

After using parse_qs to convert the query into a dictionary, you can modify the parameter value and then convert it back again.

Since the .query property of the ParseResult object is a read-only property and cannot be overwritten, we need to call an internal method ._replace to replace the new .query field and generate a new ParseResult object. Finally, convert it back to a URL.

The above is what we introduced today, how to use the functions that come with urllib to replace the fields in the URL.