Python threading单线程与多线程爬虫实战速度测试对比

Python Threading 多线程爬虫实战-cnblogs.py

import requests

headers = {
    "user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36"
}
urls = [
    f"https://www.cnblogs.com/#p{page}"
    for page in range(1,50+1)
]

def craw(url):
    r = requests.get(url, headers=headers)
    print(url, len(r.text))
    return r.text

Python Threading 多线程爬虫实战 单线程 single_thread.py

import cnblogs
import time

# 单线程
def single_thread():
    print("single_thread begin")

    for url in cnblogs.urls:
        cnblogs.craw(url)

    print("single_thread end")

if __name__ == "__main__":
    start = time.time()
    single_thread()
    end = time.time()

    print("single thread const:", end - start, 'seconds')

Python Threading 多线程爬虫实战-多线程 multi_thread.py

import cnblogs
import threading
import time

# 多线程
def multi_thread():
    print("multi_thread begin")
    threads = []
    # 创建线程
    for url in cnblogs.urls:
        threads.append(
            threading.Thread(target=cnblogs.craw, args=(url,))
        )

    # 启动线程
    for thread in threads:
        thread.start()

    # 等待线程结束
    for thread in threads:
        thread.join()

    print("multi_thread end")


if __name__ == "__main__":
    start = time.time()
    multi_thread()
    end = time.time()

    print("multi thread const:", end - start, 'seconds')

Python Threading 多线程爬虫实战-单线程测试结果

单线程测试用时 single thread const: 11.669063806533813 seconds

(venv) fujie@fujuhaos-MacBook-Pro threading % python single_thread.py 
single_thread begin
https://www.cnblogs.com/#p1 69472
https://www.cnblogs.com/#p2 69472
https://www.cnblogs.com/#p3 69472
https://www.cnblogs.com/#p4 69472
https://www.cnblogs.com/#p5 69472
https://www.cnblogs.com/#p6 69472
https://www.cnblogs.com/#p7 69472
https://www.cnblogs.com/#p8 69472
https://www.cnblogs.com/#p9 69472
https://www.cnblogs.com/#p10 69472
https://www.cnblogs.com/#p11 69472
https://www.cnblogs.com/#p12 69472
https://www.cnblogs.com/#p13 69472
https://www.cnblogs.com/#p14 69472
https://www.cnblogs.com/#p15 69472
https://www.cnblogs.com/#p16 69472
https://www.cnblogs.com/#p17 69472
https://www.cnblogs.com/#p18 69472
https://www.cnblogs.com/#p19 69472
https://www.cnblogs.com/#p20 69472
https://www.cnblogs.com/#p21 69472
https://www.cnblogs.com/#p22 69472
https://www.cnblogs.com/#p23 69472
https://www.cnblogs.com/#p24 69472
https://www.cnblogs.com/#p25 69472
https://www.cnblogs.com/#p26 69472
https://www.cnblogs.com/#p27 69472
https://www.cnblogs.com/#p28 69472
https://www.cnblogs.com/#p29 69472
https://www.cnblogs.com/#p30 69472
https://www.cnblogs.com/#p31 69472
https://www.cnblogs.com/#p32 69472
https://www.cnblogs.com/#p33 69472
https://www.cnblogs.com/#p34 69472
https://www.cnblogs.com/#p35 69472
https://www.cnblogs.com/#p36 69472
https://www.cnblogs.com/#p37 69472
https://www.cnblogs.com/#p38 69472
https://www.cnblogs.com/#p39 69472
https://www.cnblogs.com/#p40 69472
https://www.cnblogs.com/#p41 69472
https://www.cnblogs.com/#p42 69472
https://www.cnblogs.com/#p43 69472
https://www.cnblogs.com/#p44 69472
https://www.cnblogs.com/#p45 69472
https://www.cnblogs.com/#p46 69472
https://www.cnblogs.com/#p47 69472
https://www.cnblogs.com/#p48 69472
https://www.cnblogs.com/#p49 69472
https://www.cnblogs.com/#p50 69472
single_thread end
single thread const: 11.669063806533813 seconds

Python Threading 多线程爬虫实战-多线程测试结果

多线程测试用时： multi thread const: 1.8338000774383545 seconds

(venv) fujie@fujuhaos-MacBook-Pro threading % python multi_thread.py
multi_thread begin
https://www.cnblogs.com/#p9 69472
https://www.cnblogs.com/#p6 69472
https://www.cnblogs.com/#p10 69472
https://www.cnblogs.com/#p3 69472
https://www.cnblogs.com/#p1 69472
https://www.cnblogs.com/#p7 69472
https://www.cnblogs.com/#p15 69472
https://www.cnblogs.com/#p8 69472
https://www.cnblogs.com/#p2 69472
https://www.cnblogs.com/#p18 69472
https://www.cnblogs.com/#p22 69472
https://www.cnblogs.com/#p4 69472
https://www.cnblogs.com/#p11 69472
https://www.cnblogs.com/#p26 69472
https://www.cnblogs.com/#p16 69472
https://www.cnblogs.com/#p24 69472
https://www.cnblogs.com/#p30 69472
https://www.cnblogs.com/#p19 69472
https://www.cnblogs.com/#p17 69472
https://www.cnblogs.com/#p12 69472
https://www.cnblogs.com/#p21 69472
https://www.cnblogs.com/#p20 69472
https://www.cnblogs.com/#p47 69472
https://www.cnblogs.com/#p43 69472
https://www.cnblogs.com/#p37 69472
https://www.cnblogs.com/#p31 69472
https://www.cnblogs.com/#p25 69472
https://www.cnblogs.com/#p14 69472
https://www.cnblogs.com/#p5 69472
https://www.cnblogs.com/#p29 69472
https://www.cnblogs.com/#p45 69472
https://www.cnblogs.com/#p13 69472
https://www.cnblogs.com/#p48 69472
https://www.cnblogs.com/#p39 69472
https://www.cnblogs.com/#p41 69472
https://www.cnblogs.com/#p44 69472
https://www.cnblogs.com/#p49 69472
https://www.cnblogs.com/#p34 69472
https://www.cnblogs.com/#p23 69472
https://www.cnblogs.com/#p40 69472
https://www.cnblogs.com/#p46 69472
https://www.cnblogs.com/#p42 69472
https://www.cnblogs.com/#p32 69472
https://www.cnblogs.com/#p27 69472
https://www.cnblogs.com/#p33 69472
https://www.cnblogs.com/#p28 69472
https://www.cnblogs.com/#p50 69472
https://www.cnblogs.com/#p36 69472
https://www.cnblogs.com/#p35 69472
https://www.cnblogs.com/#p38 69472
multi_thread end
multi thread const: 1.8338000774383545 seconds

以上就是Python单线程与多线程测试对比，使用Python threading多线程效果还是比较明显的！