urllib是python的一个获取url(Uniform Resource Locators,统一资源定址器)了，我们可以利用它来抓取远程的数据进行保存哦，下面整理了一些关于urllib使用中的一些关于header,代理,超时,认证,异常处理处理方法，下面一起来看看。
python3 抓取网页资源的 N 种方法
1、最简单
1importurllib.request23response = urllib.request.urlopen('http://python.org/')45html = response.read()
2、使用 Request
1importurllib.request23req = urllib.request.Request('http://python.org/')45response =urllib.request.urlopen(req)67the_page = response.read()
3、发送数据
1#! /usr/bin/env python323importurllib.parse45importurllib.request67url ='http://localhost/login.php'89user_agent ='Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)'1011values = {'act':'login','login[email]':'abc@abc.com','login[passWord]':'123456'}1213data =urllib.parse.urlencode(values)1415req =urllib.request.Request(url, data)1617req.add_header('Referer','http://www.python.org/')1819response =urllib.request.urlopen(req)2021the_page =response.read()2223PRint(the_page.decode("utf8"))
4、发送数据和header
1#! /usr/bin/env python323importurllib.parse45importurllib.request67url ='http://localhost/login.php'89user_agent ='Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)'1011values = {'act':'login','login[email]':'abc@abc.com','login[password]':'123456'}1213headers = {'User-Agent': user_agent }1415data =urllib.parse.urlencode(values)1617req =urllib.request.Request(url, data, headers)1819response =urllib.request.urlopen(req)2021the_page =response.read()2223print(the_page.decode("utf8"))
5、http 错误
1#! /usr/bin/env python323importurllib.request45req = urllib.request.Request('http://python.org/')67try:89urllib.request.urlopen(req)1011excepturllib.error.HTTPError as e:1213print(e.code)1415print(e.read().decode("utf8"))
6、异常处理1
1#! /usr/bin/env python323fromurllib.requestimportRequest, urlopen45fromurllib.errorimportURLError, HTTPError67req = Request('http://www.python.org/')89try:1011response =urlopen(req)1213exceptHTTPError as e:1415print('The (www.python.org)server couldn't fulfill the request.')1617print('Error code:', e.code)1819exceptURLError as e:2021print('We failed to reach a server.')2223print('Reason:', e.reason)2425else:2627print("good!")2829print(response.read().decode("utf8"))
7、异常处理2
1#! /usr/bin/env python323fromurllib.requestimportRequest, urlopen45fromurllib.errorimportURLError67req = Request("http://www.python.org/")89try:1011response =urlopen(req)1213exceptURLError as e:1415ifhasattr(e,'reason'):1617print('We failed to reach a server.')1819print('Reason:', e.reason)2021elifhasattr(e,'code'):2223print('The server couldn't fulfill the request.')2425print('Error code:', e.code)2627else:print("good!")2829print(response.read().decode("utf8"))
8、HTTP 认证
1#! /usr/bin/env python323importurllib.request45#create a password manager67password_mgr =urllib.request.HTTPPasswordMgrWithDefaultRealm()89#Add the username and password.1011#If we knew the realm, we could use it instead of None.1213top_level_url ="https://www.python.org/"1415password_mgr.add_password(None, top_level_url,'rekfan','xxxxxx')1617handler =urllib.request.HTTPBasicAuthHandler(password_mgr)1819#create "opener" (OpenerDirector instance)2021opener =urllib.request.build_opener(handler)2223#use the opener to fetch a URL2425a_url ="https://www.python.org/"2627x =opener.open(a_url)2829print(x.read())3031#Install the opener.3233#Now all calls to urllib.request.urlopen use our opener.3435urllib.request.install_opener(opener)3637a = urllib.request.urlopen(a_url).read().decode('utf8')3839print(a)
9、使用代理
1#! /usr/bin/env python323importurllib.request45proxy_support = urllib.request.ProxyHandler({'sock5':'localhost:1080'})67opener =urllib.request.build_opener(proxy_support)89urllib.request.install_opener(opener)1011a = urllib.request.urlopen("http://www.python.org/").read().decode("utf8")1213print(a)
10、超时
1#! /usr/bin/env python323importsocket45importurllib.request67#timeout in seconds89timeout = 21011socket.setdefaulttimeout(timeout)1213#this call to urllib.request.urlopen now uses the default timeout1415#we have set in the socket module1617req = urllib.request.Request('http://www.python.org/')1819a =urllib.request.urlopen(req).read()2021print(a)