05_spider/02_网页解析和存储.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "e78c442f",
   "metadata": {},
   "source": [
    "| [05_spider/02_网页解析和存储.ipynb](https://github.com/shibing624/python-tutorial/blob/master/05_spider/02_网页解析和存储.ipynb)  | 网页工具requests、lxml、BeautifulSoup、Selenium  |[Open In Colab](https://colab.research.google.com/github/shibing624/python-tutorial/blob/master/05_spider/02_网页解析和存储.ipynb) |\n",
    "\n",
    "# 网页解析\n",
    "\n",
    "通过上一个章节的讲解，我们已经了解到了开发一个爬虫需要做的工作以及一些常见的问题，下面我们给出一个爬虫开发相关技术的清单以及这些技术涉及到的标准库和第三方库，稍后我们会一一介绍这些内容。\n",
    "\n",
    "1. 下载数据 - **urllib** / **requests** / **aiohttp** / **httpx**。\n",
    "2. 解析数据 - **re** / **lxml** / **beautifulsoup4** / **pyquery**。\n",
    "3. 缓存和持久化 - **mysqlclient** / **sqlalchemy** / **peewee** / **redis** / **pymongo**。\n",
    "4. 生成数字签名 - **hashlib**。\n",
    "5. 序列化和压缩 - **pickle** / **json** / **zlib**。\n",
    "6. 调度器 - **multiprocessing** / **threading** / **concurrent.futures**。\n",
    "\n",
    "## 使用requests获取页面\n",
    "\n",
    "对`requests`库的用法做进一步说明。\n",
    "\n",
    "1. GET请求和POST请求。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "1ec4182a",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "200\n",
      "{'Cache-Control': 'private, no-cache, no-store, proxy-revalidate, no-transform', 'Connection': 'keep-alive', 'Content-Encoding': 'gzip', 'Content-Type': 'text/html', 'Date': 'Mon, 06 Sep 2021 07:45:48 GMT', 'Last-Modified': 'Mon, 23 Jan 2017 13:27:36 GMT', 'Pragma': 'no-cache', 'Server': 'bfe/1.0.8.18', 'Set-Cookie': 'BDORZ=27315; max-age=86400; domain=.baidu.com; path=/', 'Transfer-Encoding': 'chunked'}\n",
      "<RequestsCookieJar[<Cookie BDORZ=27315 for .baidu.com/>]>\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'<!DOCTYPE html>\\r\\n<!--STATUS OK--><html> <head><meta http-equiv=content-type content=text/html;charset=utf-8><meta http-equiv=X-UA-Compatible content=IE=Edge><meta content=always name=referrer><link rel=stylesheet type=text/css href=http://s1.bdstatic.com/r/www/cache/bdorz/baidu.min.css><title>百度一下，你就知道</title></head> <body link=#0000cc> <div id=wrapper> <div id=head> <div class=head_wrapper> <div class=s_form> <div class=s_form_wrapper> <div id=lg> <img hidefocus=true src=//www.baidu.com/img/bd_logo1.png width=270 height=129> </div> <form id=form name=f action=//www.baidu.com/s class=fm> <input type=hidden name=bdorz_come value=1> <input type=hidden name=ie value=utf-8> <input type=hidden name=f value=8> <input type=hidden name=rsv_bp value=1> <input type=hidden name=rsv_idx value=1> <input type=hidden name=tn value=baidu><span class=\"bg s_ipt_wr\"><input id=kw name=wd class=s_ipt value maxlength=255 autocomplete=off autofocus></span><span class=\"bg s_btn_wr\"><input type=submit id=su value=百度一下 class=\"bg s_btn\"></span> </form> </div> </div> <div id=u1> <a href=http://news.baidu.com name=tj_trnews class=mnav>新闻</a> <a href=http://www.hao123.com name=tj_trhao123 class=mnav>hao123</a> <a href=http://map.baidu.com name=tj_trmap class=mnav>地图</a> <a href=http://v.baidu.com name=tj_trvideo class=mnav>视频</a> <a href=http://tieba.baidu.com name=tj_trtieba class=mnav>贴吧</a> <noscript> <a href=http://www.baidu.com/bdorz/login.gif?login&amp;tpl=mn&amp;u=http%3A%2F%2Fwww.baidu.com%2f%3fbdorz_come%3d1 name=tj_login class=lb>登录</a> </noscript> <script>document.write(\\'<a href=\"http://www.baidu.com/bdorz/login.gif?login&tpl=mn&u=\\'+ encodeURIComponent(window.location.href+ (window.location.search === \"\" ? \"?\" : \"&\")+ \"bdorz_come=1\")+ \\'\" name=\"tj_login\" class=\"lb\">登录</a>\\');</script> <a href=//www.baidu.com/more/ name=tj_briicon class=bri style=\"display: block;\">更多产品</a> </div> </div> </div> <div id=ftCon> <div id=ftConw> <p id=lh> <a href=http://home.baidu.com>关于百度</a> <a href=http://ir.baidu.com>About Baidu</a> </p> <p id=cp>&copy;2017&nbsp;Baidu&nbsp;<a href=http://www.baidu.com/duty/>使用百度前必读</a>&nbsp; <a href=http://jianyi.baidu.com/ class=cp-feedback>意见反馈</a>&nbsp;京ICP证030173号&nbsp; <img src=//www.baidu.com/img/gs.gif> </p> </div> </div> </div> </body> </html>\\r\\n'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import requests\n",
    "   \n",
    "resp = requests.get('http://www.baidu.com/index.html')\n",
    "print(resp.status_code)\n",
    "print(resp.headers)\n",
    "print(resp.cookies)\n",
    "resp.content.decode('utf-8')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "5eb730fb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"args\": {}, \n",
      "  \"data\": \"\", \n",
      "  \"files\": {}, \n",
      "  \"form\": {\n",
      "    \"age\": \"40\", \n",
      "    \"name\": \"Hao\"\n",
      "  }, \n",
      "  \"headers\": {\n",
      "    \"Accept\": \"*/*\", \n",
      "    \"Accept-Encoding\": \"gzip, deflate\", \n",
      "    \"Content-Length\": \"15\", \n",
      "    \"Content-Type\": \"application/x-www-form-urlencoded\", \n",
      "    \"Host\": \"httpbin.org\", \n",
      "    \"User-Agent\": \"python-requests/2.25.1\", \n",
      "    \"X-Amzn-Trace-Id\": \"Root=1-6135c72e-53ae8dac1b4603f7047e4623\"\n",
      "  }, \n",
      "  \"json\": null, \n",
      "  \"origin\": \"206.161.232.8\", \n",
      "  \"url\": \"http://httpbin.org/post\"\n",
      "}\n",
      "\n",
      "<class 'dict'>\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'args': {},\n",
       " 'data': '',\n",
       " 'files': {},\n",
       " 'form': {'age': '40', 'name': 'Hao'},\n",
       " 'headers': {'Accept': '*/*',\n",
       "  'Accept-Encoding': 'gzip, deflate',\n",
       "  'Content-Length': '15',\n",
       "  'Content-Type': 'application/x-www-form-urlencoded',\n",
       "  'Host': 'httpbin.org',\n",
       "  'User-Agent': 'python-requests/2.25.1',\n",
       "  'X-Amzn-Trace-Id': 'Root=1-6135c72e-53ae8dac1b4603f7047e4623'},\n",
       " 'json': None,\n",
       " 'origin': '206.161.232.8',\n",
       " 'url': 'http://httpbin.org/post'}"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "resp = requests.post('http://httpbin.org/post', data={'name': 'Hao', 'age': 40})\n",
    "print(resp.text)\n",
    "data = resp.json()\n",
    "print(type(data))\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dda8f328",
   "metadata": {},
   "source": [
    "2. URL参数和请求头。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "2a0f108a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "200\n"
     ]
    }
   ],
   "source": [
    "resp = requests.get(\n",
    "   url='https://movie.douban.com/top250',\n",
    "   headers={\n",
    "       'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) '\n",
    "                     'AppleWebKit/537.36 (KHTML, like Gecko) '\n",
    "                     'Chrome/83.0.4103.97 Safari/537.36',\n",
    "       'Accept': 'text/html,application/xhtml+xml,application/xml;'\n",
    "                 'q=0.9,image/webp,image/apng,*/*;'\n",
    "                 'q=0.8,application/signed-exchange;v=b3;q=0.9',\n",
    "       'Accept-Language': 'zh-CN,zh;q=0.9,en;q=0.8',\n",
    "   }\n",
    ")\n",
    "print(resp.status_code)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "80c7ba40",
   "metadata": {},
   "source": [
    "3. 复杂的POST请求（文件上传）。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "a7792714",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"args\": {}, \n",
      "  \"data\": \"\", \n",
      "  \"files\": {\n",
      "    \"file\": \"123\"\n",
      "  }, \n",
      "  \"form\": {}, \n",
      "  \"headers\": {\n",
      "    \"Accept\": \"*/*\", \n",
      "    \"Accept-Encoding\": \"gzip, deflate\", \n",
      "    \"Content-Length\": \"143\", \n",
      "    \"Content-Type\": \"multipart/form-data; boundary=03d4024124e0c144beb31decbaf31f04\", \n",
      "    \"Host\": \"httpbin.org\", \n",
      "    \"User-Agent\": \"python-requests/2.25.1\", \n",
      "    \"X-Amzn-Trace-Id\": \"Root=1-6135c72e-3097cafa2307cff613eafde7\"\n",
      "  }, \n",
      "  \"json\": null, \n",
      "  \"origin\": \"206.161.232.8\", \n",
      "  \"url\": \"http://httpbin.org/post\"\n",
      "}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "resp = requests.post(\n",
    "url='http://httpbin.org/post',\n",
    "   files={'file': '123'}\n",
    ")\n",
    "print(resp.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d2f2b198",
   "metadata": {},
   "source": [
    "4. 操作Cookie。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "dc5dcc75",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"cookies\": {\n",
      "    \"key1\": \"value1\", \n",
      "    \"key2\": \"value2\"\n",
      "  }\n",
      "}\n",
      "\n",
      "{\n",
      "  \"cookies\": {\n",
      "    \"tasty_cookie\": \"yum\"\n",
      "  }\n",
      "}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "cookies = {'key1': 'value1', 'key2': 'value2'}\n",
    "resp = requests.get('http://httpbin.org/cookies', cookies=cookies)\n",
    "print(resp.text)\n",
    "\n",
    "jar = requests.cookies.RequestsCookieJar()\n",
    "jar.set('tasty_cookie', 'yum', domain='httpbin.org', path='/cookies')\n",
    "jar.set('gross_cookie', 'blech', domain='httpbin.org', path='/elsewhere')\n",
    "resp = requests.get('http://httpbin.org/cookies', cookies=jar)\n",
    "print(resp.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1821b1b",
   "metadata": {},
   "source": [
    "5. 设置代理服务器。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "a2d0bde8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Response [418]>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r = requests.get('https://movie.douban.com')\n",
    "r"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "d72fda9d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Response [418]>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "requests.get('https://movie.douban.com', proxies={\n",
    "   'http': 'http://10.10.1.10:3128',\n",
    "})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe519045",
   "metadata": {},
   "source": [
    "> **说明**：关于`requests`库的相关知识，还是强烈建议大家自行阅读它的[官方文档](https://requests.readthedocs.io/zh_CN/latest/)。\n",
    "   \n",
    "6. 设置请求超时。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "95f63c77",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Response [200]>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "requests.get('https://baidu.com', timeout=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce625d99",
   "metadata": {},
   "source": [
    "## 页面内容解析\n",
    "\n",
    "### 解析方式的比较\n",
    "\n",
    "| 解析方式       | 对应的模块    | 速度   | 使用难度 | 备注                                        |\n",
    "| -------------- | ------------- | ------ | -------- | ------------------------------------------- |\n",
    "| 正则表达式解析 | re            | 快     | 困难     | 常用正则表达式<br/>在线正则表达式测试       |\n",
    "| XPath解析      | lxml          | 快     | 一般     | 需要安装C语言依赖库<br/>唯一支持XML的解析器 |\n",
    "| CSS选择器解析  | bs4 / pyquery | 不确定 | 简单     |                                             |\n",
    "\n",
    "> **说明**：`BeautifulSoup`可选的解析器包括：Python标准库中的`html.parser`、`lxml`的HTML解析器、`lxml`的XML解析器和`html5lib`。\n",
    "\n",
    "\n",
    "#### 正则表达式解析\n",
    "\n",
    "获取豆瓣电影Top250”中的中文电影名称："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "656fbb13",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "肖申克的救赎\n",
      "霸王别姬\n",
      "阿甘正传\n",
      "这个杀手不太冷\n",
      "泰坦尼克号\n",
      "美丽人生\n",
      "千与千寻\n",
      "辛德勒的名单\n",
      "盗梦空间\n",
      "忠犬八公的故事\n",
      "星际穿越\n",
      "楚门的世界\n",
      "海上钢琴师\n",
      "三傻大闹宝莱坞\n",
      "机器人总动员\n",
      "放牛班的春天\n",
      "无间道\n",
      "疯狂动物城\n",
      "大话西游之大圣娶亲\n",
      "熔炉\n",
      "教父\n",
      "当幸福来敲门\n",
      "龙猫\n",
      "怦然心动\n",
      "控方证人\n"
     ]
    }
   ],
   "source": [
    "import random\n",
    "import re\n",
    "import time\n",
    "\n",
    "import requests\n",
    "\n",
    "PATTERN = re.compile(r'<a[^>]*?>\\s*<span class=\"title\">(.*?)</span>')\n",
    "\n",
    "for page in range(1):\n",
    "    resp = requests.get(\n",
    "        url=f'https://movie.douban.com/top250?start={page * 25}',\n",
    "        headers={\n",
    "            'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) '\n",
    "                          'AppleWebKit/537.36 (KHTML, like Gecko) '\n",
    "                          'Chrome/83.0.4103.97 Safari/537.36',\n",
    "            'Accept': 'text/html,application/xhtml+xml,application/xml;'\n",
    "                      'q=0.9,image/webp,image/apng,*/*;'\n",
    "                      'q=0.8,application/signed-exchange;v=b3;q=0.9',\n",
    "            'Accept-Language': 'zh-CN,zh;q=0.9,en;q=0.8',\n",
    "        },\n",
    "    )\n",
    "    items = PATTERN.findall(resp.text)\n",
    "    for item in items:\n",
    "        print(item)\n",
    "    time.sleep(random.randint(1, 5))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a97c7111",
   "metadata": {},
   "source": [
    "#### XPath解析"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "262ac134",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "肖申克的救赎\n",
      "霸王别姬\n",
      "阿甘正传\n",
      "这个杀手不太冷\n",
      "泰坦尼克号\n",
      "美丽人生\n",
      "千与千寻\n",
      "辛德勒的名单\n",
      "盗梦空间\n",
      "忠犬八公的故事\n",
      "星际穿越\n",
      "楚门的世界\n",
      "海上钢琴师\n",
      "三傻大闹宝莱坞\n",
      "机器人总动员\n",
      "放牛班的春天\n",
      "无间道\n",
      "疯狂动物城\n",
      "大话西游之大圣娶亲\n",
      "熔炉\n",
      "教父\n",
      "当幸福来敲门\n",
      "龙猫\n",
      "怦然心动\n",
      "控方证人\n"
     ]
    }
   ],
   "source": [
    "from lxml import etree\n",
    "\n",
    "import requests\n",
    "\n",
    "for page in range(1):\n",
    "    resp = requests.get(\n",
    "        url=f'https://movie.douban.com/top250?start={page * 25}',\n",
    "        headers={\n",
    "            'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) '\n",
    "                          'AppleWebKit/537.36 (KHTML, like Gecko) '\n",
    "                          'Chrome/83.0.4103.97 Safari/537.36',\n",
    "            'Accept': 'text/html,application/xhtml+xml,application/xml;'\n",
    "                      'q=0.9,image/webp,image/apng,*/*;'\n",
    "                      'q=0.8,application/signed-exchange;v=b3;q=0.9',\n",
    "            'Accept-Language': 'zh-CN,zh;q=0.9,en;q=0.8',\n",
    "        }\n",
    "    )\n",
    "    html = etree.HTML(resp.text)\n",
    "    spans = html.xpath('/html/body/div[3]/div[1]/div/div[1]/ol/li/div/div[2]/div[1]/a/span[1]')\n",
    "    for span in spans:\n",
    "        print(span.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "246a6016",
   "metadata": {},
   "source": [
    "## BeautifulSoup的使用\n",
    "\n",
    "BeautifulSoup是一个可以从HTML或XML文件中提取数据的Python库。它能够通过你喜欢的转换器实现惯用的文档导航、查找、修改文档的方式。\n",
    "\n",
    "1. 遍历文档树\n",
    "   - 获取标签\n",
    "   - 获取标签属性\n",
    "   - 获取标签内容\n",
    "   - 获取子（孙）节点\n",
    "   - 获取父节点/祖先节点\n",
    "   - 获取兄弟节点\n",
    "2. 搜索树节点\n",
    "   - find / find_all\n",
    "   - select_one / select\n",
    "\n",
    "> **说明**：更多内容可以参考BeautifulSoup的[官方文档](https://beautifulsoup.readthedocs.io/zh_CN/v4.4.0/)。\n",
    "\n",
    "\n",
    "#### CSS选择器解析\n",
    "下面的例子演示了如何用CSS选择器解析“豆瓣电影Top250”中的中文电影名称。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "b1c29e59",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "肖申克的救赎\n",
      "霸王别姬\n",
      "阿甘正传\n",
      "这个杀手不太冷\n",
      "泰坦尼克号\n",
      "美丽人生\n",
      "千与千寻\n",
      "辛德勒的名单\n",
      "盗梦空间\n",
      "忠犬八公的故事\n",
      "星际穿越\n",
      "楚门的世界\n",
      "海上钢琴师\n",
      "三傻大闹宝莱坞\n",
      "机器人总动员\n",
      "放牛班的春天\n",
      "无间道\n",
      "疯狂动物城\n",
      "大话西游之大圣娶亲\n",
      "熔炉\n",
      "教父\n",
      "当幸福来敲门\n",
      "龙猫\n",
      "怦然心动\n",
      "控方证人\n"
     ]
    }
   ],
   "source": [
    "import random\n",
    "import time\n",
    "\n",
    "import bs4\n",
    "import requests\n",
    "\n",
    "for page in range(1):\n",
    "    resp = requests.get(\n",
    "        url=f'https://movie.douban.com/top250?start={page * 25}',\n",
    "        headers={\n",
    "            'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) '\n",
    "                          'AppleWebKit/537.36 (KHTML, like Gecko) '\n",
    "                          'Chrome/83.0.4103.97 Safari/537.36',\n",
    "            'Accept': 'text/html,application/xhtml+xml,application/xml;'\n",
    "                      'q=0.9,image/webp,image/apng,*/*;'\n",
    "                      'q=0.8,application/signed-exchange;v=b3;q=0.9',\n",
    "            'Accept-Language': 'zh-CN,zh;q=0.9,en;q=0.8',\n",
    "        },\n",
    "    )\n",
    "    soup = bs4.BeautifulSoup(resp.text, 'lxml')\n",
    "    elements = soup.select('.info>div>a')\n",
    "    for element in elements:\n",
    "        span = element.select_one('.title')\n",
    "        print(span.text)\n",
    "    time.sleep(random.random() * 5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f7e6d4b9",
   "metadata": {},
   "source": [
    "例子：获取知乎发现上的问题链接："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "57124b81",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total 19 question pages found.\n",
      "{'https://www.zhihu.com/question/34846729', 'https://www.zhihu.com/question/484149396', 'https://www.zhihu.com/question/484051164', 'https://www.zhihu.com/question/476716498', 'https://www.zhihu.com/question/39266680', 'https://www.zhihu.com/question/483983107', 'https://www.zhihu.com/question/469309297', 'https://www.zhihu.com/question/483642230', 'https://www.zhihu.com/question/483554401', 'https://www.zhihu.com/question/397141335', 'https://www.zhihu.com/question/435831351', 'https://www.zhihu.com/question/25022797', 'https://www.zhihu.com/question/483642352', 'https://www.zhihu.com/question/20616492', 'https://www.zhihu.com/question/35504318', 'https://www.zhihu.com/question/484209444', 'https://www.zhihu.com/question/378758565', 'https://www.zhihu.com/question/461141381', 'https://www.zhihu.com/question/403034095'}\n"
     ]
    }
   ],
   "source": [
    "import re\n",
    "from urllib.parse import urljoin\n",
    "\n",
    "import bs4\n",
    "import requests\n",
    "\n",
    "links_set = set()\n",
    "\n",
    "def main():\n",
    "    headers = {'user-agent': 'Baiduspider'}\n",
    "    base_url = 'https://www.zhihu.com/'\n",
    "    resp = requests.get(urljoin(base_url, 'explore'), headers=headers)\n",
    "    soup = bs4.BeautifulSoup(resp.text, 'lxml')\n",
    "    href_regex = re.compile(r'^/question')\n",
    "    \n",
    "    for a_tag in soup.find_all('a', {'href': href_regex}):\n",
    "        if 'href' in a_tag.attrs:\n",
    "            href = a_tag.attrs['href']\n",
    "            full_url = urljoin(base_url, href)\n",
    "            links_set.add(full_url)\n",
    "    print('Total %d question pages found.' % len(links_set))\n",
    "    print(links_set)\n",
    "\n",
    "\n",
    "main()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "7ea549c1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'https://www.zhihu.com/question/34846729'"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(links_set)[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "f6717ff2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# !curl https://www.zhihu.com/question/484209444 | head"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d9e91ae",
   "metadata": {},
   "source": [
    "## 使用Selenium动态解析内容\n",
    "\n",
    "使用自动化测试工具Selenium，它提供了浏览器自动化的API接口，这样就可以通过操控浏览器来获取动态内容。首先可以使用pip来安装Selenium。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "1425fd8b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple\r\n",
      "Requirement already satisfied: selenium in /Users/xuming/opt/anaconda3/lib/python3.8/site-packages (3.141.0)\r\n",
      "Requirement already satisfied: urllib3 in /Users/xuming/opt/anaconda3/lib/python3.8/site-packages (from selenium) (1.26.4)\r\n"
     ]
    }
   ],
   "source": [
    "! pip install selenium"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "d7198e6f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://img.alicdn.com/tfs/TB166h7bEY1gK0jSZFCXXcwqXXa-292-68.png\n",
      "https://img.alicdn.com/tfs/TB166h7bEY1gK0jSZFCXXcwqXXa-292-68.png\n",
      "https://img.alicdn.com/tfs/TB1MFjSVET1gK0jSZFrXXcNCXXa-58-62.png\n",
      "https://img.alicdn.com/tfs/TB1mdLHVEH1gK0jSZSyXXXtlpXa-58-58.png\n",
      "https://img.alicdn.com/tfs/TB1WHHxVAL0gK0jSZFtXXXQCXXa-58-58.png\n"
     ]
    }
   ],
   "source": [
    "from bs4 import BeautifulSoup\n",
    "from selenium import webdriver\n",
    "from selenium.webdriver.common.keys import Keys\n",
    "\n",
    "\n",
    "def main():\n",
    "    driver = webdriver.Chrome()\n",
    "    driver.get('https://v.taobao.com/v/content/live?catetype=704&from=taonvlang')\n",
    "    soup = BeautifulSoup(driver.page_source, 'lxml')\n",
    "    for img_tag in soup.body.select('img[src]'):\n",
    "        print(img_tag.attrs['src'])\n",
    "\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    main()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0f83499",
   "metadata": {},
   "source": [
    "从 https://sites.google.com/chromium.org/driver/ 下载 chromedriver，放置于 PATH 环境变量中，然后执行即可。更为简单的办法是把chromedriver直接放在虚拟环境中，跟Python解释器位于同一个路径下就可以了。\n",
    "\n",
    "在上面的程序中，我们通过Selenium实现对Chrome浏览器的操控，如果要操控其他的浏览器，可以创对应的浏览器对象，例如Firefox、IE等。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "47a5227f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/Users/xuming/opt/anaconda3/bin:/Users/xuming/opt/anaconda3/condabin:/Users/xuming/opt/anaconda3/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin\r\n"
     ]
    }
   ],
   "source": [
    "! echo $PATH"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b69789c",
   "metadata": {},
   "source": [
    "# 存储\n",
    "\n",
    "我们可以选择存入文本文件，也可以选择存入[MySQL](http://www.mysql.com/)或[MongoDB](https://www.mongodb.org/)数据库等。  \n",
    "\n",
    "## 数据缓存\n",
    "\n",
    "使用Redis缓存网页页面信息，方便下次直接读取缓存的页面数据，不用重新爬取或者下载。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "027ae8bf",
   "metadata": {},
   "source": [
    "本节完。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a3b938e1",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}