6-1. Election Result.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 데이터 획득 준비작업"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "import platform\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "%matplotlib inline \n",
    "\n",
    "path = \"c:/Windows/Fonts/malgun.ttf\"\n",
    "from matplotlib import font_manager, rc\n",
    "if platform.system() == 'Darwin':\n",
    "    rc('font', family='AppleGothic')\n",
    "elif platform.system() == 'Windows':\n",
    "    font_name = font_manager.FontProperties(fname=path).get_name()\n",
    "    rc('font', family=font_name)\n",
    "else:\n",
    "    print('Unknown system... sorry~~~~')    \n",
    "\n",
    "plt.rcParams['axes.unicode_minus'] = False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [],
   "source": [
    "from selenium import webdriver\n",
    "import time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 개정된 변경내용으로 아래와 같이 작성해야 함"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {},
   "outputs": [],
   "source": [
    "driver = webdriver.Chrome('../driver/chromedriver')\n",
    "driver.get(\"http://info.nec.go.kr/main/showDocument.xhtml?electionId=0000000000&topMenuId=VC&secondMenuId=VCCP09\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 대통령선거라는 글자부분을 클릭합니다. 해당 글자에서 앞 장에서 설명한 크롬 개발자 도구로 코드를 확인하면 id가 electionType1으로 나타납니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {},
   "outputs": [],
   "source": [
    "driver.find_element_by_id(\"electionType1\").click() "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 그리고 나타나는 선택항목에서 제19대를 선택하도록 합니다. 역시 크롬 개발자 도구에서 해당 위치의 id는 electionName이라는 것을 찾을 수 있습니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {},
   "outputs": [],
   "source": [
    "driver.find_element_by_id(\"electionName\").send_keys(\"제19대\") "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 그리고 나타나는 선택에서 대통령선거를 선택하도록 합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {},
   "outputs": [],
   "source": [
    "driver.find_element_by_id(\"electionCode\").send_keys(\"대통령선거\") "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 이제 나타나는 시도 항목에서 선택 부분의 XPath를 찾고, 해당 리스트를 가져옵니다"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['서울특별시',\n",
       " '부산광역시',\n",
       " '대구광역시',\n",
       " '인천광역시',\n",
       " '광주광역시',\n",
       " '대전광역시',\n",
       " '울산광역시',\n",
       " '세종특별자치시',\n",
       " '경기도',\n",
       " '강원도',\n",
       " '충청북도',\n",
       " '충청남도',\n",
       " '전라북도',\n",
       " '전라남도',\n",
       " '경상북도',\n",
       " '경상남도',\n",
       " '제주특별자치도']"
      ]
     },
     "execution_count": 126,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sido_list_raw = driver.find_element_by_xpath(\"\"\"//*[@id=\"cityCode\"]\"\"\")\n",
    "sido_list = sido_list_raw.find_elements_by_tag_name(\"option\")\n",
    "sido_names_values = [option.text for option in sido_list]\n",
    "sido_names_values = sido_names_values[2:]\n",
    "sido_names_values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6-2. 19대 대선 개표 결과 데이터 획득하기\n",
    "\n",
    "<img src=\"../img/002.jpg\" width=\"600\" height=\"400\"></img>\n",
    "\n",
    "* 위 그림이 지역을 선택하면 나타나는 화면중 하나인데 득표수에 득표율이 괄호()로 함께 나타납니다.\n",
    "* 이를 제거하고 **(**를 기준으로 왼쪽 숫자만 얻어서, 콤마(,)를 제거하고, float형으로 변경하는 함수를 **get_num**으로 준비해 둡니다"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 152,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "\n",
    "def get_num(temp):\n",
        if type(temp) == int:
            str(temp)
    "    return float(re.split('\\(',temp)[0].replace(',',''))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* 책에서는 특별히 다루고 있지 않지만, 아래 모듈을 import해서 wait.until 함수를 사용할 수 있습니다.\n",
    "\n",
    "<img src=\"../img/003.jpg\" width=\"600\" height=\"200\"></img>\n",
    "\n",
    "* 이 함수는 **검색**버튼이 클릭가능할 때 까지 기다리는 기능을 합니다.\n",
    "* 그리고 **move_sido** 함수는 광역시도 이름을 리스트에 전송하고 검색 버튼을 누르는 역할을 합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 154,
   "metadata": {},
   "outputs": [],
   "source": [
    "from selenium.webdriver.common.by import By\n",
    "from selenium.webdriver.support.ui import WebDriverWait\n",
    "from selenium.webdriver.support import expected_conditions as EC\n",
    "\n",
    "wait = WebDriverWait(driver,10)\n",
    "\n",
    "def move_sido(name):\n",
    "    element = driver.find_element_by_id(\"cityCode\")\n",
    "    element.send_keys(name)\n",
    "    make_xpath = \"\"\"//*[@id=\"searchBtn\"]\"\"\"\n",
    "    wait.until(EC.element_to_be_clickable((By.XPATH,make_xpath)))\n",
    "    driver.find_element_by_xpath(make_xpath).click() "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***append_data** 함수는 빈 내용으로 미리 준비한 DataFrame에 append 명령으로 읽은 데이터를 하나씩 추가하는 기능입니다.\n",
    "\n",
    "<img src=\"../img/002.jpg\" width=\"600\" height=\"400\"></img>\n",
    "\n",
    "* 이 함수가 실행되면 전체 투표수, 문재인후보, 홍준표후보, 안철수후보의 득표수가 추가됩니다"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 155,
   "metadata": {},
   "outputs": [],
   "source": [
    "def append_data(df, sido_name, data):\n",
    "    for each in df[0].values[1:]:\n",
    "        data['광역시도'].append(sido_name)\n",
    "        data['시군'].append(each[0])\n",
    "        data['pop'].append(get_num(each[2]))\n",
    "        data['moon'].append(get_num(each[3]))\n",
    "        data['hong'].append(get_num(each[4]))\n",
    "        data['ahn'].append(get_num(each[5]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "미리 변수를 하나 만들어 둡니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 156,
   "metadata": {},
   "outputs": [],
   "source": [
    "election_result_raw = {'광역시도' : [],\n",
    "                       '시군' : [],\n",
    "                       'pop' : [],\n",
    "                       'moon' : [],\n",
    "                       'hong' : [],    \n",
    "                       'ahn' : [] }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 이후 에러, 이후 검토 못하고 올립니다. 뵙고.. 정리하겠습니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 157,
   "metadata": {},
   "outputs": [
    {
     "ename": "TypeError",
     "evalue": "expected string or bytes-like object",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-157-bba46a152212>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[0;32m     10\u001b[0m     \u001b[0mdf\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mpd\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mread_html\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mstr\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mtable\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     11\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 12\u001b[1;33m     \u001b[0mappend_data\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mdf\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0msido_name\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0melection_result_raw\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;32m<ipython-input-155-2fea3a34f347>\u001b[0m in \u001b[0;36mappend_data\u001b[1;34m(df, sido_name, data)\u001b[0m\n\u001b[0;32m      3\u001b[0m         \u001b[0mdata\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m'광역시도'\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0msido_name\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      4\u001b[0m         \u001b[0mdata\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m'시군'\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0meach\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;36m0\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 5\u001b[1;33m         \u001b[0mdata\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m'pop'\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mget_num\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0meach\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;36m2\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      6\u001b[0m         \u001b[0mdata\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m'moon'\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mget_num\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0meach\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;36m3\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      7\u001b[0m         \u001b[0mdata\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m'hong'\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mget_num\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0meach\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;36m4\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m<ipython-input-152-5e4233a8aff0>\u001b[0m in \u001b[0;36mget_num\u001b[1;34m(temp)\u001b[0m\n\u001b[0;32m      2\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      3\u001b[0m \u001b[1;32mdef\u001b[0m \u001b[0mget_num\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mtemp\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 4\u001b[1;33m     \u001b[1;32mreturn\u001b[0m \u001b[0mfloat\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mre\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msplit\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'\\('\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0mtemp\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;36m0\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mreplace\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m','\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;34m''\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;32mC:\\Anaconda\\lib\\re.py\u001b[0m in \u001b[0;36msplit\u001b[1;34m(pattern, string, maxsplit, flags)\u001b[0m\n\u001b[0;32m    210\u001b[0m     \u001b[1;32mand\u001b[0m \u001b[0mthe\u001b[0m \u001b[0mremainder\u001b[0m \u001b[0mof\u001b[0m \u001b[0mthe\u001b[0m \u001b[0mstring\u001b[0m \u001b[1;32mis\u001b[0m \u001b[0mreturned\u001b[0m \u001b[1;32mas\u001b[0m \u001b[0mthe\u001b[0m \u001b[0mfinal\u001b[0m \u001b[0melement\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    211\u001b[0m     of the list.\"\"\"\n\u001b[1;32m--> 212\u001b[1;33m     \u001b[1;32mreturn\u001b[0m \u001b[0m_compile\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mpattern\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mflags\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msplit\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mstring\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mmaxsplit\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    213\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    214\u001b[0m \u001b[1;32mdef\u001b[0m \u001b[0mfindall\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mpattern\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mstring\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mflags\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;36m0\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mTypeError\u001b[0m: expected string or bytes-like object"
     ]
    }
   ],
   "source": [
    "from bs4 import BeautifulSoup\n",
    "\n",
    "for each_sido in sido_names_values:\n",
    "    move_sido(each_sido)\n",
    "    \n",
    "    html = driver.page_source\n",
    "    soup = BeautifulSoup(html, 'html.parser')\n",
    "    table = soup.find('table')\n",
    "    \n",
    "    df = pd.read_html(str(table))\n",
    "    \n",
    "    append_data(df, each_sido, election_result_raw)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "election_result = pd.DataFrame(election_result_raw, \n",
    "                               columns=['광역시도', '시군', 'pop', 'moon','hong','ahn'])\n",
    "election_result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "election_result.to_csv('../data/05. election_result.csv', encoding='utf-8', sep=',')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "driver.close()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}