eliminate_opinion.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extract only hard news\n",
    "\n",
    "For project analyzing the evaluative content of quotes vs. non-quotes.\n",
    "\n",
    "This notebook parses URLs and gets rid of \"first person\", \"analysis\", and \"opinion\". This only works for CBC articles, which have a clear(ish) structure in the URLs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Some notes on the method\n",
    "\n",
    "The rule basically looks for \"first-person\" with either a slash before or a hyphen after in the URL. This catches most cases, where the structure is one of the following:\n",
    "\n",
    "`63ba88f5f642cd45622d61fc,https://www.cbc.ca/news/canada/calgary/dave-cheke-first-person-hostage-1.6686436`\n",
    "\n",
    "`63c3c395f642cd456271cb7f,https://www.cbc.ca/news/canada/first-person-winter-s-darkness-1.6702896`\n",
    "\n",
    "There seem to be some first-person essays that do not have \"first-person\" in the URL. Those have been included, because searching for the string \"first-person\" in the text would exclude other, legitimate articles. The decision is to include the first article below, which is first-person opinion. But the rule allows us to include the second, legitimate news article.\n",
    "\n",
    "Should be out:\n",
    "\n",
    "\n",
    "`63b40d36f642cd4562015142,https://www.cbc.ca/news/canada/manitoba/climate-crisis-should-i-have-kids-michaela-keegan-1.6653649,\"CBC Manitoba's Creator Network asked gen-Zers and millennials to contemplate the choice to have children.\n",
    "[...]\n",
    "Should I Have Kids? is a new series that launches with this first-person essay by Michaela Keegan, a young Winnipegger contemplating her choice to have children or not.`\n",
    "\n",
    "Should be in:\n",
    "\n",
    "`3d58217f642cd4562f6943e,https://www.cbc.ca/news/world/tyre-nichols-memphis-police-beating-video-questions-1.6729731,\n",
    "[...]\n",
    "As Nichols is slumped up against a car, not one of the officers renders aid. The body camera footage shows a first-person view of one of them reaching down and tying his shoe.`\n",
    "\n",
    "For analysis, the string in the URL seems to be either `/analysis-` or `-analysis-`.\n",
    "\n",
    "For opinion, the situation is slightly different. The true opinion pieces seem to be `/opinion`. After analysis of many instances of `-opinion-`, I have concluded that the stories refer to opinion polls or public opinion. They are not opinion pieces. So the procedure only removes `/opinion`. \n",
    "\n",
    "`63c0551df642cd4562586cd4,https://www.cbc.ca/news/world/prince-harry-spare-sales-opinion-poll-1.6711510,`\n",
    "\n",
    "`King Charles and Prince William made their first public appearances on Thursday since the release of Prince Harry's tell-all memoir — which is racking up sales but apparently hurting his once-strong popularity.`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import json\n",
    "from pandas import json_normalize\n",
    "import ast\n",
    "from ast import literal_eval\n",
    "import os\n",
    "import glob"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "os.chdir(r'C:\\Maite\\MOD\\projects\\Monika_Bednarek\\Evaluation_quotes\\Data\\CBC_input')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## One month at a time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv('2023_12_dec_CBC_hardnews.csv', encoding = 'utf8')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extract the right information\n",
    "\n",
    "Filter out the rows that contain:\n",
    "* `/first-person`\n",
    "* `-first-person-`\n",
    "* `/analysis-`\n",
    "* `-analysis-`\n",
    "* `/opinion`\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_no_first = df[~df['url'].str.contains(r'/first-person|-first-person-')]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_no_first"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_no_first_no_analysis = df_no_first[~df_no_first['url'].str.contains(r'/analysis|-analysis-')]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_no_first_no_analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_no_first_no_analysis_no_opinion = df_no_first_no_analysis[~df_no_first_no_analysis['url'].str.contains(r'/opinion')]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_no_first_no_analysis_no_opinion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Remember to change file names here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# CHANGE THE NAME OF THE MONTH EACH TIME!\n",
    "\n",
    "df_no_first_no_analysis_no_opinion.to_csv('2023_12_dec_CBC_news.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Concatenate all csvs - once I have produced all 12 months\n",
    "\n",
    "Once I have all the months, concatenate them and count."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "extension = 'csv'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_filenames = [i for i in glob.glob('*.{}'.format(extension))]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#combine all files in the list\n",
    "\n",
    "CBC_news_all_2023 = pd.concat([pd.read_csv(f) for f in all_filenames])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "CBC_news_all_2023"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export to csv\n",
    "\n",
    "CBC_news_all_2023.to_csv( \"CBC_news_all_2023.csv\", index=False, encoding='utf-8-sig')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}