notebooks/data_extraction/process_blog_data.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Processing the blog dataset - EDA\n",
    "\n",
    "https://nbviewer.org/github/Data-Science-for-Linguists-2023/Text-Based-Age-Recognition/blob/main/notebooks/data_extraction/process_blog_data.ipynb "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "check the directories to see where the data is and where to put the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LICENSE                 \u001b[34mdata_samples\u001b[m\u001b[m            project_plan.md\n",
      "README.md               progress_report.md\n",
      "\u001b[34mdata\u001b[m\u001b[m                    progress_report_1.ipynb\n"
     ]
    }
   ],
   "source": [
    "!ls"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "import the main blog dataset into a dataframe using pandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>gender</th>\n",
       "      <th>age</th>\n",
       "      <th>topic</th>\n",
       "      <th>sign</th>\n",
       "      <th>date</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2059027</td>\n",
       "      <td>male</td>\n",
       "      <td>15</td>\n",
       "      <td>Student</td>\n",
       "      <td>Leo</td>\n",
       "      <td>14,May,2004</td>\n",
       "      <td>Info has been found (+/- 100 pages,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2059027</td>\n",
       "      <td>male</td>\n",
       "      <td>15</td>\n",
       "      <td>Student</td>\n",
       "      <td>Leo</td>\n",
       "      <td>13,May,2004</td>\n",
       "      <td>These are the team members:   Drewe...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2059027</td>\n",
       "      <td>male</td>\n",
       "      <td>15</td>\n",
       "      <td>Student</td>\n",
       "      <td>Leo</td>\n",
       "      <td>12,May,2004</td>\n",
       "      <td>In het kader van kernfusie op aarde...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2059027</td>\n",
       "      <td>male</td>\n",
       "      <td>15</td>\n",
       "      <td>Student</td>\n",
       "      <td>Leo</td>\n",
       "      <td>12,May,2004</td>\n",
       "      <td>testing!!!  testing!!!</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>3581210</td>\n",
       "      <td>male</td>\n",
       "      <td>33</td>\n",
       "      <td>InvestmentBanking</td>\n",
       "      <td>Aquarius</td>\n",
       "      <td>11,June,2004</td>\n",
       "      <td>Thanks to Yahoo!'s Toolbar I can ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        id gender  age              topic      sign          date  \\\n",
       "0  2059027   male   15            Student       Leo   14,May,2004   \n",
       "1  2059027   male   15            Student       Leo   13,May,2004   \n",
       "2  2059027   male   15            Student       Leo   12,May,2004   \n",
       "3  2059027   male   15            Student       Leo   12,May,2004   \n",
       "4  3581210   male   33  InvestmentBanking  Aquarius  11,June,2004   \n",
       "\n",
       "                                                text  \n",
       "0             Info has been found (+/- 100 pages,...  \n",
       "1             These are the team members:   Drewe...  \n",
       "2             In het kader van kernfusie op aarde...  \n",
       "3                   testing!!!  testing!!!            \n",
       "4               Thanks to Yahoo!'s Toolbar I can ...  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "df = pd.read_csv('data/blogtext 2.csv')\n",
    "df.head()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Examine the data rows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "id                                                  2059027\n",
       "gender                                                 male\n",
       "age                                                      15\n",
       "topic                                               Student\n",
       "sign                                                    Leo\n",
       "date                                            14,May,2004\n",
       "text                 Info has been found (+/- 100 pages,...\n",
       "Name: 0, dtype: object"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.iloc[0]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now look at the valuable text column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'           Info has been found (+/- 100 pages, and 4.5 MB of .pdf files) Now i have to wait untill our team leader has processed it and learns html.         '"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['text'][0]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "More text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"                 i can't wait for things to get moving, i want out of here. im lookin forward to uni although it means another 3 years in england. it should be so much fun from my experiences of it so far and my brother's experiences. after uni though it will be another year i guess before i can move, as i'll need to work for 'bout that long to get some money. will be quite lonely i imagine as every1 will have moved on by then. i wana work in a pub, duno why, not any pub, not a big pub, i'd love to work the fox n hound by jill's - it's small n friendly, i think thats what attracks me to bein a barmaid - every1 thinks its somit to look down on but what's so bad bein surrounded by people in a friendly (if smoky :s) atmosphere? mayb im naive... also hav a job lined up with jill i hope :) i love the sound of what she plans to do and she says she'll employ me :) ur pretty much self-employed to be honest so it should happen n im into herbal homeopathic stuff so sounds good for me. after that id love to work in italy or canada. i cant speak italian so that leaves canada lol. although i don't have enough points for a visa :( i hav a friend who's gona help me as he's canadian (thats gota help!) and if i end up workin for a whole year before i go then that shld give me more points, plus im gona try learn french alongside my psychology degree, i wana speak french newas but now i hav a goal for it and will give me more points the better at it i am. the biggest problem is probably death - i.e. getting in a plane... ive been thinkin bout takin a flight course to help me feel more prepared for flying, but i don't see how im gona erase the knowledge that it could crash - and if it does it's the worst possible thing to be in that crashes. i dont wana die. if i do, i dont wana die like that. but then more than not wanting to die, i wana live, i know wot i mean... which brings me onto the fact that i realy should start jogging again. i wana b able to do the 10mile fun run next year to raise money for something that keeps catching my eye, i seem to read about it in magazines n papers all the time, n things through the door, it breaks my heart, if there's nethin i can do i wana do it. not because im such a great person just because ive seen the effects.... newas...will b a new years resolution lol im pathetic camp america should be cool too, wish it was this summer though as im gona b bored outa my brain n probs comit suicide for somit fun to do...talkin of which my mum thinks im on drugs, seriously do i come across as that abnormal n screwed? i guess we just have different views on what's normal, what emotions n behaviour are acceptable, whats good for us etc. o well arrivedechi x      \""
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['text'][100]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "sample out 5 data points, let's see how they turn out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>gender</th>\n",
       "      <th>age</th>\n",
       "      <th>topic</th>\n",
       "      <th>sign</th>\n",
       "      <th>date</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>197911</th>\n",
       "      <td>1472995</td>\n",
       "      <td>female</td>\n",
       "      <td>27</td>\n",
       "      <td>indUnk</td>\n",
       "      <td>Aries</td>\n",
       "      <td>19,June,2003</td>\n",
       "      <td>Cara- the pics on his website rem...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>488021</th>\n",
       "      <td>1334247</td>\n",
       "      <td>female</td>\n",
       "      <td>17</td>\n",
       "      <td>Student</td>\n",
       "      <td>Libra</td>\n",
       "      <td>31,March,2004</td>\n",
       "      <td>OK!!! Any boys in high school who are g...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>552483</th>\n",
       "      <td>3056329</td>\n",
       "      <td>male</td>\n",
       "      <td>17</td>\n",
       "      <td>Government</td>\n",
       "      <td>Libra</td>\n",
       "      <td>17,May,2004</td>\n",
       "      <td>Hello out there~ I was just wonde...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>491016</th>\n",
       "      <td>698952</td>\n",
       "      <td>female</td>\n",
       "      <td>34</td>\n",
       "      <td>indUnk</td>\n",
       "      <td>Sagittarius</td>\n",
       "      <td>02,August,2004</td>\n",
       "      <td>What I thought I  Would Never S...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>507087</th>\n",
       "      <td>2999514</td>\n",
       "      <td>male</td>\n",
       "      <td>27</td>\n",
       "      <td>Engineering</td>\n",
       "      <td>Pisces</td>\n",
       "      <td>05,June,2004</td>\n",
       "      <td>I'm about to 'port over all my old work...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             id  gender  age        topic         sign            date  \\\n",
       "197911  1472995  female   27       indUnk        Aries    19,June,2003   \n",
       "488021  1334247  female   17      Student        Libra   31,March,2004   \n",
       "552483  3056329    male   17   Government        Libra     17,May,2004   \n",
       "491016   698952  female   34       indUnk  Sagittarius  02,August,2004   \n",
       "507087  2999514    male   27  Engineering       Pisces    05,June,2004   \n",
       "\n",
       "                                                     text  \n",
       "197911               Cara- the pics on his website rem...  \n",
       "488021         OK!!! Any boys in high school who are g...  \n",
       "552483               Hello out there~ I was just wonde...  \n",
       "491016                 What I thought I  Would Never S...  \n",
       "507087         I'm about to 'port over all my old work...  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_sample = df.sample(5)\n",
    "df_sample"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks like we have 2 teen posts, 2 posts from twenties and 1 from thirties"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We store the samples (CSV format in the sample folder)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_sample.to_csv('data_samples/csv_sample.csv')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Examine texts one by one and add it into sample folders for alternative data viewing. Eg. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"               What I thought I  Would Never See    Didn't think she'd come back. Interesting that she has. Could she just not stay away?   I fade more and more from IRC. Miss it, but far less than I ever thought I would. Why play? Why be around? There is nobody but a handful that even notice or want to do more than sit and stare at a screen. I'm restless. I want ......something.         \""
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(df_sample['text'])[3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.9.6 (default, Oct 18 2022, 12:41:40) \n[Clang 14.0.0 (clang-1400.0.29.202)]"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}