tutorials/PELIC_concordancing_tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# PELIC concordancing tutorial<a name=\"top\"></a>\n",
    "\n",
    "This notebook provides a short example of the type of linguistic investigation that can be carried out with the data in PELIC. The focus of the investigation is a set of verbs which are important indicators of syntactic complexity (described in more detail in the [`Background`](#Background) section of this notebook). The subsequent tutorial has two aims:\n",
    "1. to present a straightforward and replicable way of accessing and processing the corpus data necessary to answer genuine research questions, using tools from the [`Pitt ELI Toolkit (pelitk)`](https://github.com/ELI-Data-Mining-Group/pelitk)\n",
    "2. to demonstrate how to build a concordance list and dataframe using the PELIC data\n",
    "\n",
    "\n",
    "#### Sections of the notebook\n",
    "- [Background](#Background)\n",
    "- [Initial setup](#Initial-setup)\n",
    "- [Building a concordance list](#Building-a-concordance-list)\n",
    "- [Summary](#Summary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Background\n",
    "This tutorial is based on the work by Dr. Alan Juffs and Dr. Na-Rae Han (2019) which investigates the development of syntactic complexity in learners' writing by focusing on a set of key words. In this tutorial we analyize a selection of these key words, nine verbs which we would expect to be followed by either a noun phrase (NP) or a complement clause (CP) in varying degrees: \n",
    "\n",
    "_consider, suggest, explain, realize, admit, deny, conclude, recommend, suppose_  \n",
    "\n",
    "For example, with _suppose_ we would expect a CP but not an NP, i.e.:\n",
    "1. _Sam supposed the answer was correct. (CP)_ √ \n",
    "2. _Sam supposed the answer (NP)_. X  \n",
    "\n",
    "In contrast, with _conclude_ both options are acceptable: \n",
    "1. _Andrea concluded her speech with a joke. (NP)_ √ \n",
    "2. _Andrea concluded that he was telling the truth. (CP)_ √\n",
    "\n",
    "However, considering that the use of a CP necessitates greater syntactic complexity, we hypothesize that learners will underuse the CP constructions with these verbs compared to expert speakers, especially at lower levels of proficiency. Therefore, by analyzing occurences of these verbs and their syntactic patterns, we can answer the following research questions:\n",
    "\n",
    "1. With verbs that allow for both CP and NP constructions, do learners show a preference for NP constructions compared to expert speakers?\n",
    "2. To what extent do factors such as first language and verb frequency affect learners' choice of constructions with these verbs?\n",
    "\n",
    "For a more detailed discussion of this work, please see the [slides](https://github.com/ELI-Data-Mining-Group/Pitt-ELI-Corpus/blob/master/AAAL-2019-FREQ-Mar-12.pdf) or the [abstract](https://aaal.confex.com/aaal/2019/meetingapp.cgi/Session/1553) from the conference where this work was presented."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initial setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pretty printing has been turned OFF\n"
     ]
    }
   ],
   "source": [
    "# Import necessary modules\n",
    "\n",
    "from pelitk import conc\n",
    "import pandas as pd\n",
    "import pprint\n",
    "import pickle as pkl\n",
    "import operator\n",
    "import csv\n",
    "from more_itertools import unique_everseen\n",
    "from ast import literal_eval\n",
    "\n",
    "\n",
    "# Set preferred notebook format\n",
    "\n",
    "%pprint # turn off pretty printing\n",
    "from IPython.core.interactiveshell import InteractiveShell\n",
    "InteractiveShell.ast_node_interactivity = \"all\" # Show all output\n",
    "pd.set_option('display.max_columns', 999) # Allow viewing of all columns in dataframe\n",
    "pd.options.mode.chained_assignment = None # Turn off SettingWithCopy warning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 46230 entries, 1 to 48420\n",
      "Data columns (total 13 columns):\n",
      "anon_id        46230 non-null object\n",
      "L1             46230 non-null object\n",
      "gender         46230 non-null object\n",
      "level_id       46230 non-null object\n",
      "class_id       46230 non-null object\n",
      "question_id    46230 non-null object\n",
      "version        46230 non-null object\n",
      "text_len       46230 non-null int64\n",
      "text           46230 non-null object\n",
      "tokens         46230 non-null object\n",
      "tok_POS        46230 non-null object\n",
      "lemmas         46230 non-null object\n",
      "lemma_POS      46230 non-null object\n",
      "dtypes: int64(1), object(12)\n",
      "memory usage: 4.9+ MB\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>anon_id</th>\n",
       "      <th>L1</th>\n",
       "      <th>gender</th>\n",
       "      <th>level_id</th>\n",
       "      <th>class_id</th>\n",
       "      <th>question_id</th>\n",
       "      <th>version</th>\n",
       "      <th>text_len</th>\n",
       "      <th>text</th>\n",
       "      <th>tokens</th>\n",
       "      <th>tok_POS</th>\n",
       "      <th>lemmas</th>\n",
       "      <th>lemma_POS</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>answer_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>eq0</td>\n",
       "      <td>Arabic</td>\n",
       "      <td>Male</td>\n",
       "      <td>4</td>\n",
       "      <td>g</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>177</td>\n",
       "      <td>I met my friend Nife while I was studying in a...</td>\n",
       "      <td>[I, met, my, friend, Nife, while, I, was, stud...</td>\n",
       "      <td>[(I, PRP), (met, VBD), (my, PRP$), (friend, NN...</td>\n",
       "      <td>[i, meet, my, friend, nife, while, i, be, stud...</td>\n",
       "      <td>[(i, PRP), (meet, VBD), (my, PRP$), (friend, N...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>am8</td>\n",
       "      <td>Thai</td>\n",
       "      <td>Female</td>\n",
       "      <td>4</td>\n",
       "      <td>g</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>137</td>\n",
       "      <td>Ten years ago, I met a women on the train betw...</td>\n",
       "      <td>[Ten, years, ago, ,, I, met, a, women, on, the...</td>\n",
       "      <td>[(Ten, CD), (years, NNS), (ago, RB), (,, ,), (...</td>\n",
       "      <td>[ten, year, ago, ,, i, meet, a, woman, on, the...</td>\n",
       "      <td>[(ten, CD), (year, NNS), (ago, RB), (,, ,), (i...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>dk5</td>\n",
       "      <td>Turkish</td>\n",
       "      <td>Female</td>\n",
       "      <td>4</td>\n",
       "      <td>w</td>\n",
       "      <td>12</td>\n",
       "      <td>1</td>\n",
       "      <td>64</td>\n",
       "      <td>In my country we usually don't use tea bags. F...</td>\n",
       "      <td>[In, my, country, we, usually, do, n't, use, t...</td>\n",
       "      <td>[(In, IN), (my, PRP$), (country, NN), (we, PRP...</td>\n",
       "      <td>[in, my, country, we, usually, do, not, use, t...</td>\n",
       "      <td>[(in, IN), (my, PRP$), (country, NN), (we, PRP...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>dk5</td>\n",
       "      <td>Turkish</td>\n",
       "      <td>Female</td>\n",
       "      <td>4</td>\n",
       "      <td>w</td>\n",
       "      <td>13</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>I organized the instructions by time.</td>\n",
       "      <td>[I, organized, the, instructions, by, time, .]</td>\n",
       "      <td>[(I, PRP), (organized, VBD), (the, DT), (instr...</td>\n",
       "      <td>[i, organize, the, instruction, by, time, .]</td>\n",
       "      <td>[(i, PRP), (organize, VBD), (the, DT), (instru...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>ad1</td>\n",
       "      <td>Korean</td>\n",
       "      <td>Female</td>\n",
       "      <td>4</td>\n",
       "      <td>w</td>\n",
       "      <td>12</td>\n",
       "      <td>1</td>\n",
       "      <td>59</td>\n",
       "      <td>First, prepare a port, loose tea, and cup.\\nSe...</td>\n",
       "      <td>[First, ,, prepare, a, port, ,, loose, tea, ,,...</td>\n",
       "      <td>[(First, RB), (,, ,), (prepare, VB), (a, DT), ...</td>\n",
       "      <td>[first, ,, prepare, a, port, ,, loose, tea, ,,...</td>\n",
       "      <td>[(first, RB), (,, ,), (prepare, VB), (a, DT), ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          anon_id       L1  gender level_id class_id question_id version  \\\n",
       "answer_id                                                                  \n",
       "1             eq0   Arabic    Male        4        g           5       1   \n",
       "2             am8     Thai  Female        4        g           5       1   \n",
       "3             dk5  Turkish  Female        4        w          12       1   \n",
       "4             dk5  Turkish  Female        4        w          13       1   \n",
       "5             ad1   Korean  Female        4        w          12       1   \n",
       "\n",
       "           text_len                                               text  \\\n",
       "answer_id                                                                \n",
       "1               177  I met my friend Nife while I was studying in a...   \n",
       "2               137  Ten years ago, I met a women on the train betw...   \n",
       "3                64  In my country we usually don't use tea bags. F...   \n",
       "4                 6              I organized the instructions by time.   \n",
       "5                59  First, prepare a port, loose tea, and cup.\\nSe...   \n",
       "\n",
       "                                                      tokens  \\\n",
       "answer_id                                                      \n",
       "1          [I, met, my, friend, Nife, while, I, was, stud...   \n",
       "2          [Ten, years, ago, ,, I, met, a, women, on, the...   \n",
       "3          [In, my, country, we, usually, do, n't, use, t...   \n",
       "4             [I, organized, the, instructions, by, time, .]   \n",
       "5          [First, ,, prepare, a, port, ,, loose, tea, ,,...   \n",
       "\n",
       "                                                     tok_POS  \\\n",
       "answer_id                                                      \n",
       "1          [(I, PRP), (met, VBD), (my, PRP$), (friend, NN...   \n",
       "2          [(Ten, CD), (years, NNS), (ago, RB), (,, ,), (...   \n",
       "3          [(In, IN), (my, PRP$), (country, NN), (we, PRP...   \n",
       "4          [(I, PRP), (organized, VBD), (the, DT), (instr...   \n",
       "5          [(First, RB), (,, ,), (prepare, VB), (a, DT), ...   \n",
       "\n",
       "                                                      lemmas  \\\n",
       "answer_id                                                      \n",
       "1          [i, meet, my, friend, nife, while, i, be, stud...   \n",
       "2          [ten, year, ago, ,, i, meet, a, woman, on, the...   \n",
       "3          [in, my, country, we, usually, do, not, use, t...   \n",
       "4               [i, organize, the, instruction, by, time, .]   \n",
       "5          [first, ,, prepare, a, port, ,, loose, tea, ,,...   \n",
       "\n",
       "                                                   lemma_POS  \n",
       "answer_id                                                     \n",
       "1          [(i, PRP), (meet, VBD), (my, PRP$), (friend, N...  \n",
       "2          [(ten, CD), (year, NNS), (ago, RB), (,, ,), (i...  \n",
       "3          [(in, IN), (my, PRP$), (country, NN), (we, PRP...  \n",
       "4          [(i, PRP), (organize, VBD), (the, DT), (instru...  \n",
       "5          [(first, RB), (,, ,), (prepare, VB), (a, DT), ...  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pelic_df = pd.read_csv(\"../PELIC_compiled.csv\", index_col = 'answer_id', # answer_id is unique\n",
    "                      dtype = {'level_id':'object','question_id':'object','version':'object'}, #str not ints\n",
    "                               converters={'tokens':literal_eval,'tok_POS':literal_eval,'lemmas':literal_eval,\n",
    "                                          'lemma_POS': literal_eval,}) # Read in as lists\n",
    "pelic_df.info()\n",
    "pelic_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** Here we have read in the PELIC_compiled csv. To see how this csv was created from the raw data files, please see the [build_PELIC_compiled.ipynb](https://github.com/ELI-Data-Mining-Group/PELIC-dataset/blob/master/PELIC_compiled.csv) notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Narrowing the dataset\n",
    "- For valid natural production, we may want to include only free written production, i.e. from writing classes and not reading or grammar classes.\n",
    "- Only the first versions of texts will be included to avoid duplicates and corrections made due to teacher feedback.\n",
    "- Only levels 3,4,5 will be included as there are very few level 2 classes, making later analysis of proficiency as a factor unuseful."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "New number of texts: 14873\n",
      "New number of tokens: 2654899\n"
     ]
    }
   ],
   "source": [
    "# Keep only writing texts (class id 'w')\n",
    "\n",
    "pelic_df = pelic_df.loc[pelic_df.class_id == 'w']\n",
    "print('New number of texts:',len(pelic_df))\n",
    "print('New number of tokens:',pelic_df['text_len'].sum())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "New number of texts: 12981\n",
      "New number of tokens: 2256633\n"
     ]
    }
   ],
   "source": [
    "# Keep only version 1 of texts\n",
    "\n",
    "pelic_df = pelic_df.loc[pelic_df.version == '1']\n",
    "print('New number of texts:',len(pelic_df))\n",
    "print('New number of tokens:',pelic_df['text_len'].sum())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4    4888\n",
       "5    4273\n",
       "3    3463\n",
       "2     357\n",
       "Name: level_id, dtype: int64"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "New number of texts: 12624\n",
      "New number of tokens: 2252791\n"
     ]
    }
   ],
   "source": [
    "# Remove level 2 students\n",
    "\n",
    "texts_per_level = pelic_df.level_id.value_counts()\n",
    "texts_per_level\n",
    "pelic_df = pelic_df.loc[pelic_df.level_id != '2']\n",
    "print('New number of texts:',len(pelic_df))\n",
    "print('New number of tokens:',pelic_df['text_len'].sum())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building a concordance list\n",
    "This section shows how to use the concordance function from [`pelitk`](https://github.com/ELI-Data-Mining-Group/pelitk) to build a create a concordance list for the set of verbs being analyzed. These concordances are stored in a dataframe containing other useful identifying information, but can also be printed as a stand-alone list."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Creating the verb list\n",
    "First, we can create a list of the nine verbs we want to find concordances for.  \n",
    "**Note:** This is a list of the lemma forms, so that _consider_ also includes inflections like _considers,_ _considered,_ etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "verbs = ['consider', 'suggest', 'explain', 'realize', 'admit', 'deny', 'conclude', 'recommend', 'suppose']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Concordancing a list of items\n",
    "The concordance function takes one word or tuple as the 'node' argument, as seen in the example below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pretty printing has been turned ON\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['                                  Andrea  concluded   her speech with a joke                  ',\n",
       " '                    with a joke . Andrea  concluded   that he was telling the                 ']"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "example_text = 'Andrea concluded her speech with a joke. Andrea concluded that he was telling the truth.'\n",
    "example_tok_text = ['Andrea', 'concluded', 'her', 'speech', 'with', 'a', 'joke', '.', 'Andrea',\n",
    "                    'concluded', 'that', 'he', 'was', 'telling', 'the', 'truth', '.']\n",
    "\n",
    "%pprint\n",
    "conc.concordance(example_tok_text,'concluded',5,pretty=True)\n",
    "# Update to use lex.tokenize once it's ready"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, as we want to find concordances for a **list** of node words, an additional function needs to be created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create function get_concs function which creates a concordance line for each occurrence of any item in our list\n",
    "\n",
    "def get_concs(tok_text, forms_list):\n",
    "    conclist = []\n",
    "    for x in tok_text:\n",
    "        if x.lower() in [x for x in forms_list]: \n",
    "            conclist.append(conc.concordance(tok_text, x, 5))\n",
    "    return [x for y in list(unique_everseen(conclist)) for x in y]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Testing out this function on the same example text, we see that it returns concordance lines for _concluded_ and _speech_."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('    Andrea', 'concluded', 'her speech with a joke'),\n",
       " ('with a joke . Andrea', 'concluded', 'that he was telling the'),\n",
       " ('  Andrea concluded her', 'speech', 'with a joke . Andrea')]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "['                                  Andrea  concluded   her speech with a joke                  ',\n",
       " '                    with a joke . Andrea  concluded   that he was telling the                 ',\n",
       " '                    Andrea concluded her    speech    with a joke . Andrea                    ']"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "example_forms_list = ['concluded','speech']\n",
    "\n",
    "get_concs(example_tok_text, example_forms_list)\n",
    "\n",
    "#The 'prettify' function can also be applied if desired:\n",
    "conc.prettify(get_concs(example_tok_text, example_forms_list))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now apply this new function to our entire dataframe, creating a new `concordance` column which will include all the concordances for the verbs in our list which appear in each text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "pelic_df['concordance'] = pelic_df['lemmas'].apply(lambda x: get_concs(x,verbs))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total number of texts: 12624\n",
      "Number of texts containing at least one lemma from verb list 1856\n",
      "Percentage of texts containing lemma from verb list: 14.7 %\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>anon_id</th>\n",
       "      <th>L1</th>\n",
       "      <th>gender</th>\n",
       "      <th>level_id</th>\n",
       "      <th>class_id</th>\n",
       "      <th>question_id</th>\n",
       "      <th>version</th>\n",
       "      <th>text_len</th>\n",
       "      <th>text</th>\n",
       "      <th>tokens</th>\n",
       "      <th>tok_POS</th>\n",
       "      <th>lemmas</th>\n",
       "      <th>lemma_POS</th>\n",
       "      <th>concordance</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>answer_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>29379</td>\n",
       "      <td>gj2</td>\n",
       "      <td>Arabic</td>\n",
       "      <td>Male</td>\n",
       "      <td>5</td>\n",
       "      <td>w</td>\n",
       "      <td>3899</td>\n",
       "      <td>1</td>\n",
       "      <td>467</td>\n",
       "      <td>Favorite Restaurant\\n My friend graduated from...</td>\n",
       "      <td>[Favorite, Restaurant, My, friend, graduated, ...</td>\n",
       "      <td>[(Favorite, NNP), (Restaurant, NNP), (My, NNP)...</td>\n",
       "      <td>[favorite, restaurant, my, friend, graduate, f...</td>\n",
       "      <td>[(favorite, NNP), (restaurant, NNP), (my, NNP)...</td>\n",
       "      <td>[(, he call me and, suggest, two restaurant th...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>45035</td>\n",
       "      <td>fw7</td>\n",
       "      <td>Chinese</td>\n",
       "      <td>Female</td>\n",
       "      <td>4</td>\n",
       "      <td>w</td>\n",
       "      <td>5717</td>\n",
       "      <td>1</td>\n",
       "      <td>484</td>\n",
       "      <td>\\n Topic: Online Learning \\n\\n Nowadays is a m...</td>\n",
       "      <td>[Topic, :, Online, Learning, Nowadays, is, a, ...</td>\n",
       "      <td>[(Topic, NN), (:, :), (Online, NNP), (Learning...</td>\n",
       "      <td>[topic, :, online, ##NO-MATCHING-POS##, nowada...</td>\n",
       "      <td>[(topic, NN), (:, :), (online, NNP), (##NO-MAT...</td>\n",
       "      <td>[(. next , let us, consider, that if student t...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>23696</td>\n",
       "      <td>ck4</td>\n",
       "      <td>Arabic</td>\n",
       "      <td>Male</td>\n",
       "      <td>4</td>\n",
       "      <td>w</td>\n",
       "      <td>3072</td>\n",
       "      <td>1</td>\n",
       "      <td>825</td>\n",
       "      <td>Nowadays, prices and life expenses have becom...</td>\n",
       "      <td>[Nowadays, ,, prices, and, life, expenses, hav...</td>\n",
       "      <td>[(Nowadays, NNS), (,, ,), (prices, NNS), (and,...</td>\n",
       "      <td>[nowadays, ,, price, and, life, expense, have,...</td>\n",
       "      <td>[(nowadays, NNS), (,, ,), (price, NNS), (and, ...</td>\n",
       "      <td>[(for another offer and he, realize, he have l...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>26580</td>\n",
       "      <td>ay3</td>\n",
       "      <td>Japanese</td>\n",
       "      <td>Female</td>\n",
       "      <td>5</td>\n",
       "      <td>w</td>\n",
       "      <td>3466</td>\n",
       "      <td>1</td>\n",
       "      <td>609</td>\n",
       "      <td>The Difference between Macintosh and Windows\\n...</td>\n",
       "      <td>[The, Difference, between, Macintosh, and, Win...</td>\n",
       "      <td>[(The, DT), (Difference, NNP), (between, IN), ...</td>\n",
       "      <td>[the, difference, between, macintosh, and, win...</td>\n",
       "      <td>[(the, DT), (difference, NNP), (between, IN), ...</td>\n",
       "      <td>[(, it be possible to, conclude, the macintosh...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>40665</td>\n",
       "      <td>ct4</td>\n",
       "      <td>Arabic</td>\n",
       "      <td>Male</td>\n",
       "      <td>4</td>\n",
       "      <td>w</td>\n",
       "      <td>5189</td>\n",
       "      <td>1</td>\n",
       "      <td>57</td>\n",
       "      <td>1. Chefs enjouy preparing usual meals.\\n\\n2. T...</td>\n",
       "      <td>[1, ., Chefs, enjouy, preparing, usual, meals,...</td>\n",
       "      <td>[(1, CD), (., .), (Chefs, NNP), (enjouy, VBD),...</td>\n",
       "      <td>[1, ., chef, enjouy, prepare, usual, meal, ., ...</td>\n",
       "      <td>[(1, CD), (., .), (chef, NNP), (enjouy, VBD), ...</td>\n",
       "      <td>[(. the head of state, consider, form a allian...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          anon_id        L1  gender level_id class_id question_id version  \\\n",
       "answer_id                                                                   \n",
       "29379         gj2    Arabic    Male        5        w        3899       1   \n",
       "45035         fw7   Chinese  Female        4        w        5717       1   \n",
       "23696         ck4    Arabic    Male        4        w        3072       1   \n",
       "26580         ay3  Japanese  Female        5        w        3466       1   \n",
       "40665         ct4    Arabic    Male        4        w        5189       1   \n",
       "\n",
       "           text_len                                               text  \\\n",
       "answer_id                                                                \n",
       "29379           467  Favorite Restaurant\\n My friend graduated from...   \n",
       "45035           484  \\n Topic: Online Learning \\n\\n Nowadays is a m...   \n",
       "23696           825   Nowadays, prices and life expenses have becom...   \n",
       "26580           609  The Difference between Macintosh and Windows\\n...   \n",
       "40665            57  1. Chefs enjouy preparing usual meals.\\n\\n2. T...   \n",
       "\n",
       "                                                      tokens  \\\n",
       "answer_id                                                      \n",
       "29379      [Favorite, Restaurant, My, friend, graduated, ...   \n",
       "45035      [Topic, :, Online, Learning, Nowadays, is, a, ...   \n",
       "23696      [Nowadays, ,, prices, and, life, expenses, hav...   \n",
       "26580      [The, Difference, between, Macintosh, and, Win...   \n",
       "40665      [1, ., Chefs, enjouy, preparing, usual, meals,...   \n",
       "\n",
       "                                                     tok_POS  \\\n",
       "answer_id                                                      \n",
       "29379      [(Favorite, NNP), (Restaurant, NNP), (My, NNP)...   \n",
       "45035      [(Topic, NN), (:, :), (Online, NNP), (Learning...   \n",
       "23696      [(Nowadays, NNS), (,, ,), (prices, NNS), (and,...   \n",
       "26580      [(The, DT), (Difference, NNP), (between, IN), ...   \n",
       "40665      [(1, CD), (., .), (Chefs, NNP), (enjouy, VBD),...   \n",
       "\n",
       "                                                      lemmas  \\\n",
       "answer_id                                                      \n",
       "29379      [favorite, restaurant, my, friend, graduate, f...   \n",
       "45035      [topic, :, online, ##NO-MATCHING-POS##, nowada...   \n",
       "23696      [nowadays, ,, price, and, life, expense, have,...   \n",
       "26580      [the, difference, between, macintosh, and, win...   \n",
       "40665      [1, ., chef, enjouy, prepare, usual, meal, ., ...   \n",
       "\n",
       "                                                   lemma_POS  \\\n",
       "answer_id                                                      \n",
       "29379      [(favorite, NNP), (restaurant, NNP), (my, NNP)...   \n",
       "45035      [(topic, NN), (:, :), (online, NNP), (##NO-MAT...   \n",
       "23696      [(nowadays, NNS), (,, ,), (price, NNS), (and, ...   \n",
       "26580      [(the, DT), (difference, NNP), (between, IN), ...   \n",
       "40665      [(1, CD), (., .), (chef, NNP), (enjouy, VBD), ...   \n",
       "\n",
       "                                                 concordance  \n",
       "answer_id                                                     \n",
       "29379      [(, he call me and, suggest, two restaurant th...  \n",
       "45035      [(. next , let us, consider, that if student t...  \n",
       "23696      [(for another offer and he, realize, he have l...  \n",
       "26580      [(, it be possible to, conclude, the macintosh...  \n",
       "40665      [(. the head of state, consider, form a allian...  "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check how many texts in PELIC contain our target items\n",
    "\n",
    "print('Total number of texts:',len(pelic_df))\n",
    "print('Number of texts containing at least one lemma from verb list',len(pelic_df.loc[~pelic_df.concordance.str.len().eq(0)]))\n",
    "print('Percentage of texts containing lemma from verb list:', \n",
    "      round((len(pelic_df.loc[~pelic_df.concordance.str.len().eq(0)])/len(pelic_df))*100,2),'%')\n",
    "pelic_df.loc[~pelic_df.concordance.str.len().eq(0)].sample(5) # Sample of ten rows"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since we are only interested in texts containing lemmas from our verb list, we will remove all other rows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "verbs_df = pelic_df.loc[~pelic_df.concordance.str.len().eq(0)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Labelling the concordance lines\n",
    "To be able to easily refer back to concordance lines later, it is useful to attach identifying information to each one, e.g. what the node word is and where it can be found in the text (i.e. the index)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create function to label each concordance with the offset, i.e. the node and its index in the text\n",
    "\n",
    "def get_offset(tok_list, forms):\n",
    "    new_list =  [x.lower() for x in tok_list.copy()] # lower case all tokens in text\n",
    "    new_forms = forms.copy()\n",
    "    return [x for x in list(enumerate(new_list)) if x[1] in new_forms] # enumerate function finds index of each token in the text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>anon_id</th>\n",
       "      <th>L1</th>\n",
       "      <th>gender</th>\n",
       "      <th>level_id</th>\n",
       "      <th>class_id</th>\n",
       "      <th>question_id</th>\n",
       "      <th>version</th>\n",
       "      <th>text_len</th>\n",
       "      <th>text</th>\n",
       "      <th>tokens</th>\n",
       "      <th>tok_POS</th>\n",
       "      <th>lemmas</th>\n",
       "      <th>lemma_POS</th>\n",
       "      <th>concordance</th>\n",
       "      <th>offset</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>answer_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>34</td>\n",
       "      <td>bf0</td>\n",
       "      <td>Arabic</td>\n",
       "      <td>Male</td>\n",
       "      <td>4</td>\n",
       "      <td>w</td>\n",
       "      <td>10</td>\n",
       "      <td>1</td>\n",
       "      <td>171</td>\n",
       "      <td>How to fail a test\\n\\n There are several ways ...</td>\n",
       "      <td>[How, to, fail, a, test, There, are, several, ...</td>\n",
       "      <td>[(How, WRB), (to, TO), (fail, VB), (a, DT), (t...</td>\n",
       "      <td>[how, to, fail, a, test, there, be, several, w...</td>\n",
       "      <td>[(how, WRB), (to, TO), (fail, VB), (a, DT), (t...</td>\n",
       "      <td>[(with you then , i, recommend, you to cheat i...</td>\n",
       "      <td>[(114, recommend)]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>111</td>\n",
       "      <td>cz7</td>\n",
       "      <td>Arabic</td>\n",
       "      <td>Male</td>\n",
       "      <td>4</td>\n",
       "      <td>w</td>\n",
       "      <td>15</td>\n",
       "      <td>1</td>\n",
       "      <td>51</td>\n",
       "      <td>Every one like a special kind of food. For me ...</td>\n",
       "      <td>[Every, one, like, a, special, kind, of, food,...</td>\n",
       "      <td>[(Every, DT), (one, CD), (like, IN), (a, DT), ...</td>\n",
       "      <td>[every, one, like, a, special, kind, of, food,...</td>\n",
       "      <td>[(every, DT), (one, CD), (like, IN), (a, DT), ...</td>\n",
       "      <td>[(specially kabsah and i will, explain, how to...</td>\n",
       "      <td>[(22, explain)]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>119</td>\n",
       "      <td>cs3</td>\n",
       "      <td>Japanese</td>\n",
       "      <td>Male</td>\n",
       "      <td>4</td>\n",
       "      <td>w</td>\n",
       "      <td>6</td>\n",
       "      <td>1</td>\n",
       "      <td>235</td>\n",
       "      <td>It's difficult to succeed in getting a higher ...</td>\n",
       "      <td>[It, 's, difficult, to, succeed, in, getting, ...</td>\n",
       "      <td>[(It, PRP), ('s, VBZ), (difficult, JJ), (to, T...</td>\n",
       "      <td>[it, 's, difficult, to, succeed, in, get, a, h...</td>\n",
       "      <td>[(it, PRP), ('s, VBZ), (difficult, JJ), (to, T...</td>\n",
       "      <td>[(. but he would not, realize, that it 's the ...</td>\n",
       "      <td>[(200, realize), (231, explain)]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>133</td>\n",
       "      <td>az2</td>\n",
       "      <td>Korean</td>\n",
       "      <td>Male</td>\n",
       "      <td>5</td>\n",
       "      <td>w</td>\n",
       "      <td>17</td>\n",
       "      <td>1</td>\n",
       "      <td>130</td>\n",
       "      <td>When I was in Germany, I met a friend who was ...</td>\n",
       "      <td>[When, I, was, in, Germany, ,, I, met, a, frie...</td>\n",
       "      <td>[(When, WRB), (I, PRP), (was, VBD), (in, IN), ...</td>\n",
       "      <td>[when, i, be, in, germany, ,, i, meet, a, frie...</td>\n",
       "      <td>[(when, WRB), (i, PRP), (be, VBD), (in, IN), (...</td>\n",
       "      <td>[(about my country . i, realize, i be wrong wh...</td>\n",
       "      <td>[(113, realize)]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>152</td>\n",
       "      <td>dj0</td>\n",
       "      <td>Korean</td>\n",
       "      <td>Female</td>\n",
       "      <td>5</td>\n",
       "      <td>w</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>299</td>\n",
       "      <td>There are many qualities of a good neighbor in...</td>\n",
       "      <td>[There, are, many, qualities, of, a, good, nei...</td>\n",
       "      <td>[(There, EX), (are, VBP), (many, JJ), (qualiti...</td>\n",
       "      <td>[there, be, many, quality, of, a, good, neighb...</td>\n",
       "      <td>[(there, EX), (be, VBP), (many, JJ), (quality,...</td>\n",
       "      <td>[(bad neighbor once , i, realize, a big instru...</td>\n",
       "      <td>[(308, realize), (321, consider)]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          anon_id        L1  gender level_id class_id question_id version  \\\n",
       "answer_id                                                                   \n",
       "34            bf0    Arabic    Male        4        w          10       1   \n",
       "111           cz7    Arabic    Male        4        w          15       1   \n",
       "119           cs3  Japanese    Male        4        w           6       1   \n",
       "133           az2    Korean    Male        5        w          17       1   \n",
       "152           dj0    Korean  Female        5        w           4       1   \n",
       "\n",
       "           text_len                                               text  \\\n",
       "answer_id                                                                \n",
       "34              171  How to fail a test\\n\\n There are several ways ...   \n",
       "111              51  Every one like a special kind of food. For me ...   \n",
       "119             235  It's difficult to succeed in getting a higher ...   \n",
       "133             130  When I was in Germany, I met a friend who was ...   \n",
       "152             299  There are many qualities of a good neighbor in...   \n",
       "\n",
       "                                                      tokens  \\\n",
       "answer_id                                                      \n",
       "34         [How, to, fail, a, test, There, are, several, ...   \n",
       "111        [Every, one, like, a, special, kind, of, food,...   \n",
       "119        [It, 's, difficult, to, succeed, in, getting, ...   \n",
       "133        [When, I, was, in, Germany, ,, I, met, a, frie...   \n",
       "152        [There, are, many, qualities, of, a, good, nei...   \n",
       "\n",
       "                                                     tok_POS  \\\n",
       "answer_id                                                      \n",
       "34         [(How, WRB), (to, TO), (fail, VB), (a, DT), (t...   \n",
       "111        [(Every, DT), (one, CD), (like, IN), (a, DT), ...   \n",
       "119        [(It, PRP), ('s, VBZ), (difficult, JJ), (to, T...   \n",
       "133        [(When, WRB), (I, PRP), (was, VBD), (in, IN), ...   \n",
       "152        [(There, EX), (are, VBP), (many, JJ), (qualiti...   \n",
       "\n",
       "                                                      lemmas  \\\n",
       "answer_id                                                      \n",
       "34         [how, to, fail, a, test, there, be, several, w...   \n",
       "111        [every, one, like, a, special, kind, of, food,...   \n",
       "119        [it, 's, difficult, to, succeed, in, get, a, h...   \n",
       "133        [when, i, be, in, germany, ,, i, meet, a, frie...   \n",
       "152        [there, be, many, quality, of, a, good, neighb...   \n",
       "\n",
       "                                                   lemma_POS  \\\n",
       "answer_id                                                      \n",
       "34         [(how, WRB), (to, TO), (fail, VB), (a, DT), (t...   \n",
       "111        [(every, DT), (one, CD), (like, IN), (a, DT), ...   \n",
       "119        [(it, PRP), ('s, VBZ), (difficult, JJ), (to, T...   \n",
       "133        [(when, WRB), (i, PRP), (be, VBD), (in, IN), (...   \n",
       "152        [(there, EX), (be, VBP), (many, JJ), (quality,...   \n",
       "\n",
       "                                                 concordance  \\\n",
       "answer_id                                                      \n",
       "34         [(with you then , i, recommend, you to cheat i...   \n",
       "111        [(specially kabsah and i will, explain, how to...   \n",
       "119        [(. but he would not, realize, that it 's the ...   \n",
       "133        [(about my country . i, realize, i be wrong wh...   \n",
       "152        [(bad neighbor once , i, realize, a big instru...   \n",
       "\n",
       "                                      offset  \n",
       "answer_id                                     \n",
       "34                        [(114, recommend)]  \n",
       "111                          [(22, explain)]  \n",
       "119         [(200, realize), (231, explain)]  \n",
       "133                         [(113, realize)]  \n",
       "152        [(308, realize), (321, consider)]  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create an offset column by applying the above function\n",
    "\n",
    "verbs_df['offset'] = verbs_df.lemmas.apply(lambda x: get_offset(x,verbs))\n",
    "verbs_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# For clarity, we want to sort the concordance and offset columns alphabetically by the node word\n",
    "\n",
    "# Sort offset column\n",
    "for x in verbs_df['offset']:\n",
    "    x.sort(key = operator.itemgetter(1))\n",
    "    \n",
    "    \n",
    "# Sort concordance column\n",
    "verbs_df['concordance'] = [sorted(x, key=lambda x: x[1]) for x in verbs_df.concordance]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>anon_id</th>\n",
       "      <th>L1</th>\n",
       "      <th>gender</th>\n",
       "      <th>level_id</th>\n",
       "      <th>class_id</th>\n",
       "      <th>question_id</th>\n",
       "      <th>version</th>\n",
       "      <th>text_len</th>\n",
       "      <th>text</th>\n",
       "      <th>tokens</th>\n",
       "      <th>tok_POS</th>\n",
       "      <th>lemmas</th>\n",
       "      <th>lemma_POS</th>\n",
       "      <th>concordance</th>\n",
       "      <th>offset</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>answer_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>34</td>\n",
       "      <td>bf0</td>\n",
       "      <td>Arabic</td>\n",
       "      <td>Male</td>\n",
       "      <td>4</td>\n",
       "      <td>w</td>\n",
       "      <td>10</td>\n",
       "      <td>1</td>\n",
       "      <td>171</td>\n",
       "      <td>How to fail a test\\n\\n There are several ways ...</td>\n",
       "      <td>[How, to, fail, a, test, There, are, several, ...</td>\n",
       "      <td>[(How, WRB), (to, TO), (fail, VB), (a, DT), (t...</td>\n",
       "      <td>[how, to, fail, a, test, there, be, several, w...</td>\n",
       "      <td>[(how, WRB), (to, TO), (fail, VB), (a, DT), (t...</td>\n",
       "      <td>[(with you then , i, recommend, you to cheat i...</td>\n",
       "      <td>[(114, recommend)]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          anon_id      L1 gender level_id class_id question_id version  \\\n",
       "answer_id                                                                \n",
       "34            bf0  Arabic   Male        4        w          10       1   \n",
       "\n",
       "           text_len                                               text  \\\n",
       "answer_id                                                                \n",
       "34              171  How to fail a test\\n\\n There are several ways ...   \n",
       "\n",
       "                                                      tokens  \\\n",
       "answer_id                                                      \n",
       "34         [How, to, fail, a, test, There, are, several, ...   \n",
       "\n",
       "                                                     tok_POS  \\\n",
       "answer_id                                                      \n",
       "34         [(How, WRB), (to, TO), (fail, VB), (a, DT), (t...   \n",
       "\n",
       "                                                      lemmas  \\\n",
       "answer_id                                                      \n",
       "34         [how, to, fail, a, test, there, be, several, w...   \n",
       "\n",
       "                                                   lemma_POS  \\\n",
       "answer_id                                                      \n",
       "34         [(how, WRB), (to, TO), (fail, VB), (a, DT), (t...   \n",
       "\n",
       "                                                 concordance  \\\n",
       "answer_id                                                      \n",
       "34         [(with you then , i, recommend, you to cheat i...   \n",
       "\n",
       "                       offset  \n",
       "answer_id                      \n",
       "34         [(114, recommend)]  "
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "verbs_df.head(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Creating the conc_df\n",
    "Up until now, the dataframe has had each **text** as a row. However, we may want to have each **concordance** as a row."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>answer_id</th>\n",
       "      <th>level_id</th>\n",
       "      <th>L1</th>\n",
       "      <th>gender</th>\n",
       "      <th>offset</th>\n",
       "      <th>concordance</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>34</td>\n",
       "      <td>4</td>\n",
       "      <td>Arabic</td>\n",
       "      <td>Male</td>\n",
       "      <td>[(114, recommend)]</td>\n",
       "      <td>[(with you then , i, recommend, you to cheat i...</td>\n",
       "      <td>How to fail a test\\n\\n There are several ways ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>111</td>\n",
       "      <td>4</td>\n",
       "      <td>Arabic</td>\n",
       "      <td>Male</td>\n",
       "      <td>[(22, explain)]</td>\n",
       "      <td>[(specially kabsah and i will, explain, how to...</td>\n",
       "      <td>Every one like a special kind of food. For me ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   answer_id level_id      L1 gender              offset  \\\n",
       "0         34        4  Arabic   Male  [(114, recommend)]   \n",
       "1        111        4  Arabic   Male     [(22, explain)]   \n",
       "\n",
       "                                         concordance  \\\n",
       "0  [(with you then , i, recommend, you to cheat i...   \n",
       "1  [(specially kabsah and i will, explain, how to...   \n",
       "\n",
       "                                                text  \n",
       "0  How to fail a test\\n\\n There are several ways ...  \n",
       "1  Every one like a special kind of food. For me ...  "
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create the new dataframe based on verbs_df\n",
    "\n",
    "# Keep only most relevant columns and reset the index \n",
    "conc_df = verbs_df[['level_id','L1','gender','offset','concordance','text']].reset_index()\n",
    "conc_df.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** Depending on the desired analysis, we could keep other columns too, e.g. `question_id` to analyze the prompts or `toks_re_len` to consider the impact of text length. For clarity, here we are maintaining only essential information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>answer_id</th>\n",
       "      <th>level_id</th>\n",
       "      <th>L1</th>\n",
       "      <th>gender</th>\n",
       "      <th>text</th>\n",
       "      <th>offset_conc</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>34</td>\n",
       "      <td>4</td>\n",
       "      <td>Arabic</td>\n",
       "      <td>Male</td>\n",
       "      <td>How to fail a test\\n\\n There are several ways ...</td>\n",
       "      <td>[((114, recommend), (with you then , i, recomm...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>111</td>\n",
       "      <td>4</td>\n",
       "      <td>Arabic</td>\n",
       "      <td>Male</td>\n",
       "      <td>Every one like a special kind of food. For me ...</td>\n",
       "      <td>[((22, explain), (specially kabsah and i will,...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   answer_id level_id      L1 gender  \\\n",
       "0         34        4  Arabic   Male   \n",
       "1        111        4  Arabic   Male   \n",
       "\n",
       "                                                text  \\\n",
       "0  How to fail a test\\n\\n There are several ways ...   \n",
       "1  Every one like a special kind of food. For me ...   \n",
       "\n",
       "                                         offset_conc  \n",
       "0  [((114, recommend), (with you then , i, recomm...  \n",
       "1  [((22, explain), (specially kabsah and i will,...  "
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a new column with tuples of the offset and the concordance and delete old offset and concordance columns\n",
    "conc_df['offset_conc'] = list(zip(conc_df.offset, conc_df.concordance)) # Zip together two columns\n",
    "conc_df['offset_conc'] = [list(zip(x[0],x[1])) for x in conc_df.offset_conc] # Zip together the items in each row\n",
    "conc_df = conc_df.drop(['offset','concordance'], axis = 1)\n",
    "\n",
    "conc_df.head(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 'Explode' the dataframe so that each item in the 'offset_conc' column becomes its own row\n",
    "conc_df = conc_df.explode('offset_conc')\n",
    "conc_df = conc_df.reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>answer_id</th>\n",
       "      <th>level_id</th>\n",
       "      <th>L1</th>\n",
       "      <th>gender</th>\n",
       "      <th>text</th>\n",
       "      <th>offset</th>\n",
       "      <th>concordance</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>34</td>\n",
       "      <td>4</td>\n",
       "      <td>Arabic</td>\n",
       "      <td>Male</td>\n",
       "      <td>How to fail a test\\n\\n There are several ways ...</td>\n",
       "      <td>(114, recommend)</td>\n",
       "      <td>(with you then , i, recommend, you to cheat in...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>111</td>\n",
       "      <td>4</td>\n",
       "      <td>Arabic</td>\n",
       "      <td>Male</td>\n",
       "      <td>Every one like a special kind of food. For me ...</td>\n",
       "      <td>(22, explain)</td>\n",
       "      <td>(specially kabsah and i will, explain, how to ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>119</td>\n",
       "      <td>4</td>\n",
       "      <td>Japanese</td>\n",
       "      <td>Male</td>\n",
       "      <td>It's difficult to succeed in getting a higher ...</td>\n",
       "      <td>(231, explain)</td>\n",
       "      <td>(high education . 2 ., explain, a difficulty a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>119</td>\n",
       "      <td>4</td>\n",
       "      <td>Japanese</td>\n",
       "      <td>Male</td>\n",
       "      <td>It's difficult to succeed in getting a higher ...</td>\n",
       "      <td>(200, realize)</td>\n",
       "      <td>(. but he would not, realize, that it 's the c...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>133</td>\n",
       "      <td>5</td>\n",
       "      <td>Korean</td>\n",
       "      <td>Male</td>\n",
       "      <td>When I was in Germany, I met a friend who was ...</td>\n",
       "      <td>(113, realize)</td>\n",
       "      <td>(about my country . i, realize, i be wrong whe...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   answer_id level_id        L1 gender  \\\n",
       "0         34        4    Arabic   Male   \n",
       "1        111        4    Arabic   Male   \n",
       "2        119        4  Japanese   Male   \n",
       "3        119        4  Japanese   Male   \n",
       "4        133        5    Korean   Male   \n",
       "\n",
       "                                                text            offset  \\\n",
       "0  How to fail a test\\n\\n There are several ways ...  (114, recommend)   \n",
       "1  Every one like a special kind of food. For me ...     (22, explain)   \n",
       "2  It's difficult to succeed in getting a higher ...    (231, explain)   \n",
       "3  It's difficult to succeed in getting a higher ...    (200, realize)   \n",
       "4  When I was in Germany, I met a friend who was ...    (113, realize)   \n",
       "\n",
       "                                         concordance  \n",
       "0  (with you then , i, recommend, you to cheat in...  \n",
       "1  (specially kabsah and i will, explain, how to ...  \n",
       "2  (high education . 2 ., explain, a difficulty a...  \n",
       "3  (. but he would not, realize, that it 's the c...  \n",
       "4  (about my country . i, realize, i be wrong whe...  "
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Re-split the offset_conc back into two columns and then dropping it\n",
    "conc_df['offset'] = [x[0] for x in conc_df.offset_conc]\n",
    "conc_df['concordance'] = [x[1] for x in conc_df.offset_conc] # NOTE: commas are around the node word (not from Ss)\n",
    "conc_df = conc_df.drop(['offset_conc'], axis = 1)\n",
    "conc_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sort dataframe by offset (word then offset number)\n",
    "conc_df['offset_num'] = [x[0] for x in conc_df.offset]\n",
    "conc_df['offset_word'] = [x[1] for x in conc_df.offset]\n",
    "conc_df = conc_df.sort_values(by = ['offset_word', 'answer_id','offset_num']).reset_index(drop=True)\n",
    "conc_df = conc_df.drop(['offset_num','offset_word'], axis = 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2795"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>answer_id</th>\n",
       "      <th>level_id</th>\n",
       "      <th>L1</th>\n",
       "      <th>gender</th>\n",
       "      <th>text</th>\n",
       "      <th>offset</th>\n",
       "      <th>concordance</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>1551</td>\n",
       "      <td>5</td>\n",
       "      <td>Chinese</td>\n",
       "      <td>Male</td>\n",
       "      <td>In Chinese saying, students are \"like warms in...</td>\n",
       "      <td>(442, admit)</td>\n",
       "      <td>(them . most of us, admit, that this behavior ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>1736</td>\n",
       "      <td>4</td>\n",
       "      <td>Korean</td>\n",
       "      <td>Female</td>\n",
       "      <td>Cats and dogs are the most popular pets in the...</td>\n",
       "      <td>(312, admit)</td>\n",
       "      <td>(their master . when it, admit, someone as a m...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>1736</td>\n",
       "      <td>4</td>\n",
       "      <td>Korean</td>\n",
       "      <td>Female</td>\n",
       "      <td>Cats and dogs are the most popular pets in the...</td>\n",
       "      <td>(384, admit)</td>\n",
       "      <td>(us that cat do not, admit, its master . there...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2534</td>\n",
       "      <td>4</td>\n",
       "      <td>Arabic</td>\n",
       "      <td>Male</td>\n",
       "      <td>Writing in my language makes me do my best to ...</td>\n",
       "      <td>(134, admit)</td>\n",
       "      <td>(english , i have to, admit, the new vocabular...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>3830</td>\n",
       "      <td>4</td>\n",
       "      <td>Korean</td>\n",
       "      <td>Female</td>\n",
       "      <td>\\nWriting 4P\\n\\n31 Jan 07\\n\\nThe Problem and S...</td>\n",
       "      <td>(283, admit)</td>\n",
       "      <td>(to understanding each other and, admit, the f...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   answer_id level_id       L1  gender  \\\n",
       "0       1551        5  Chinese    Male   \n",
       "1       1736        4   Korean  Female   \n",
       "2       1736        4   Korean  Female   \n",
       "5       2534        4   Arabic    Male   \n",
       "6       3830        4   Korean  Female   \n",
       "\n",
       "                                                text        offset  \\\n",
       "0  In Chinese saying, students are \"like warms in...  (442, admit)   \n",
       "1  Cats and dogs are the most popular pets in the...  (312, admit)   \n",
       "2  Cats and dogs are the most popular pets in the...  (384, admit)   \n",
       "5  Writing in my language makes me do my best to ...  (134, admit)   \n",
       "6  \\nWriting 4P\\n\\n31 Jan 07\\n\\nThe Problem and S...  (283, admit)   \n",
       "\n",
       "                                         concordance  \n",
       "0  (them . most of us, admit, that this behavior ...  \n",
       "1  (their master . when it, admit, someone as a m...  \n",
       "2  (us that cat do not, admit, its master . there...  \n",
       "5  (english , i have to, admit, the new vocabular...  \n",
       "6  (to understanding each other and, admit, the f...  "
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "2418"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Dropping duplicates - from manual checking, it was found that some texts have two versions back to back, \n",
    "# some might be repetitions of task prompts, and some are mislabelled as version 1.\n",
    "len(conc_df)\n",
    "conc_df = conc_df.drop_duplicates(subset='concordance', keep=\"first\")\n",
    "conc_df.head()\n",
    "len(conc_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Concordance csv file\n",
    "It may be useful to create a csv file of the conc_df dataframe, e.g. to use in annotations, to share, or to use with other programs. Here, we create a simple csv containing just the identifying information and concordances."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Zip desired columns together to prepare the concordance csv\n",
    "conc_csv = zip(conc_df.answer_id, conc_df.offset, conc_df.concordance)\n",
    "conc_csv = list(conc_csv)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run the prettify function to prepare the csv for printing\n",
    "conc_csv = [(x[0], x[1], conc.prettify([x[2]])) for x in conc_csv]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2418"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "[(1551,\n",
       "  (442, 'admit'),\n",
       "  ['                       them . most of us    admit     that this behavior will lead            ']),\n",
       " (1736,\n",
       "  (312, 'admit'),\n",
       "  ['                  their master . when it    admit     someone as a master ,                   ']),\n",
       " (1736,\n",
       "  (384, 'admit'),\n",
       "  ['                      us that cat do not    admit     its master . there be                   ']),\n",
       " (2534,\n",
       "  (134, 'admit'),\n",
       "  ['                     english , i have to    admit     the new vocabulary word be              ']),\n",
       " (3830,\n",
       "  (283, 'admit'),\n",
       "  ['         to understanding each other and    admit     the fault each of them                  '])]"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(conc_csv)\n",
    "conc_csv[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['                       them . most of us    admit     that this behavior will lead            '],\n",
       " ['                  their master . when it    admit     someone as a master ,                   '],\n",
       " ['                      us that cat do not    admit     its master . there be                   '],\n",
       " ['                     english , i have to    admit     the new vocabulary word be              '],\n",
       " ['         to understanding each other and    admit     the fault each of them                  ']]"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Or alternatively a list with only the concordance lines\n",
    "conc_simple_csv = [x[2] for x in conc_csv]\n",
    "conc_simple_csv[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "In this notebook, we have accomplished the following:\n",
    "- narrowed PELIC down to just those texts containing the verbs we are interested in\n",
    "- organized these data into three useful formats:\n",
    "    - a dataframe where each row is a text containing the key verbs\n",
    "    - a dataframe where each row is one concordance line\n",
    "    - a csv in a traditional concordance format\n",
    "\n",
    "This data could then be analyzed qualitatively, reading the concordance vertically, looking for patterns. Further quantitative analysis can also be conducted, considering factors like level, L1, gender, etc., or extracting patterns related to these node words. In the case of our original research questions, the next step would be to parse the sentences we have identified, in order to determine whether NPs or CPs follow the verbs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Contact\n",
    "If you have any questions about this tutorial, or more generally about using PELIC, please contact Ben Naismith at bnaismith@pitt.edu. Thank you for visiting.\n",
    "\n",
    "[Back to top](#top)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}