examples/converting_movie_corpus.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Converting the Cornell Movie-Dialogs Corpus into ConvoKit format \n",
    "\n",
    "This notebook is a demonstration of how custom datasets can be converted into Corpus with ConvoKit. See also this example on how to construct a corpus from pandas dataframe: https://github.com/CornellNLP/ConvoKit/blob/master/examples/corpus_from_pandas.ipynb. \n",
    "\n",
    "The original version of the Cornell Movie-Dialogs Corpus can be downloaded from:  https://www.cs.cornell.edu/~cristian/Cornell_Movie-Dialogs_Corpus.html. It contains the following files:\n",
    "\n",
    "* __movie_characters_metadata.txt__ contains information about each movie character\n",
    "* __movie_lines.txt contains__ the actual text of each utterance\n",
    "* __movie_conversations.txt__ contains the structure of the conversations\n",
    "* __movie_titles_metadata.txt__ contains information about each movie title"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Since the GPL-licensed package `unidecode` is not installed, using Python's `unicodedata` package which yields worse results.\n"
     ]
    }
   ],
   "source": [
    "from tqdm import tqdm\n",
    "from convokit import Corpus, Speaker, Utterance\n",
    "from collections import defaultdict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### __Constructing the Corpus from a list of Utterances__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Corpus can be constructed from a list of utterances with:\n",
    "\n",
    "    corpus = Corpus(utterances= custom_utterance_list)\n",
    "    \n",
    "Our goal is to convert the original dataset into this \"custom_utterance_list\", and let ConvoKit will do the rest of the conversion for us. In the context of converting the Movie-Dialogs corpus, we will need the following steps, which will be explained in further detail below:\n",
    "\n",
    "    1. create Speaker objects who are the speakers of the Utterances. Each speaker will correspond to a character in a movie. \n",
    "    2. create the Utterance objects that corresponds to utterances in the movie dialogs  \n",
    "    3. construct the Corpus from the list of Utterance objects \n",
    "    4. incorporate additional information as Conversation/Corpus metadata. \n",
    "\n",
    "We will additionally show how some simple processing can be done. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### __1. Creating speakers__\n",
    "\n",
    "Each character in a movie is considered a speaker, and there are 9,035 characters in total in this dataset. We will read off metadata for each speaker from __movie_characters_metadata.txt__. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# replace the directory with where your downloaded cornell movie dialogs corpus is saved\n",
    "data_dir = \"/cornell-movie-dialogs-corpus/\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(data_dir + \"movie_characters_metadata.txt\", \"r\", encoding='utf-8', errors='ignore') as f:\n",
    "    speaker_data = f.readlines()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In general, we would directly use the name of the speaker as the name. However, in our case, since only the first name of the movie character is given for most characters, these names may not uniquely map to a character. We will instead use speaker_id provided in the original dataset as speakername, whereas the actual charatcter name will be saved in speaker metadata. Note that this also means we are not able to account for characters that show up in a series of moviews (i.e., characters who share the same name and should actually be regarded as the same character). \n",
    "\n",
    "For this dataset, we include the following information for each speaker:  \n",
    "* name of the character.\n",
    "* idx and name of the movie this charater is from\n",
    "* gender(available for 3,774 characters)\n",
    "* position on movie credits (3,321 characters available)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "speaker_meta = {}\n",
    "for speaker in speaker_data:\n",
    "    speaker_info = [info.strip() for info in speaker.split(\"+++$+++\")]\n",
    "    speaker_meta[speaker_info[0]] = {\"character_name\": speaker_info[1],\n",
    "                               \"movie_idx\": speaker_info[2],\n",
    "                               \"movie_name\": speaker_info[3],\n",
    "                               \"gender\": speaker_info[4],\n",
    "                               \"credit_pos\": speaker_info[5]}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In general, a Speaker object can be initiated with `speaker(id = <speaker_name>, meta = <speaker_metadata>)`. The following example shows how we create a Speaker object for each unique character in the dataset, which will be used to create Utterances objects later. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "corpus_speakers = {k: Speaker(id = k, meta = v) for k,v in speaker_meta.items()}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sanity checking use-level data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "number of speakers in the data = 9035\n"
     ]
    }
   ],
   "source": [
    "print(\"number of speakers in the data = {}\".format(len(corpus_speakers)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'character_name': 'BIANCA',\n",
       " 'movie_idx': 'm0',\n",
       " 'movie_name': '10 things i hate about you',\n",
       " 'gender': 'f',\n",
       " 'credit_pos': '4'}"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus_speakers['u0'].meta"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### __2. Creating utterance objects__\n",
    "Utterances can be found in __movie_lines.txt__. There are 304,713 utterances in total. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(data_dir + \"movie_lines.txt\", \"r\", encoding='utf-8', errors='ignore') as f:\n",
    "    utterance_data = f.readlines()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To instantiate an utterance object, we generally need the following information (all ids should be of type string):\n",
    "- id: representing the unique id of the utterance. \n",
    "- speaker: a ConvoKit speaker object representing the speaker giving the utterance.\n",
    "- root: the id of the root utterance of the conversation.\n",
    "- reply_to: id of the utterance this was a reply to.\n",
    "- timestamp: timestamp of the utterance. \n",
    "- text: text of the utterance.\n",
    "\n",
    "Additional information associated with the utterance may be saved as utterance level metadata. In this case, we consider the movie_id from which this utterance is extracted as an example for metadata. \n",
    "\n",
    "An utterance possessing all the above information may be initiated by `Utterance(id=..., speaker =..., conversation_id =..., reply_to=..., timestamp=..., text =..., meta =...)`. We now create such `Utterance` objects for the utterances in our dataset. Note that normally we would provide `conversation_id` and `reply_to` information at the time of instantiation, but we will defer it to later as such information need to be retrieved from a different file. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 304713/304713 [00:04<00:00, 71861.45it/s] "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total number of utterances = 304713\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "utterance_corpus = {}\n",
    "\n",
    "count = 0\n",
    "for utterance in tqdm(utterance_data):\n",
    "    \n",
    "    utterance_info = [info.strip() for info in utterance.split(\"+++$+++\")]\n",
    "    \n",
    "    if len(utterance_info) < 4:\n",
    "        print(utterance_info)\n",
    "        \n",
    "    try:\n",
    "        idx, speaker, movie_id, text = utterance_info[0], utterance_info[1], utterance_info[2], utterance_info[4]\n",
    "    except:\n",
    "        print(utterance_info)\n",
    "    \n",
    "    meta = {'movie_id': movie_id}\n",
    "    \n",
    "    # root & reply_to will be updated later, timestamp is not applicable \n",
    "    utterance_corpus[idx] = Utterance(id=idx, speaker=corpus_speakers[speaker], text=text, meta=meta)\n",
    "\n",
    "print(\"Total number of utterances = {}\".format(len(utterance_corpus)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we check on the status of an Utterance object, it should now contain an id, the speakers who said them, the actual texts, as well as the movie ids as the metadata: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Utterance({'obj_type': 'utterance', 'meta': {'movie_id': 'm0'}, 'vectors': [], 'speaker': Speaker({'obj_type': 'speaker', 'meta': {'character_name': 'CAMERON', 'movie_idx': 'm0', 'movie_name': '10 things i hate about you', 'gender': 'm', 'credit_pos': '3'}, 'vectors': [], 'owner': None, 'id': 'u2'}), 'conversation_id': None, 'reply_to': None, 'timestamp': None, 'text': 'They do to!', 'owner': None, 'id': 'L1044'})"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "utterance_corpus['L1044'] "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### __Updating root and reply_to information to utterances__\n",
    "__movie_conversations.txt__ provides the structure of conversations that organizes the above utterances. This will allow us to add the missing root and reply_to information to individual utterances. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(data_dir + \"movie_conversations.txt\", \"r\", encoding='utf-8', errors='ignore') as f:\n",
    "    convo_data = f.readlines()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "import ast"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 83097/83097 [00:02<00:00, 28279.64it/s]\n"
     ]
    }
   ],
   "source": [
    "for info in tqdm(convo_data):\n",
    "        \n",
    "    speaker1, speaker2, m, convo = [info.strip() for info in info.split(\"+++$+++\")]\n",
    "\n",
    "    convo_seq = ast.literal_eval(convo)\n",
    "    \n",
    "    # update utterance\n",
    "    conversation_id = convo_seq[0]\n",
    "    \n",
    "    # convo_seq is a list of utterances ids, arranged in conversational order\n",
    "    for i, line in enumerate(convo_seq):\n",
    "        \n",
    "        # sanity checking: speaker giving the utterance is indeed in the pair of characters provided\n",
    "        if utterance_corpus[line].speaker.id not in [speaker1, speaker2]:\n",
    "            print(\"speaker mismatch in line {0}\".format(i))\n",
    "        \n",
    "        utterance_corpus[line].conversation_id = conversation_id\n",
    "        \n",
    "        if i == 0:\n",
    "            utterance_corpus[line].reply_to = None\n",
    "        else:\n",
    "            utterance_corpus[line].reply_to = convo_seq[i-1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sanity checking on the status of utterances. After updating root and reply_to information, they should now contain all mandatory fields:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Utterance({'obj_type': 'utterance', 'meta': {'movie_id': 'm616'}, 'vectors': [], 'speaker': Speaker({'obj_type': 'speaker', 'meta': {'character_name': 'COGHILL', 'movie_idx': 'm616', 'movie_name': 'zulu dawn', 'gender': '?', 'credit_pos': '?'}, 'vectors': [], 'owner': None, 'id': 'u9028'}), 'conversation_id': 'L666497', 'reply_to': 'L666498', 'timestamp': None, 'text': 'How quickly can you move your artillery forward?', 'owner': None, 'id': 'L666499'})"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "utterance_corpus['L666499']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### __3. Creating corpus from list of utterances__\n",
    "We are now ready to create the movie-corpus. Recall that to instantiate a `Corpus`, we need a list of `Utterance`s."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "utterance_list = utterance_corpus.values()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Note that by default the version number is incremented \n",
    "movie_corpus = Corpus(utterances=utterance_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "ConvoKit will automatically help us create conversations based on the information about the utterances we provide. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "number of conversations in the dataset = 83097\n"
     ]
    }
   ],
   "source": [
    "print(\"number of conversations in the dataset = {}\".format(len(movie_corpus.get_conversation_ids())))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sample conversation 0:\n",
      "['L1045', 'L1044']\n",
      "sample conversation 1:\n",
      "['L985', 'L984']\n",
      "sample conversation 2:\n",
      "['L925', 'L924']\n",
      "sample conversation 3:\n",
      "['L872', 'L871', 'L870']\n",
      "sample conversation 4:\n",
      "['L869', 'L868', 'L867', 'L866']\n"
     ]
    }
   ],
   "source": [
    "convo_ids = movie_corpus.get_conversation_ids()\n",
    "for i, convo_idx in enumerate(convo_ids[0:5]):\n",
    "    print(\"sample conversation {}:\".format(i))\n",
    "    print(movie_corpus.get_conversation(convo_idx).get_utterance_ids())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### __4. Updating Conversation and Corpus level metadata__\n",
    "\n",
    "For each `Conversation`, we can add contextual information about the movie, including genres, release year to as `Conversation` metadata. To do that,  we will read off such meta data for each movie from __movie_titles_metadata.txt__, and we will attach them to all `Conversation`s taken from the movie. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(data_dir + \"movie_titles_metadata.txt\", \"r\", encoding='utf-8', errors='ignore') as f:\n",
    "    movie_extra = f.readlines()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "movie_meta = defaultdict(dict)\n",
    "\n",
    "for movie in movie_extra:\n",
    "    movie_id, title, year, rating, votes, genre  = [info.strip() for info in movie.split(\"+++$+++\")]\n",
    "    movie_meta[movie_id] = {\"movie_name\": title,\n",
    "                            \"release_year\": year,\n",
    "                            \"rating\": rating,\n",
    "                            \"votes\": votes,\n",
    "                            \"genre\": genre}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For our purpose, the movie_id of a given conversation can be retrieved from the root of the conversation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "for convo in movie_corpus.iter_conversations():\n",
    "    \n",
    "    # get the movie_id for the conversation by checking from utterance info\n",
    "    convo_id = convo.get_id()\n",
    "    movie_idx = movie_corpus.get_utterance(convo_id).meta['movie_id']\n",
    "    \n",
    "    # add movie idx as meta, and update meta with additional movie information\n",
    "    convo.meta['movie_idx'] = movie_idx\n",
    "    convo.meta.update(movie_meta[movie_idx])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we check the `conversation` metadata, it now includes the above-mentioned fields"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'movie_idx': 'm570',\n",
       " 'movie_name': 'three kings',\n",
       " 'release_year': '1999',\n",
       " 'rating': '7.30',\n",
       " 'votes': '69757',\n",
       " 'genre': \"['action', 'adventure', 'comedy', 'drama', 'war']\"}"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movie_corpus.get_conversation(\"L609301\").meta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also include the original urls from which these conversations are extracted as corpus metadata. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(data_dir + \"raw_script_urls.txt\", \"r\", encoding='utf-8', errors='ignore') as f:\n",
    "    urls = f.readlines()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "movie2url = {}\n",
    "for movie in urls:\n",
    "    movie_id, _, url = [info.strip() for info in movie.split(\"+++$+++\")]\n",
    "    movie2url[movie_id] = url"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "movie_corpus.meta['url'] = movie2url"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Optionally, we can also the original name of the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "movie_corpus.meta['name'] = \"Cornell Movie-Dialogs Corpus\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### __5. Processing utterance texts__\n",
    "\n",
    "We can also \"annotate\" the utterances, e.g., getting dependency parses for them, and save the resultant parses. Here is an example of how this can be done, more examples related to text processing can be found at https://github.com/CornellNLP/ConvoKit/blob/master/examples/text-processing/text_preprocessing_demo.ipynb:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "from convokit.text_processing import TextParser"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "parser = TextParser(verbosity=10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10000/304713 utterances processed\n",
      "20000/304713 utterances processed\n",
      "30000/304713 utterances processed\n",
      "40000/304713 utterances processed\n",
      "50000/304713 utterances processed\n",
      "60000/304713 utterances processed\n",
      "70000/304713 utterances processed\n",
      "80000/304713 utterances processed\n",
      "90000/304713 utterances processed\n",
      "100000/304713 utterances processed\n",
      "110000/304713 utterances processed\n",
      "120000/304713 utterances processed\n",
      "130000/304713 utterances processed\n",
      "140000/304713 utterances processed\n",
      "150000/304713 utterances processed\n",
      "160000/304713 utterances processed\n",
      "170000/304713 utterances processed\n",
      "180000/304713 utterances processed\n",
      "190000/304713 utterances processed\n",
      "200000/304713 utterances processed\n",
      "210000/304713 utterances processed\n",
      "220000/304713 utterances processed\n",
      "230000/304713 utterances processed\n",
      "240000/304713 utterances processed\n",
      "250000/304713 utterances processed\n",
      "260000/304713 utterances processed\n",
      "270000/304713 utterances processed\n",
      "280000/304713 utterances processed\n",
      "290000/304713 utterances processed\n",
      "300000/304713 utterances processed\n",
      "304713/304713 utterances processed\n"
     ]
    }
   ],
   "source": [
    "movie_corpus = parser.transform(movie_corpus)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- parses are saved under 'parsed' in utterance meta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'rt': 4,\n",
       "  'toks': [{'tok': 'How', 'tag': 'WRB', 'dep': 'advmod', 'up': 1, 'dn': []},\n",
       "   {'tok': 'quickly', 'tag': 'RB', 'dep': 'advmod', 'up': 4, 'dn': [0]},\n",
       "   {'tok': 'can', 'tag': 'MD', 'dep': 'aux', 'up': 4, 'dn': []},\n",
       "   {'tok': 'you', 'tag': 'PRP', 'dep': 'nsubj', 'up': 4, 'dn': []},\n",
       "   {'tok': 'move', 'tag': 'VB', 'dep': 'ROOT', 'dn': [1, 2, 3, 6, 7, 8]},\n",
       "   {'tok': 'your', 'tag': 'PRP$', 'dep': 'poss', 'up': 6, 'dn': []},\n",
       "   {'tok': 'artillery', 'tag': 'NN', 'dep': 'dobj', 'up': 4, 'dn': [5]},\n",
       "   {'tok': 'forward', 'tag': 'RB', 'dep': 'advmod', 'up': 4, 'dn': []},\n",
       "   {'tok': '?', 'tag': '.', 'dep': 'punct', 'up': 4, 'dn': []}]}]"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movie_corpus.get_utterance('L666499').retrieve_meta('parsed')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### __Saving created datasets__\n",
    "To complete the final step of dataset conversion, we want to save the dataset such that it can be loaded later for reuse. You may want to specify a name. The default location to find the saved datasets will be __./convokit/saved-copora__ in your home directory, but you can also specify where you want the saved corpora to be. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "# movie_corpus.dump(\"movie-corpus\", base_path = <specify where you prefer to save it to>)\n",
    "# the following would save the Corpus to the default location, i.e., ./convokit/saved-corpora\n",
    "movie_corpus.dump(\"movie-corpus\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After saving, the available info from dataset can be checked directly, without loading. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "from convokit import meta_index\n",
    "import os.path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'utterances-index': {'movie_id': [\"<class 'str'>\"],\n",
       "  'parsed': [\"<class 'list'>\"]},\n",
       " 'speakers-index': {'character_name': [\"<class 'str'>\"],\n",
       "  'movie_idx': [\"<class 'str'>\"],\n",
       "  'movie_name': [\"<class 'str'>\"],\n",
       "  'gender': [\"<class 'str'>\"],\n",
       "  'credit_pos': [\"<class 'str'>\"]},\n",
       " 'conversations-index': {'movie_idx': [\"<class 'str'>\"],\n",
       "  'movie_name': [\"<class 'str'>\"],\n",
       "  'release_year': [\"<class 'str'>\"],\n",
       "  'rating': [\"<class 'str'>\"],\n",
       "  'votes': [\"<class 'str'>\"],\n",
       "  'genre': [\"<class 'str'>\"]},\n",
       " 'overall-index': {'url': [\"<class 'dict'>\"], 'name': [\"<class 'str'>\"]},\n",
       " 'version': 1,\n",
       " 'vectors': []}"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "meta_index(filename = os.path.join(os.path.expanduser(\"~\"), \".convokit/saved-corpora/movie-corpus\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### __Other ways of conversion__\n",
    "\n",
    "The above method is only one way to convert the dataset. Alternatively, one may follow strictly with the specifications of the expected data format described [here](https://github.com/CornellNLP/ConvoKit/blob/master/doc/source/data_format.rst) and write out the component files directly. \n",
    "\n",
    "Additional examples of converting datasets originally released in other formats can be found inside the [datasets](https://github.com/CornellNLP/ConvoKit/tree/master/datasets) folder. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  },
  "pycharm": {
   "stem_cell": {
    "cell_type": "raw",
    "metadata": {
     "collapsed": false
    },
    "source": []
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}