Lectures/Week-10-JSONandCSV.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# JSON and CSVs\n",
    "\n",
    "## On boilerplate code\n",
    "\n",
    "We are finally done with learning new syntax and structures in python.  Now we can start using all the patterns we've been learning in the context of real data.  \n",
    "\n",
    "You'll be learning some new packages and have your first real interaction with boilerplate code.\n",
    "\n",
    "Even though you've learned a lot these past few weeks, there are still structures and needs that some tools have that won't really make sense at first interaction.  Some tools are just designed to work in a certain way, and you are provided with documentation and example code that accomplishes them.  This is what we call boilerplate code.  \n",
    "\n",
    "Much like how a syllabus is required to have statements about inclusion and diversity, not all professors are experts in the appropriate language to say for these things.  So we look to those relevant departments for guidance.  They often provide expertly written language for these documents to help ensure standards are met for all students and to avoid a misinterpretation of the policies.\n",
    "\n",
    "These are often things that we read through and they make sense, but we often think \"there's no way I could have written that on my own!\" And that's kind of the point!  That's boilerplate code and that's boilerplate language.  For times when things are so complex and nuanced that the template is just given to you.  As your expertise in that area develops, you are better able to understand the inner workings and the design choices.\n",
    "\n",
    "A lot of code is a bit of boilerplate that you'll need to adapt to fit your needs, but you'll need to leave the rest of it alone.  A bit part of the challenge is that you'll need to be able to look at the boilerplate code and make an educated guess about what parts of the code you need to change and what parts you should leave alone.  The trick is to find the input and the output of that chunk of code.  Where is the data going into it and where is it coming out.\n",
    "\n",
    "I'm going to be giving you some boilerplate code for reading in and writing data of these two formats, but there are also some boilerplate patterns that are just the patterns that we use to accomplish these things. \n",
    "\n",
    "\n",
    "## What are json and csv?\n",
    "\n",
    "### From a data structure perspective\n",
    "\n",
    "These are less two different data types are more like two different data structures.  You can have many kinds of data within them, but they are two different ways of organizing data.\n",
    "\n",
    "Much like having two different ways of organizing a kitchen or closet, each will get the job done so long as long as you know where things are, but you may find that:\n",
    "\n",
    "* One structure makes more sense to you and is your preferred way of organizing the information (remember the method of reading files!).\n",
    "* The types of data and questions that you have are more easily processed in one structure than another.\n",
    "* You can generally accomplish the same things in each structure, but the different organization models will be better equipped to store certain data relationships than the other.\n",
    "\n",
    "### From a format perspective\n",
    "\n",
    "Each of these types of files are stored in a plain text file.  This means that you should be able to open it up and view it in any plain text editor, but you may not be able to really make sense of the structure just by looking at it.  That's because this format wasn't necessarily designed to organize data for human eyes.  The data is formatted in plain text to make it easy for a computer to process those things.  It is text designed for a program to parse.\n",
    "\n",
    "Think of a CSV file that you might open up in excel.  You can see the content in the raw file, but you may really need the rendering of the excel workbook to be able to cleanly scan the data.\n",
    "\n",
    "JSON files are much more like this. They are not designed for human eyes at all and require programmatic processing for everything to get anything done to it. What it has sacrificed in terms of readability, it did nso to gain an exceptional amount of depth of detail in the kind of data content and relationships that it can explore.\n",
    "\n",
    "### From a technical perspective (parsing the data)\n",
    "\n",
    "Each of these files are plain text, so we process them in python much the same way we are used to other files. We read them in as a string or open them up as an infile as normal, and then pass that information off to another part of the program that knows what to do with that kind of data structure.  That code then does the job of translating the data into a structure that we know how to work with in python, and then we can get to business with that resulting structure.\n",
    "\n",
    "This process is often called _parsing of the data_. While we could write our own code to parse the content into a structure we can work with, we usually leave it up to python to deal with for us because we have better things to do with our time. This what modules do for us!\n",
    "\n",
    "#### CSV structures in python\n",
    "\n",
    "CSV documents are rectangular structures that are often best represented as 2D lists. This means that it creates an outer list that holds many inner lists. Each inner list corresponds to a specific row in the data, and the elements within the list are the values for each cell going across that row. Lists are a great way to store this data because the position of the data within the file determines the meaning of it. Lists have a primary access and identification structure around positions. However, the thing that isn't well represented within a 2D list is the idea of the column. You can easily grab an individual row from a 2D list, because that's just one of the inner lists. We can use normal slicing syntax to grab an entire row of data (`list[pos_num]`).\n",
    "\n",
    "However, columns in CSVs aren't all contained in a single data structure. The values that correspond to a column all live across all the inner lists.  Thus, to get the values from an entire column we must first access every inner list and then grab that column position number.  There are special tools (like pandas) that were created to store this kind of 2D array of data in a way that we can access things like columns in a more normal fashion.  From a normal perspective you'd think that accessing a row and a column are about the same, but in the reality of the code, accessing data from a column looks completely different and much more complex.\n",
    "\n",
    "There are tools you can use to read a CSV into an python `dict` structure, although we will not be using that here.\n",
    "\n",
    "#### JSON structures in python\n",
    "\n",
    "json has a completely different structure from a rectangular array of data like a CSV. In fact, json has no specific shape this this that we can call it, other than a tree structure.  Whereas CSVs are very position first and everything is called via that position value, json has a very name or label oriented structure.\n",
    "\n",
    "Operationally, you've been dealing with this kind of structure already with dictionaries in Python.  In fact, when you parse a json file into python, it is saved as either one big dictionary or a list of many dictionaries in there.  There are no rows of data, but instead clusters of trees or one large tree that you can access. Even if those dictionaries are stored within a list, that doesn't mean that the position values for those dictionaries have an actual meaning to the data. You may be able to infer the meaning based on your understanding of how the data file was created, but that won't always hold true.\n",
    "\n",
    "Basically, JSON can be confusing but that's because they're meant for machines, not us. Thankfully, there are a lot of tools and boilerplate to help us!\n",
    "\n",
    "## How to parse these in python\n",
    "\n",
    "You're going to use boilerplate code for each of these to read them in.  There are modules in the Python standard library for parsing json and csv data. We'll be using both of those.\n",
    "\n",
    "Much of the credit for the patterns in use for this lesson are from [The Python Cookbook 3rd edition](http://shop.oreilly.com/product/0636920027072.do)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The data we'll be using\n",
    "\n",
    "Take a moment to look through the `lunaresults.json` file in this directory.  It contains the wikipedia api results for the search term: `luna lovegood`. \n",
    "\n",
    "Remember that an API is an application program interface. Think of these as a gated and controlled means through which companies like wikipedia can control who and how people access their data. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Parsing json with the `json` library\n",
    "\n",
    "Here's the boilerplate code:\n",
    "\n",
    "You'll see that you'll be reading in the json file like a normal text file with `.read()` so it is all one string.  Then passing that string into a special function within the library called `json.loads()` This function will parse a string and return back the relevant python data structure.\n",
    "\n",
    "``` python\n",
    "import json \n",
    "\n",
    "infile = open('yourjsonfile.json', 'r')\n",
    "text = infile.read()\n",
    "infile.close()\n",
    "\n",
    "data = json.loads(text)\n",
    "```\n",
    "\n",
    "Your `data` variable is now a full python object.  json can actulaly have many structures internally, so you may have a dictionary or a list.\n",
    "\n",
    "The first thing you should do is inspect your data variable and learn some things about it.  Only that can tell you what you should be doing with it.  \n",
    "\n",
    "Also, there's now nothing particularly special about this variable.  The data is stored in pure python data structures.  You'll find dictionaries, lists, strings, numbers, etc within it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import json \n",
    "\n",
    "infile = open('lunaresults.json', 'r')\n",
    "text = infile.read()\n",
    "infile.close()\n",
    "\n",
    "data = json.loads(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'dict'>\n"
     ]
    }
   ],
   "source": [
    "print(type(data))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "We can see that this is a dictionary.  What are the keys?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dict_keys(['batchcomplete', 'query'])\n"
     ]
    }
   ],
   "source": [
    "print(data.keys())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look inside the data file and get a sense of the structure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'batchcomplete': '',\n",
       " 'query': {'search': [{'ns': 0,\n",
       "    'pageid': 391371,\n",
       "    'size': 47836,\n",
       "    'snippet': 'Xenophilius <span class=\"searchmatch\">Lovegood</span>, is the editor-in-chief of The Quibbler, a strange magazine claiming supposedly untrue things <span class=\"searchmatch\">Luna</span> and her father believe. When <span class=\"searchmatch\">Luna</span> was',\n",
       "    'timestamp': '2018-10-16T20:06:57Z',\n",
       "    'title': \"Dumbledore's Army\",\n",
       "    'wordcount': 4967},\n",
       "   {'ns': 0,\n",
       "    'pageid': 3931304,\n",
       "    'size': 39384,\n",
       "    'snippet': 'podcast host and narrator. She rose to prominence for her portrayal of <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> in the Harry Potter film series, appearing in the last four films and',\n",
       "    'timestamp': '2018-10-10T09:09:37Z',\n",
       "    'title': 'Evanna Lynch',\n",
       "    'wordcount': 3060},\n",
       "   {'ns': 0,\n",
       "    'pageid': 874930,\n",
       "    'size': 86965,\n",
       "    'snippet': 'semi-humans, greatly attractive Wood Nymphs <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span>, a pupil at Hogwarts, and her father Xenophilius <span class=\"searchmatch\">Lovegood</span> profess belief in a number of magical creatures',\n",
       "    'timestamp': '2018-10-24T12:21:36Z',\n",
       "    'title': 'Magical creatures in Harry Potter',\n",
       "    'wordcount': 12101},\n",
       "   {'ns': 0,\n",
       "    'pageid': 23483380,\n",
       "    'size': 5111,\n",
       "    'snippet': 'Shrek Super Slam. <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> in the Harry Potter series <span class=\"searchmatch\">Luna</span> Maximoff, supporting character in the Marvel Comics universe <span class=\"searchmatch\">Luna</span> Mothews, the teenage',\n",
       "    'timestamp': '2018-10-18T19:04:14Z',\n",
       "    'title': 'Luna (name)',\n",
       "    'wordcount': 527},\n",
       "   {'ns': 0,\n",
       "    'pageid': 38290297,\n",
       "    'size': 18775,\n",
       "    'snippet': 'the company to reprise his role as Harry Potter, and Evanna Lynch as <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span>, who played the character in the original film series. The script and',\n",
       "    'timestamp': '2018-06-30T10:59:29Z',\n",
       "    'title': 'A Very Potter Senior Year',\n",
       "    'wordcount': 2018},\n",
       "   {'ns': 0,\n",
       "    'pageid': 891112,\n",
       "    'size': 19444,\n",
       "    'snippet': '(2010). He played Xenophillius <span class=\"searchmatch\">Lovegood</span>, editor of the wizarding magazine The Quibbler and father of the eccentric <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span>. In the same interview, he',\n",
       "    'timestamp': '2018-09-12T07:15:17Z',\n",
       "    'title': 'Rhys Ifans',\n",
       "    'wordcount': 1441},\n",
       "   {'ns': 0,\n",
       "    'pageid': 11061022,\n",
       "    'size': 65170,\n",
       "    'snippet': 'the mini-serial Proof. During the same time, Ronan auditioned to play <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> in Harry Potter and the Order of the Phoenix, a role she eventually',\n",
       "    'timestamp': '2018-10-24T12:23:42Z',\n",
       "    'title': 'Saoirse Ronan',\n",
       "    'wordcount': 4796},\n",
       "   {'ns': 0,\n",
       "    'pageid': 3512033,\n",
       "    'size': 89484,\n",
       "    'snippet': 'escorting Neville to the Hogwarts Express. Xenophilius <span class=\"searchmatch\">Lovegood</span> is Pandora <span class=\"searchmatch\">Lovegood</span>\\'s husband, <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span>\\'s father, and the editor of The Quibbler, a magazine',\n",
       "    'timestamp': '2018-10-22T17:36:29Z',\n",
       "    'title': 'List of supporting Harry Potter characters',\n",
       "    'wordcount': 12755},\n",
       "   {'ns': 0,\n",
       "    'pageid': 670407,\n",
       "    'size': 94795,\n",
       "    'snippet': 'she attacks and insults them. Harry, Hermione, Ron, Ginny Weasley, <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span>, and Neville Longbottom fly to the Ministry of Magic on Thestrals in',\n",
       "    'timestamp': '2018-10-25T09:55:53Z',\n",
       "    'title': 'Harry Potter and the Order of the Phoenix (film)',\n",
       "    'wordcount': 9674},\n",
       "   {'ns': 0,\n",
       "    'pageid': 45100,\n",
       "    'size': 48610,\n",
       "    'snippet': 'that Hermione is an only child. Rowling has described the character of <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> as the &quot;anti-Hermione&quot; as they are so different. Hermione\\'s nemesis',\n",
       "    'timestamp': '2018-10-20T15:44:28Z',\n",
       "    'title': 'Hermione Granger',\n",
       "    'wordcount': 5898},\n",
       "   {'ns': 0,\n",
       "    'pageid': 667601,\n",
       "    'size': 29845,\n",
       "    'snippet': 'Dumbledore\\'s Army. <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> – Ravenclaw student one year below Harry, a member of Dumbledore\\'s Army. Xenophilius <span class=\"searchmatch\">Lovegood</span> – Father of <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span>, and editor',\n",
       "    'timestamp': '2018-10-23T05:01:51Z',\n",
       "    'title': 'List of Harry Potter characters',\n",
       "    'wordcount': 3585},\n",
       "   {'ns': 0,\n",
       "    'pageid': 827110,\n",
       "    'size': 6141,\n",
       "    'snippet': 'Hallows, a character named Xenophilius <span class=\"searchmatch\">Lovegood</span> (the father of one of Harry Potter\\'s more eccentric friends, <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span>) is characterized by his interest',\n",
       "    'timestamp': '2018-06-23T13:19:23Z',\n",
       "    'title': 'Xenophilia',\n",
       "    'wordcount': 848},\n",
       "   {'ns': 0,\n",
       "    'pageid': 10093585,\n",
       "    'size': 17042,\n",
       "    'snippet': 'Quincey in Atonement has been called &quot;impressive&quot;. She auditioned to play <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> in Harry Potter and the Order of the Phoenix, though the role ultimately',\n",
       "    'timestamp': '2018-10-03T06:52:19Z',\n",
       "    'title': 'Juno Temple',\n",
       "    'wordcount': 1108},\n",
       "   {'ns': 0,\n",
       "    'pageid': 2298654,\n",
       "    'size': 8444,\n",
       "    'snippet': 'Bonnie Wright (Ginny Weasley), Katie Leung (Cho Chang), Evanna Lynch (<span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span>), and Scarlett Byrne (Pansy Parkinson). Azad and Shefali Chowdhury (who',\n",
       "    'timestamp': '2018-09-20T08:31:42Z',\n",
       "    'title': 'Afshan Azad',\n",
       "    'wordcount': 597},\n",
       "   {'ns': 0,\n",
       "    'pageid': 241855,\n",
       "    'size': 15333,\n",
       "    'snippet': 'Hoppy, Milly and Mauler. He had a grandson named Rolf, who married <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> (a character who appeared in the last three Harry Potter novels) some',\n",
       "    'timestamp': '2018-10-20T22:08:41Z',\n",
       "    'title': 'Fantastic Beasts and Where to Find Them',\n",
       "    'wordcount': 1704},\n",
       "   {'ns': 0,\n",
       "    'pageid': 9834441,\n",
       "    'size': 75188,\n",
       "    'snippet': 'the trio awkwardly attempt to leave but are stopped by <span class=\"searchmatch\">Lovegood</span>. He reveals that <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> has been kidnapped before summoning the Death Eaters, intending',\n",
       "    'timestamp': '2018-10-23T17:40:58Z',\n",
       "    'title': 'Harry Potter and the Deathly Hallows – Part 1',\n",
       "    'wordcount': 6698},\n",
       "   {'ns': 0,\n",
       "    'pageid': 48543224,\n",
       "    'size': 36067,\n",
       "    'snippet': 'the striking similarities between Emily and her former character of <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> in the Harry Potter films: There are a lot of similarities, I think',\n",
       "    'timestamp': '2018-08-01T10:48:49Z',\n",
       "    'title': 'My Name Is Emily',\n",
       "    'wordcount': 4655},\n",
       "   {'ns': 0,\n",
       "    'pageid': 771537,\n",
       "    'size': 59133,\n",
       "    'snippet': 'Laughland Lee Jordan Luke Youngblood \\xa0 Neville Longbottom Matthew Lewis <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> \\xa0 Evanna Lynch Ernie Macmillan \\xa0 Louis Doyle \\xa0 Louis Doyle \\xa0 Draco Malfoy',\n",
       "    'timestamp': '2018-10-02T10:52:47Z',\n",
       "    'title': 'List of Harry Potter cast members',\n",
       "    'wordcount': 1246},\n",
       "   {'ns': 0,\n",
       "    'pageid': 271050,\n",
       "    'size': 10253,\n",
       "    'snippet': 'George Hotz, hacker and founder of comma.ai. Evanna Lynch, who portrays <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> in the Harry Potter movies, attended the Irish Centre For Talented Youth',\n",
       "    'timestamp': '2018-07-30T17:39:03Z',\n",
       "    'title': 'Center for Talented Youth',\n",
       "    'wordcount': 589},\n",
       "   {'ns': 0,\n",
       "    'pageid': 855389,\n",
       "    'size': 62366,\n",
       "    'snippet': '(Discworld) Alice Longbottom (Harry Potter) Augusta Longbottom (Harry Potter) <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> (Harry Potter) Brenda Loveknot (Discworld) Lythande (Thieves\\' World',\n",
       "    'timestamp': '2018-10-16T00:47:44Z',\n",
       "    'title': 'List of fictional witches',\n",
       "    'wordcount': 5662},\n",
       "   {'ns': 0,\n",
       "    'pageid': 32402649,\n",
       "    'size': 31962,\n",
       "    'snippet': 'his love for Hermione, but she shoots him down. However, Draco meets <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span>, and the two bond over Pigfarts, the wizarding school on Mars. In Dumbledore\\'s',\n",
       "    'timestamp': '2018-09-29T07:35:20Z',\n",
       "    'title': 'A Very Potter Sequel',\n",
       "    'wordcount': 3539},\n",
       "   {'ns': 0,\n",
       "    'pageid': 3833378,\n",
       "    'size': 8540,\n",
       "    'snippet': 'resident. Evanna Lynch, actress, starred in the Harry Potter films as <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> Arthur Mathews, co-writer of sitcom Father Ted, was a resident. Peter',\n",
       "    'timestamp': '2018-09-27T15:22:48Z',\n",
       "    'title': 'Termonfeckin',\n",
       "    'wordcount': 909},\n",
       "   {'ns': 0,\n",
       "    'pageid': 5032848,\n",
       "    'size': 20087,\n",
       "    'snippet': 'These sets have newly decorated mini-figures and animals including <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span>, Bellatrix Lestrange, new owls and new pigs. Also announced was a Lego',\n",
       "    'timestamp': '2018-10-07T14:36:24Z',\n",
       "    'title': 'Lego Harry Potter',\n",
       "    'wordcount': 1092},\n",
       "   {'ns': 0,\n",
       "    'pageid': 34085574,\n",
       "    'size': 4611,\n",
       "    'snippet': 'Potter and the Order of the Phoenix (<span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> (Evanna Lynch)) Harry Potter and the Half-Blood Prince (<span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> (Evanna Lynch)) Horrid Henry: The',\n",
       "    'timestamp': '2018-08-13T14:21:43Z',\n",
       "    'title': 'Shira Naor',\n",
       "    'wordcount': 443},\n",
       "   {'ns': 0,\n",
       "    'pageid': 44334,\n",
       "    'size': 34448,\n",
       "    'snippet': 'even more. Harry is left stranded on the train until Nymphadora Tonks (<span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> in the film adaptation) rescues him. Harry spends much of the year trailing',\n",
       "    'timestamp': '2018-09-22T05:46:20Z',\n",
       "    'title': 'Draco Malfoy',\n",
       "    'wordcount': 4395},\n",
       "   {'ns': 0,\n",
       "    'pageid': 21419143,\n",
       "    'size': 16180,\n",
       "    'snippet': 'goblin Griphook\\'s legs are broken in the escape from Malfoy Manor. <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> uses Skele-Gro to repair them. Sleekeazy\\'s Hair Potion is a beauty treatment',\n",
       "    'timestamp': '2018-10-23T18:33:17Z',\n",
       "    'title': 'Potions in Harry Potter',\n",
       "    'wordcount': 2101},\n",
       "   {'ns': 0,\n",
       "    'pageid': 19156567,\n",
       "    'size': 41338,\n",
       "    'snippet': 'Death Eaters alongside Harry, Hermione, Ginny, Neville Longbottom and <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> at the Department of Mysteries. He is injured in the fight, but makes',\n",
       "    'timestamp': '2018-10-07T16:35:30Z',\n",
       "    'title': 'Ron Weasley',\n",
       "    'wordcount': 4990},\n",
       "   {'ns': 0,\n",
       "    'pageid': 906492,\n",
       "    'size': 76774,\n",
       "    'snippet': 'Matthew Lewis (the actor who portrays Neville Longbottom), Evanna Lynch (<span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span>), Jamie Waylett (Vincent Crabbe), Rupert Grint (Ron Weasley), Chris',\n",
       "    'timestamp': '2018-10-16T04:54:32Z',\n",
       "    'title': 'Harry Potter fandom',\n",
       "    'wordcount': 8107},\n",
       "   {'ns': 0,\n",
       "    'pageid': 59572,\n",
       "    'size': 19811,\n",
       "    'snippet': '&quot;fiendish glee&quot; writing this scene, which features memorable commentary by <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span>. In 2014 Rowling started publishing a series of match reports from the',\n",
       "    'timestamp': '2018-09-26T22:29:29Z',\n",
       "    'title': 'Quidditch',\n",
       "    'wordcount': 2505},\n",
       "   {'ns': 0,\n",
       "    'pageid': 156489,\n",
       "    'size': 29708,\n",
       "    'snippet': 'including Neville Longbottom, Fred and George Weasley, Ginny Weasley and <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span>. Struggling to find a place to practise, Dobby the house elf tells him',\n",
       "    'timestamp': '2018-10-21T03:34:28Z',\n",
       "    'title': 'Harry Potter and the Order of the Phoenix',\n",
       "    'wordcount': 3268},\n",
       "   {'ns': 0,\n",
       "    'pageid': 156587,\n",
       "    'size': 61973,\n",
       "    'snippet': 'on Voldemort\\'s orders, preparing to capture Harry, but is stunned by <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> after touching her Dark Mark to summon Voldemort. Amycus, after seeing',\n",
       "    'timestamp': '2018-10-13T05:30:42Z',\n",
       "    'title': 'Death Eater',\n",
       "    'wordcount': 7104},\n",
       "   {'ns': 0,\n",
       "    'pageid': 33976749,\n",
       "    'size': 15741,\n",
       "    'snippet': '7, 2016. The announcement featured actress Evanna Lynch, who played <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> in the Harry Potter films. However, the soft opening was on March 31',\n",
       "    'timestamp': '2018-10-02T11:15:44Z',\n",
       "    'title': 'The Wizarding World of Harry Potter (Universal Studios Hollywood)',\n",
       "    'wordcount': 1264},\n",
       "   {'ns': 0,\n",
       "    'pageid': 776049,\n",
       "    'size': 93552,\n",
       "    'snippet': 'help of Dobby the house-elf they escape along with fellow prisoners <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span>, Mr. Ollivander, Dean Thomas, and the goblin Griphook. During the escape',\n",
       "    'timestamp': '2018-10-18T23:38:23Z',\n",
       "    'title': 'Harry Potter and the Deathly Hallows',\n",
       "    'wordcount': 9661},\n",
       "   {'ns': 0,\n",
       "    'pageid': 667658,\n",
       "    'size': 81208,\n",
       "    'snippet': 'prisoners are being kept in the basement on Voldemort\\'s orders, including <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span>, Dean Thomas, Griphook the Goblin, and Mr. Ollivander. When Snatchers',\n",
       "    'timestamp': '2018-10-21T03:12:48Z',\n",
       "    'title': 'Places in Harry Potter',\n",
       "    'wordcount': 10751},\n",
       "   {'ns': 0,\n",
       "    'pageid': 858575,\n",
       "    'size': 98258,\n",
       "    'snippet': 'but is spotted and petrified by Malfoy. Harry is found and saved by <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span>. At Hogwarts, Harry and Ron borrow textbooks for Slughorn\\'s Potions',\n",
       "    'timestamp': '2018-10-21T21:19:48Z',\n",
       "    'title': 'Harry Potter and the Half-Blood Prince (film)',\n",
       "    'wordcount': 9182},\n",
       "   {'ns': 0,\n",
       "    'pageid': 284598,\n",
       "    'size': 68965,\n",
       "    'snippet': 'for Magic&quot;. Pottermore. Retrieved 5 May 2017. Rowling, J.\\xa0K. (2003). &quot;<span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span>&quot;. Harry Potter and the Order of the Phoenix. Bloomsbury. ISBN\\xa00747551006',\n",
       "    'timestamp': '2018-10-19T08:13:25Z',\n",
       "    'title': 'Ministry of Magic',\n",
       "    'wordcount': 8228},\n",
       "   {'ns': 0,\n",
       "    'pageid': 17763993,\n",
       "    'size': 6306,\n",
       "    'snippet': 'Retrieved 7 January 2011.\\xa0   Andy (7 October 2010). &quot;Evanna Lynch talks <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> on the set of \\'Deathly Hallows\\'&quot;. Harry Potter Fan Zone. Retrieved 7',\n",
       "    'timestamp': '2018-09-15T04:43:24Z',\n",
       "    'title': 'Harry Potter Fan Zone',\n",
       "    'wordcount': 501},\n",
       "   {'ns': 0,\n",
       "    'pageid': 28373862,\n",
       "    'size': 37822,\n",
       "    'snippet': 'her teammates along the way. Dawn is mostly based on the character, <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> from the Harry Potter series. Voiced by Caitlynne Medrek. Jo, known',\n",
       "    'timestamp': '2018-10-10T15:29:59Z',\n",
       "    'title': 'Total Drama: Revenge of the Island',\n",
       "    'wordcount': 4352},\n",
       "   {'ns': 0,\n",
       "    'pageid': 54465045,\n",
       "    'size': 201532,\n",
       "    'snippet': 'Archived from the original on 10 July 2017. Retrieved 6 July 2017. &quot;<span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> role has been cast&quot;. CBBC Newsround. 2 February 2006. Archived from',\n",
       "    'timestamp': '2018-10-21T21:05:26Z',\n",
       "    'title': 'Wizarding World',\n",
       "    'wordcount': 11476},\n",
       "   {'ns': 0,\n",
       "    'pageid': 31941988,\n",
       "    'size': 89883,\n",
       "    'snippet': 'to a duel. McGonagall gathers the Hogwarts community for battle. At <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span>\\'s insistence, Harry speaks to Helena Ravenclaw\\'s ghost, who reveals',\n",
       "    'timestamp': '2018-10-23T13:09:36Z',\n",
       "    'title': 'Harry Potter and the Deathly Hallows – Part 2',\n",
       "    'wordcount': 7804},\n",
       "   {'ns': 0,\n",
       "    'pageid': 779883,\n",
       "    'size': 124700,\n",
       "    'snippet': 'Hallows, Neville Longbottom uses the DA\\'s coins to alert people such as <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> and Ginny Weasley that Harry, Ron and Hermione have returned to Hogwarts',\n",
       "    'timestamp': '2018-10-02T19:41:43Z',\n",
       "    'title': 'Magical objects in Harry Potter',\n",
       "    'wordcount': 18571},\n",
       "   {'ns': 0,\n",
       "    'pageid': 27511982,\n",
       "    'size': 52160,\n",
       "    'snippet': 'Weasley Wand Scabbers  4  Draco Malfoy Wand Golden Snitch Broomstick  5  <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> Wand Quibbler 2x3  6  Neville Longbottom Wand Mandrake Cauldron  7 ',\n",
       "    'timestamp': '2018-08-25T15:23:22Z',\n",
       "    'title': 'Lego Minifigures (theme)',\n",
       "    'wordcount': 1575},\n",
       "   {'ns': 0,\n",
       "    'pageid': 32446375,\n",
       "    'size': 4465,\n",
       "    'snippet': 'Chance Max in The Suite Life of Zack &amp; Cody Bridgette Dubois in Medium <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> in Harry Potter Violet Beauregarde in Charlie and the Chocolate Factory',\n",
       "    'timestamp': '2018-08-10T04:08:47Z',\n",
       "    'title': 'Kitty Kántor',\n",
       "    'wordcount': 338},\n",
       "   {'ns': 0,\n",
       "    'pageid': 12976982,\n",
       "    'size': 83013,\n",
       "    'snippet': 'Jaquish wrote two additional scenes which added Peeves, Moaning Myrtle, <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> and the Hogwarts Kitchen House Elves. The show first played at the Firehouse',\n",
       "    'timestamp': '2018-10-03T15:42:45Z',\n",
       "    'title': 'Parodies of Harry Potter',\n",
       "    'wordcount': 10133},\n",
       "   {'ns': 0,\n",
       "    'pageid': 3381404,\n",
       "    'size': 11798,\n",
       "    'snippet': '- Actress who plays Ginny Weasley. Evanna Lynch - Actress who plays <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span>. (episode 69) Chris Rankin - Actor who plays Percy Weasley. Crew members',\n",
       "    'timestamp': '2018-10-14T09:01:44Z',\n",
       "    'title': 'PotterCast',\n",
       "    'wordcount': 1474},\n",
       "   {'ns': 0,\n",
       "    'pageid': 12294600,\n",
       "    'size': 12016,\n",
       "    'snippet': '&quot;Stick it to Dolores&quot;2:3210.&quot;S.P.E.W.&quot;1:0111.&quot;The Human Hosepipe&quot;3:3312.&quot;<span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> is OK&quot;2:2013.&quot;The Godfather: Part II&quot;2:2514.&quot;The Weapon&quot;4:27Total length:41:58',\n",
       "    'timestamp': '2018-07-16T03:35:48Z',\n",
       "    'title': \"Voldemort Can't Stop the Rock!\",\n",
       "    'wordcount': 1084},\n",
       "   {'ns': 0,\n",
       "    'pageid': 24870495,\n",
       "    'size': 15929,\n",
       "    'snippet': 'Silk Spectre Evanna Lynch, Harry Potter and the Half-Blood Prince as <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> Shirley Manson, Terminator: The Sarah Connor Chronicles as Catherine',\n",
       "    'timestamp': '2018-03-05T11:16:37Z',\n",
       "    'title': '2009 Scream Awards',\n",
       "    'wordcount': 1352},\n",
       "   {'ns': 0,\n",
       "    'pageid': 20997070,\n",
       "    'size': 7748,\n",
       "    'snippet': 'Chile Evanna Lynch (born 1991), Irish actress, known for her role as <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> in the Harry Potter films Fionán Lynch, Irish politician Genevieve Springston',\n",
       "    'timestamp': '2018-09-13T01:10:46Z',\n",
       "    'title': 'List of people with surname Lynch',\n",
       "    'wordcount': 936},\n",
       "   {'ns': 0,\n",
       "    'pageid': 29494722,\n",
       "    'size': 9907,\n",
       "    'snippet': 'Cho Chang Charlie Weasley (uncredited) Arielle Goldman Lily Potter <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> Hedwig Fred Weasley Brian Rosenthal Seamus Finnigan James Potter Past',\n",
       "    'timestamp': '2018-09-29T07:35:24Z',\n",
       "    'title': 'A Very Potter Sequel (EP)',\n",
       "    'wordcount': 326},\n",
       "   {'ns': 0,\n",
       "    'pageid': 29960484,\n",
       "    'size': 6402,\n",
       "    'snippet': 'au/fashion/wrap/vraw_livestyle2010 https://bowienews.wordpress.com/2012/06/04/from-<span class=\"searchmatch\">luna</span>-<span class=\"searchmatch\">lovegood</span>-to-looking-lovely-in-bowie/ http://www.stylebistro',\n",
       "    'timestamp': '2018-10-06T15:03:49Z',\n",
       "    'title': 'Bowie Wong',\n",
       "    'wordcount': 768},\n",
       "   {'ns': 0,\n",
       "    'pageid': 45256977,\n",
       "    'size': 10437,\n",
       "    'snippet': 'Eleanor Morton (Ginny Weasley/Molly Weasley); Matt Tedford (Ron Weasley/<span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span>); Marny Godden (Hermione Granger/Uncle Albert); Jon Brittain (Hagrid);',\n",
       "    'timestamp': '2018-05-24T12:08:53Z',\n",
       "    'title': 'Weirdos Comedy Club',\n",
       "    'wordcount': 1405},\n",
       "   {'ns': 0,\n",
       "    'pageid': 43136825,\n",
       "    'size': 37894,\n",
       "    'snippet': 'comic strip character Lone Ranger (2009) character from The Lone Ranger <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> (2013) character from Harry Potter Lyra (2005) harpist or the harp from',\n",
       "    'timestamp': '2018-01-22T12:31:15Z',\n",
       "    'title': 'List of fictional characters on stamps of the United States',\n",
       "    'wordcount': 2818},\n",
       "   {'ns': 0,\n",
       "    'pageid': 947583,\n",
       "    'size': 72122,\n",
       "    'snippet': 'noted for her cleverness and creativity, and was described by Xenophilius <span class=\"searchmatch\">Lovegood</span> as beautiful. The Sorting Hat introduces her as &quot;Fair Ravenclaw, from glen&quot;',\n",
       "    'timestamp': '2018-10-24T10:40:37Z',\n",
       "    'title': 'Hogwarts staff',\n",
       "    'wordcount': 8384},\n",
       "   {'ns': 0,\n",
       "    'pageid': 29494707,\n",
       "    'size': 14359,\n",
       "    'snippet': 'Cho Chang Charlie Weasley (uncredited) Arielle Goldman Lily Potter <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> Hedwig Fred Weasley Brian Rosenthal Seamus Finnigan James Potter Past',\n",
       "    'timestamp': '2018-09-29T07:36:43Z',\n",
       "    'title': 'A Very StarKid Album',\n",
       "    'wordcount': 572},\n",
       "   {'ns': 0,\n",
       "    'pageid': 2228434,\n",
       "    'size': 68305,\n",
       "    'snippet': 'the Potters) or Tinworth in Cornwall. The all-wizard Weasley, Diggory, <span class=\"searchmatch\">Lovegood</span>, and Fawcett families live near the Muggle village of Ottery St Catchpole',\n",
       "    'timestamp': '2018-09-10T16:49:22Z',\n",
       "    'title': 'Fictional universe of Harry Potter',\n",
       "    'wordcount': 9196}],\n",
       "  'searchinfo': {'totalhits': 55}}}"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that there's a bunch of data in there, but nested within some key structures.\n",
    "\n",
    "Now you can see what is at the heart of json.  Storing things like a dictionary key/value pair structure means that you can store multiple data points and nest even more complex data structures within your data.\n",
    "\n",
    "Let's look at the query key, because that's where our data lives.\n",
    "\n",
    "This type of structure is completely normal to work with when it comes to apis.  Many times you are served up many data records all in one pile.  Being stored in json structures means that you can store a full and independent data record within a larger structure.\n",
    "\n",
    "We need to start drilling down into query to get our data out.  Remember that this is a dictionary we're working with, so we need to use the dictionary style syntax to get things out of it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'search': [{'ns': 0,\n",
       "   'pageid': 391371,\n",
       "   'size': 47836,\n",
       "   'snippet': 'Xenophilius <span class=\"searchmatch\">Lovegood</span>, is the editor-in-chief of The Quibbler, a strange magazine claiming supposedly untrue things <span class=\"searchmatch\">Luna</span> and her father believe. When <span class=\"searchmatch\">Luna</span> was',\n",
       "   'timestamp': '2018-10-16T20:06:57Z',\n",
       "   'title': \"Dumbledore's Army\",\n",
       "   'wordcount': 4967},\n",
       "  {'ns': 0,\n",
       "   'pageid': 3931304,\n",
       "   'size': 39384,\n",
       "   'snippet': 'podcast host and narrator. She rose to prominence for her portrayal of <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> in the Harry Potter film series, appearing in the last four films and',\n",
       "   'timestamp': '2018-10-10T09:09:37Z',\n",
       "   'title': 'Evanna Lynch',\n",
       "   'wordcount': 3060},\n",
       "  {'ns': 0,\n",
       "   'pageid': 874930,\n",
       "   'size': 86965,\n",
       "   'snippet': 'semi-humans, greatly attractive Wood Nymphs <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span>, a pupil at Hogwarts, and her father Xenophilius <span class=\"searchmatch\">Lovegood</span> profess belief in a number of magical creatures',\n",
       "   'timestamp': '2018-10-24T12:21:36Z',\n",
       "   'title': 'Magical creatures in Harry Potter',\n",
       "   'wordcount': 12101},\n",
       "  {'ns': 0,\n",
       "   'pageid': 23483380,\n",
       "   'size': 5111,\n",
       "   'snippet': 'Shrek Super Slam. <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> in the Harry Potter series <span class=\"searchmatch\">Luna</span> Maximoff, supporting character in the Marvel Comics universe <span class=\"searchmatch\">Luna</span> Mothews, the teenage',\n",
       "   'timestamp': '2018-10-18T19:04:14Z',\n",
       "   'title': 'Luna (name)',\n",
       "   'wordcount': 527},\n",
       "  {'ns': 0,\n",
       "   'pageid': 38290297,\n",
       "   'size': 18775,\n",
       "   'snippet': 'the company to reprise his role as Harry Potter, and Evanna Lynch as <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span>, who played the character in the original film series. The script and',\n",
       "   'timestamp': '2018-06-30T10:59:29Z',\n",
       "   'title': 'A Very Potter Senior Year',\n",
       "   'wordcount': 2018},\n",
       "  {'ns': 0,\n",
       "   'pageid': 891112,\n",
       "   'size': 19444,\n",
       "   'snippet': '(2010). He played Xenophillius <span class=\"searchmatch\">Lovegood</span>, editor of the wizarding magazine The Quibbler and father of the eccentric <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span>. In the same interview, he',\n",
       "   'timestamp': '2018-09-12T07:15:17Z',\n",
       "   'title': 'Rhys Ifans',\n",
       "   'wordcount': 1441},\n",
       "  {'ns': 0,\n",
       "   'pageid': 11061022,\n",
       "   'size': 65170,\n",
       "   'snippet': 'the mini-serial Proof. During the same time, Ronan auditioned to play <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> in Harry Potter and the Order of the Phoenix, a role she eventually',\n",
       "   'timestamp': '2018-10-24T12:23:42Z',\n",
       "   'title': 'Saoirse Ronan',\n",
       "   'wordcount': 4796},\n",
       "  {'ns': 0,\n",
       "   'pageid': 3512033,\n",
       "   'size': 89484,\n",
       "   'snippet': 'escorting Neville to the Hogwarts Express. Xenophilius <span class=\"searchmatch\">Lovegood</span> is Pandora <span class=\"searchmatch\">Lovegood</span>\\'s husband, <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span>\\'s father, and the editor of The Quibbler, a magazine',\n",
       "   'timestamp': '2018-10-22T17:36:29Z',\n",
       "   'title': 'List of supporting Harry Potter characters',\n",
       "   'wordcount': 12755},\n",
       "  {'ns': 0,\n",
       "   'pageid': 670407,\n",
       "   'size': 94795,\n",
       "   'snippet': 'she attacks and insults them. Harry, Hermione, Ron, Ginny Weasley, <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span>, and Neville Longbottom fly to the Ministry of Magic on Thestrals in',\n",
       "   'timestamp': '2018-10-25T09:55:53Z',\n",
       "   'title': 'Harry Potter and the Order of the Phoenix (film)',\n",
       "   'wordcount': 9674},\n",
       "  {'ns': 0,\n",
       "   'pageid': 45100,\n",
       "   'size': 48610,\n",
       "   'snippet': 'that Hermione is an only child. Rowling has described the character of <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> as the &quot;anti-Hermione&quot; as they are so different. Hermione\\'s nemesis',\n",
       "   'timestamp': '2018-10-20T15:44:28Z',\n",
       "   'title': 'Hermione Granger',\n",
       "   'wordcount': 5898},\n",
       "  {'ns': 0,\n",
       "   'pageid': 667601,\n",
       "   'size': 29845,\n",
       "   'snippet': 'Dumbledore\\'s Army. <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> – Ravenclaw student one year below Harry, a member of Dumbledore\\'s Army. Xenophilius <span class=\"searchmatch\">Lovegood</span> – Father of <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span>, and editor',\n",
       "   'timestamp': '2018-10-23T05:01:51Z',\n",
       "   'title': 'List of Harry Potter characters',\n",
       "   'wordcount': 3585},\n",
       "  {'ns': 0,\n",
       "   'pageid': 827110,\n",
       "   'size': 6141,\n",
       "   'snippet': 'Hallows, a character named Xenophilius <span class=\"searchmatch\">Lovegood</span> (the father of one of Harry Potter\\'s more eccentric friends, <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span>) is characterized by his interest',\n",
       "   'timestamp': '2018-06-23T13:19:23Z',\n",
       "   'title': 'Xenophilia',\n",
       "   'wordcount': 848},\n",
       "  {'ns': 0,\n",
       "   'pageid': 10093585,\n",
       "   'size': 17042,\n",
       "   'snippet': 'Quincey in Atonement has been called &quot;impressive&quot;. She auditioned to play <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> in Harry Potter and the Order of the Phoenix, though the role ultimately',\n",
       "   'timestamp': '2018-10-03T06:52:19Z',\n",
       "   'title': 'Juno Temple',\n",
       "   'wordcount': 1108},\n",
       "  {'ns': 0,\n",
       "   'pageid': 2298654,\n",
       "   'size': 8444,\n",
       "   'snippet': 'Bonnie Wright (Ginny Weasley), Katie Leung (Cho Chang), Evanna Lynch (<span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span>), and Scarlett Byrne (Pansy Parkinson). Azad and Shefali Chowdhury (who',\n",
       "   'timestamp': '2018-09-20T08:31:42Z',\n",
       "   'title': 'Afshan Azad',\n",
       "   'wordcount': 597},\n",
       "  {'ns': 0,\n",
       "   'pageid': 241855,\n",
       "   'size': 15333,\n",
       "   'snippet': 'Hoppy, Milly and Mauler. He had a grandson named Rolf, who married <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> (a character who appeared in the last three Harry Potter novels) some',\n",
       "   'timestamp': '2018-10-20T22:08:41Z',\n",
       "   'title': 'Fantastic Beasts and Where to Find Them',\n",
       "   'wordcount': 1704},\n",
       "  {'ns': 0,\n",
       "   'pageid': 9834441,\n",
       "   'size': 75188,\n",
       "   'snippet': 'the trio awkwardly attempt to leave but are stopped by <span class=\"searchmatch\">Lovegood</span>. He reveals that <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> has been kidnapped before summoning the Death Eaters, intending',\n",
       "   'timestamp': '2018-10-23T17:40:58Z',\n",
       "   'title': 'Harry Potter and the Deathly Hallows – Part 1',\n",
       "   'wordcount': 6698},\n",
       "  {'ns': 0,\n",
       "   'pageid': 48543224,\n",
       "   'size': 36067,\n",
       "   'snippet': 'the striking similarities between Emily and her former character of <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> in the Harry Potter films: There are a lot of similarities, I think',\n",
       "   'timestamp': '2018-08-01T10:48:49Z',\n",
       "   'title': 'My Name Is Emily',\n",
       "   'wordcount': 4655},\n",
       "  {'ns': 0,\n",
       "   'pageid': 771537,\n",
       "   'size': 59133,\n",
       "   'snippet': 'Laughland Lee Jordan Luke Youngblood \\xa0 Neville Longbottom Matthew Lewis <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> \\xa0 Evanna Lynch Ernie Macmillan \\xa0 Louis Doyle \\xa0 Louis Doyle \\xa0 Draco Malfoy',\n",
       "   'timestamp': '2018-10-02T10:52:47Z',\n",
       "   'title': 'List of Harry Potter cast members',\n",
       "   'wordcount': 1246},\n",
       "  {'ns': 0,\n",
       "   'pageid': 271050,\n",
       "   'size': 10253,\n",
       "   'snippet': 'George Hotz, hacker and founder of comma.ai. Evanna Lynch, who portrays <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> in the Harry Potter movies, attended the Irish Centre For Talented Youth',\n",
       "   'timestamp': '2018-07-30T17:39:03Z',\n",
       "   'title': 'Center for Talented Youth',\n",
       "   'wordcount': 589},\n",
       "  {'ns': 0,\n",
       "   'pageid': 855389,\n",
       "   'size': 62366,\n",
       "   'snippet': '(Discworld) Alice Longbottom (Harry Potter) Augusta Longbottom (Harry Potter) <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> (Harry Potter) Brenda Loveknot (Discworld) Lythande (Thieves\\' World',\n",
       "   'timestamp': '2018-10-16T00:47:44Z',\n",
       "   'title': 'List of fictional witches',\n",
       "   'wordcount': 5662},\n",
       "  {'ns': 0,\n",
       "   'pageid': 32402649,\n",
       "   'size': 31962,\n",
       "   'snippet': 'his love for Hermione, but she shoots him down. However, Draco meets <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span>, and the two bond over Pigfarts, the wizarding school on Mars. In Dumbledore\\'s',\n",
       "   'timestamp': '2018-09-29T07:35:20Z',\n",
       "   'title': 'A Very Potter Sequel',\n",
       "   'wordcount': 3539},\n",
       "  {'ns': 0,\n",
       "   'pageid': 3833378,\n",
       "   'size': 8540,\n",
       "   'snippet': 'resident. Evanna Lynch, actress, starred in the Harry Potter films as <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> Arthur Mathews, co-writer of sitcom Father Ted, was a resident. Peter',\n",
       "   'timestamp': '2018-09-27T15:22:48Z',\n",
       "   'title': 'Termonfeckin',\n",
       "   'wordcount': 909},\n",
       "  {'ns': 0,\n",
       "   'pageid': 5032848,\n",
       "   'size': 20087,\n",
       "   'snippet': 'These sets have newly decorated mini-figures and animals including <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span>, Bellatrix Lestrange, new owls and new pigs. Also announced was a Lego',\n",
       "   'timestamp': '2018-10-07T14:36:24Z',\n",
       "   'title': 'Lego Harry Potter',\n",
       "   'wordcount': 1092},\n",
       "  {'ns': 0,\n",
       "   'pageid': 34085574,\n",
       "   'size': 4611,\n",
       "   'snippet': 'Potter and the Order of the Phoenix (<span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> (Evanna Lynch)) Harry Potter and the Half-Blood Prince (<span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> (Evanna Lynch)) Horrid Henry: The',\n",
       "   'timestamp': '2018-08-13T14:21:43Z',\n",
       "   'title': 'Shira Naor',\n",
       "   'wordcount': 443},\n",
       "  {'ns': 0,\n",
       "   'pageid': 44334,\n",
       "   'size': 34448,\n",
       "   'snippet': 'even more. Harry is left stranded on the train until Nymphadora Tonks (<span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> in the film adaptation) rescues him. Harry spends much of the year trailing',\n",
       "   'timestamp': '2018-09-22T05:46:20Z',\n",
       "   'title': 'Draco Malfoy',\n",
       "   'wordcount': 4395},\n",
       "  {'ns': 0,\n",
       "   'pageid': 21419143,\n",
       "   'size': 16180,\n",
       "   'snippet': 'goblin Griphook\\'s legs are broken in the escape from Malfoy Manor. <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> uses Skele-Gro to repair them. Sleekeazy\\'s Hair Potion is a beauty treatment',\n",
       "   'timestamp': '2018-10-23T18:33:17Z',\n",
       "   'title': 'Potions in Harry Potter',\n",
       "   'wordcount': 2101},\n",
       "  {'ns': 0,\n",
       "   'pageid': 19156567,\n",
       "   'size': 41338,\n",
       "   'snippet': 'Death Eaters alongside Harry, Hermione, Ginny, Neville Longbottom and <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> at the Department of Mysteries. He is injured in the fight, but makes',\n",
       "   'timestamp': '2018-10-07T16:35:30Z',\n",
       "   'title': 'Ron Weasley',\n",
       "   'wordcount': 4990},\n",
       "  {'ns': 0,\n",
       "   'pageid': 906492,\n",
       "   'size': 76774,\n",
       "   'snippet': 'Matthew Lewis (the actor who portrays Neville Longbottom), Evanna Lynch (<span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span>), Jamie Waylett (Vincent Crabbe), Rupert Grint (Ron Weasley), Chris',\n",
       "   'timestamp': '2018-10-16T04:54:32Z',\n",
       "   'title': 'Harry Potter fandom',\n",
       "   'wordcount': 8107},\n",
       "  {'ns': 0,\n",
       "   'pageid': 59572,\n",
       "   'size': 19811,\n",
       "   'snippet': '&quot;fiendish glee&quot; writing this scene, which features memorable commentary by <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span>. In 2014 Rowling started publishing a series of match reports from the',\n",
       "   'timestamp': '2018-09-26T22:29:29Z',\n",
       "   'title': 'Quidditch',\n",
       "   'wordcount': 2505},\n",
       "  {'ns': 0,\n",
       "   'pageid': 156489,\n",
       "   'size': 29708,\n",
       "   'snippet': 'including Neville Longbottom, Fred and George Weasley, Ginny Weasley and <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span>. Struggling to find a place to practise, Dobby the house elf tells him',\n",
       "   'timestamp': '2018-10-21T03:34:28Z',\n",
       "   'title': 'Harry Potter and the Order of the Phoenix',\n",
       "   'wordcount': 3268},\n",
       "  {'ns': 0,\n",
       "   'pageid': 156587,\n",
       "   'size': 61973,\n",
       "   'snippet': 'on Voldemort\\'s orders, preparing to capture Harry, but is stunned by <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> after touching her Dark Mark to summon Voldemort. Amycus, after seeing',\n",
       "   'timestamp': '2018-10-13T05:30:42Z',\n",
       "   'title': 'Death Eater',\n",
       "   'wordcount': 7104},\n",
       "  {'ns': 0,\n",
       "   'pageid': 33976749,\n",
       "   'size': 15741,\n",
       "   'snippet': '7, 2016. The announcement featured actress Evanna Lynch, who played <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> in the Harry Potter films. However, the soft opening was on March 31',\n",
       "   'timestamp': '2018-10-02T11:15:44Z',\n",
       "   'title': 'The Wizarding World of Harry Potter (Universal Studios Hollywood)',\n",
       "   'wordcount': 1264},\n",
       "  {'ns': 0,\n",
       "   'pageid': 776049,\n",
       "   'size': 93552,\n",
       "   'snippet': 'help of Dobby the house-elf they escape along with fellow prisoners <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span>, Mr. Ollivander, Dean Thomas, and the goblin Griphook. During the escape',\n",
       "   'timestamp': '2018-10-18T23:38:23Z',\n",
       "   'title': 'Harry Potter and the Deathly Hallows',\n",
       "   'wordcount': 9661},\n",
       "  {'ns': 0,\n",
       "   'pageid': 667658,\n",
       "   'size': 81208,\n",
       "   'snippet': 'prisoners are being kept in the basement on Voldemort\\'s orders, including <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span>, Dean Thomas, Griphook the Goblin, and Mr. Ollivander. When Snatchers',\n",
       "   'timestamp': '2018-10-21T03:12:48Z',\n",
       "   'title': 'Places in Harry Potter',\n",
       "   'wordcount': 10751},\n",
       "  {'ns': 0,\n",
       "   'pageid': 858575,\n",
       "   'size': 98258,\n",
       "   'snippet': 'but is spotted and petrified by Malfoy. Harry is found and saved by <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span>. At Hogwarts, Harry and Ron borrow textbooks for Slughorn\\'s Potions',\n",
       "   'timestamp': '2018-10-21T21:19:48Z',\n",
       "   'title': 'Harry Potter and the Half-Blood Prince (film)',\n",
       "   'wordcount': 9182},\n",
       "  {'ns': 0,\n",
       "   'pageid': 284598,\n",
       "   'size': 68965,\n",
       "   'snippet': 'for Magic&quot;. Pottermore. Retrieved 5 May 2017. Rowling, J.\\xa0K. (2003). &quot;<span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span>&quot;. Harry Potter and the Order of the Phoenix. Bloomsbury. ISBN\\xa00747551006',\n",
       "   'timestamp': '2018-10-19T08:13:25Z',\n",
       "   'title': 'Ministry of Magic',\n",
       "   'wordcount': 8228},\n",
       "  {'ns': 0,\n",
       "   'pageid': 17763993,\n",
       "   'size': 6306,\n",
       "   'snippet': 'Retrieved 7 January 2011.\\xa0   Andy (7 October 2010). &quot;Evanna Lynch talks <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> on the set of \\'Deathly Hallows\\'&quot;. Harry Potter Fan Zone. Retrieved 7',\n",
       "   'timestamp': '2018-09-15T04:43:24Z',\n",
       "   'title': 'Harry Potter Fan Zone',\n",
       "   'wordcount': 501},\n",
       "  {'ns': 0,\n",
       "   'pageid': 28373862,\n",
       "   'size': 37822,\n",
       "   'snippet': 'her teammates along the way. Dawn is mostly based on the character, <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> from the Harry Potter series. Voiced by Caitlynne Medrek. Jo, known',\n",
       "   'timestamp': '2018-10-10T15:29:59Z',\n",
       "   'title': 'Total Drama: Revenge of the Island',\n",
       "   'wordcount': 4352},\n",
       "  {'ns': 0,\n",
       "   'pageid': 54465045,\n",
       "   'size': 201532,\n",
       "   'snippet': 'Archived from the original on 10 July 2017. Retrieved 6 July 2017. &quot;<span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> role has been cast&quot;. CBBC Newsround. 2 February 2006. Archived from',\n",
       "   'timestamp': '2018-10-21T21:05:26Z',\n",
       "   'title': 'Wizarding World',\n",
       "   'wordcount': 11476},\n",
       "  {'ns': 0,\n",
       "   'pageid': 31941988,\n",
       "   'size': 89883,\n",
       "   'snippet': 'to a duel. McGonagall gathers the Hogwarts community for battle. At <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span>\\'s insistence, Harry speaks to Helena Ravenclaw\\'s ghost, who reveals',\n",
       "   'timestamp': '2018-10-23T13:09:36Z',\n",
       "   'title': 'Harry Potter and the Deathly Hallows – Part 2',\n",
       "   'wordcount': 7804},\n",
       "  {'ns': 0,\n",
       "   'pageid': 779883,\n",
       "   'size': 124700,\n",
       "   'snippet': 'Hallows, Neville Longbottom uses the DA\\'s coins to alert people such as <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> and Ginny Weasley that Harry, Ron and Hermione have returned to Hogwarts',\n",
       "   'timestamp': '2018-10-02T19:41:43Z',\n",
       "   'title': 'Magical objects in Harry Potter',\n",
       "   'wordcount': 18571},\n",
       "  {'ns': 0,\n",
       "   'pageid': 27511982,\n",
       "   'size': 52160,\n",
       "   'snippet': 'Weasley Wand Scabbers  4  Draco Malfoy Wand Golden Snitch Broomstick  5  <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> Wand Quibbler 2x3  6  Neville Longbottom Wand Mandrake Cauldron  7 ',\n",
       "   'timestamp': '2018-08-25T15:23:22Z',\n",
       "   'title': 'Lego Minifigures (theme)',\n",
       "   'wordcount': 1575},\n",
       "  {'ns': 0,\n",
       "   'pageid': 32446375,\n",
       "   'size': 4465,\n",
       "   'snippet': 'Chance Max in The Suite Life of Zack &amp; Cody Bridgette Dubois in Medium <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> in Harry Potter Violet Beauregarde in Charlie and the Chocolate Factory',\n",
       "   'timestamp': '2018-08-10T04:08:47Z',\n",
       "   'title': 'Kitty Kántor',\n",
       "   'wordcount': 338},\n",
       "  {'ns': 0,\n",
       "   'pageid': 12976982,\n",
       "   'size': 83013,\n",
       "   'snippet': 'Jaquish wrote two additional scenes which added Peeves, Moaning Myrtle, <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> and the Hogwarts Kitchen House Elves. The show first played at the Firehouse',\n",
       "   'timestamp': '2018-10-03T15:42:45Z',\n",
       "   'title': 'Parodies of Harry Potter',\n",
       "   'wordcount': 10133},\n",
       "  {'ns': 0,\n",
       "   'pageid': 3381404,\n",
       "   'size': 11798,\n",
       "   'snippet': '- Actress who plays Ginny Weasley. Evanna Lynch - Actress who plays <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span>. (episode 69) Chris Rankin - Actor who plays Percy Weasley. Crew members',\n",
       "   'timestamp': '2018-10-14T09:01:44Z',\n",
       "   'title': 'PotterCast',\n",
       "   'wordcount': 1474},\n",
       "  {'ns': 0,\n",
       "   'pageid': 12294600,\n",
       "   'size': 12016,\n",
       "   'snippet': '&quot;Stick it to Dolores&quot;2:3210.&quot;S.P.E.W.&quot;1:0111.&quot;The Human Hosepipe&quot;3:3312.&quot;<span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> is OK&quot;2:2013.&quot;The Godfather: Part II&quot;2:2514.&quot;The Weapon&quot;4:27Total length:41:58',\n",
       "   'timestamp': '2018-07-16T03:35:48Z',\n",
       "   'title': \"Voldemort Can't Stop the Rock!\",\n",
       "   'wordcount': 1084},\n",
       "  {'ns': 0,\n",
       "   'pageid': 24870495,\n",
       "   'size': 15929,\n",
       "   'snippet': 'Silk Spectre Evanna Lynch, Harry Potter and the Half-Blood Prince as <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> Shirley Manson, Terminator: The Sarah Connor Chronicles as Catherine',\n",
       "   'timestamp': '2018-03-05T11:16:37Z',\n",
       "   'title': '2009 Scream Awards',\n",
       "   'wordcount': 1352},\n",
       "  {'ns': 0,\n",
       "   'pageid': 20997070,\n",
       "   'size': 7748,\n",
       "   'snippet': 'Chile Evanna Lynch (born 1991), Irish actress, known for her role as <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> in the Harry Potter films Fionán Lynch, Irish politician Genevieve Springston',\n",
       "   'timestamp': '2018-09-13T01:10:46Z',\n",
       "   'title': 'List of people with surname Lynch',\n",
       "   'wordcount': 936},\n",
       "  {'ns': 0,\n",
       "   'pageid': 29494722,\n",
       "   'size': 9907,\n",
       "   'snippet': 'Cho Chang Charlie Weasley (uncredited) Arielle Goldman Lily Potter <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> Hedwig Fred Weasley Brian Rosenthal Seamus Finnigan James Potter Past',\n",
       "   'timestamp': '2018-09-29T07:35:24Z',\n",
       "   'title': 'A Very Potter Sequel (EP)',\n",
       "   'wordcount': 326},\n",
       "  {'ns': 0,\n",
       "   'pageid': 29960484,\n",
       "   'size': 6402,\n",
       "   'snippet': 'au/fashion/wrap/vraw_livestyle2010 https://bowienews.wordpress.com/2012/06/04/from-<span class=\"searchmatch\">luna</span>-<span class=\"searchmatch\">lovegood</span>-to-looking-lovely-in-bowie/ http://www.stylebistro',\n",
       "   'timestamp': '2018-10-06T15:03:49Z',\n",
       "   'title': 'Bowie Wong',\n",
       "   'wordcount': 768},\n",
       "  {'ns': 0,\n",
       "   'pageid': 45256977,\n",
       "   'size': 10437,\n",
       "   'snippet': 'Eleanor Morton (Ginny Weasley/Molly Weasley); Matt Tedford (Ron Weasley/<span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span>); Marny Godden (Hermione Granger/Uncle Albert); Jon Brittain (Hagrid);',\n",
       "   'timestamp': '2018-05-24T12:08:53Z',\n",
       "   'title': 'Weirdos Comedy Club',\n",
       "   'wordcount': 1405},\n",
       "  {'ns': 0,\n",
       "   'pageid': 43136825,\n",
       "   'size': 37894,\n",
       "   'snippet': 'comic strip character Lone Ranger (2009) character from The Lone Ranger <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> (2013) character from Harry Potter Lyra (2005) harpist or the harp from',\n",
       "   'timestamp': '2018-01-22T12:31:15Z',\n",
       "   'title': 'List of fictional characters on stamps of the United States',\n",
       "   'wordcount': 2818},\n",
       "  {'ns': 0,\n",
       "   'pageid': 947583,\n",
       "   'size': 72122,\n",
       "   'snippet': 'noted for her cleverness and creativity, and was described by Xenophilius <span class=\"searchmatch\">Lovegood</span> as beautiful. The Sorting Hat introduces her as &quot;Fair Ravenclaw, from glen&quot;',\n",
       "   'timestamp': '2018-10-24T10:40:37Z',\n",
       "   'title': 'Hogwarts staff',\n",
       "   'wordcount': 8384},\n",
       "  {'ns': 0,\n",
       "   'pageid': 29494707,\n",
       "   'size': 14359,\n",
       "   'snippet': 'Cho Chang Charlie Weasley (uncredited) Arielle Goldman Lily Potter <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> Hedwig Fred Weasley Brian Rosenthal Seamus Finnigan James Potter Past',\n",
       "   'timestamp': '2018-09-29T07:36:43Z',\n",
       "   'title': 'A Very StarKid Album',\n",
       "   'wordcount': 572},\n",
       "  {'ns': 0,\n",
       "   'pageid': 2228434,\n",
       "   'size': 68305,\n",
       "   'snippet': 'the Potters) or Tinworth in Cornwall. The all-wizard Weasley, Diggory, <span class=\"searchmatch\">Lovegood</span>, and Fawcett families live near the Muggle village of Ottery St Catchpole',\n",
       "   'timestamp': '2018-09-10T16:49:22Z',\n",
       "   'title': 'Fictional universe of Harry Potter',\n",
       "   'wordcount': 9196}],\n",
       " 'searchinfo': {'totalhits': 55}}"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data['query'] #remember that this is how we call upon a dictionary. dict_name[key]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Working with large dictionaries like this can be really overwhelming.  So we need to use a few more techniques to investigate it and get a feel for where out data is."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'dict'>\n"
     ]
    }
   ],
   "source": [
    "print(type(data['query']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So we see we have another dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "searchinfo <class 'dict'>\n",
      "search <class 'list'>\n"
     ]
    }
   ],
   "source": [
    "for key, value in data['query'].items(): #This structure is really important to remember!\n",
    "    print(key, type(value))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This loop has gone over our keys and values and told us what the name of the key and the datatype is. *Always remember that knowing the datatype is really important so we know how to access information from it.*\n",
    "\n",
    "Now that we are digging deeper, our dictionary syntax is going to get more complicated. We can start chaining our key lookups. But we'll want to save this next result as a variable!\n",
    "\n",
    "Looks like the first key is searchinfo and it is holding some of the metadata about out api query."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'totalhits': 55}"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data['query']['searchinfo']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case, this small dictionary is telling us that there were 55 hits.  In the scope of this data file, this means that there should be 55 results waiting for us somewhere in this data structure.  We need to find something that has a length of 55.\n",
    "\n",
    "Let's look at the search key and see what that list has."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'ns': 0,\n",
       "  'pageid': 391371,\n",
       "  'size': 47836,\n",
       "  'snippet': 'Xenophilius <span class=\"searchmatch\">Lovegood</span>, is the editor-in-chief of The Quibbler, a strange magazine claiming supposedly untrue things <span class=\"searchmatch\">Luna</span> and her father believe. When <span class=\"searchmatch\">Luna</span> was',\n",
       "  'timestamp': '2018-10-16T20:06:57Z',\n",
       "  'title': \"Dumbledore's Army\",\n",
       "  'wordcount': 4967},\n",
       " {'ns': 0,\n",
       "  'pageid': 3931304,\n",
       "  'size': 39384,\n",
       "  'snippet': 'podcast host and narrator. She rose to prominence for her portrayal of <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> in the Harry Potter film series, appearing in the last four films and',\n",
       "  'timestamp': '2018-10-10T09:09:37Z',\n",
       "  'title': 'Evanna Lynch',\n",
       "  'wordcount': 3060},\n",
       " {'ns': 0,\n",
       "  'pageid': 874930,\n",
       "  'size': 86965,\n",
       "  'snippet': 'semi-humans, greatly attractive Wood Nymphs <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span>, a pupil at Hogwarts, and her father Xenophilius <span class=\"searchmatch\">Lovegood</span> profess belief in a number of magical creatures',\n",
       "  'timestamp': '2018-10-24T12:21:36Z',\n",
       "  'title': 'Magical creatures in Harry Potter',\n",
       "  'wordcount': 12101},\n",
       " {'ns': 0,\n",
       "  'pageid': 23483380,\n",
       "  'size': 5111,\n",
       "  'snippet': 'Shrek Super Slam. <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> in the Harry Potter series <span class=\"searchmatch\">Luna</span> Maximoff, supporting character in the Marvel Comics universe <span class=\"searchmatch\">Luna</span> Mothews, the teenage',\n",
       "  'timestamp': '2018-10-18T19:04:14Z',\n",
       "  'title': 'Luna (name)',\n",
       "  'wordcount': 527},\n",
       " {'ns': 0,\n",
       "  'pageid': 38290297,\n",
       "  'size': 18775,\n",
       "  'snippet': 'the company to reprise his role as Harry Potter, and Evanna Lynch as <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span>, who played the character in the original film series. The script and',\n",
       "  'timestamp': '2018-06-30T10:59:29Z',\n",
       "  'title': 'A Very Potter Senior Year',\n",
       "  'wordcount': 2018},\n",
       " {'ns': 0,\n",
       "  'pageid': 891112,\n",
       "  'size': 19444,\n",
       "  'snippet': '(2010). He played Xenophillius <span class=\"searchmatch\">Lovegood</span>, editor of the wizarding magazine The Quibbler and father of the eccentric <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span>. In the same interview, he',\n",
       "  'timestamp': '2018-09-12T07:15:17Z',\n",
       "  'title': 'Rhys Ifans',\n",
       "  'wordcount': 1441},\n",
       " {'ns': 0,\n",
       "  'pageid': 11061022,\n",
       "  'size': 65170,\n",
       "  'snippet': 'the mini-serial Proof. During the same time, Ronan auditioned to play <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> in Harry Potter and the Order of the Phoenix, a role she eventually',\n",
       "  'timestamp': '2018-10-24T12:23:42Z',\n",
       "  'title': 'Saoirse Ronan',\n",
       "  'wordcount': 4796},\n",
       " {'ns': 0,\n",
       "  'pageid': 3512033,\n",
       "  'size': 89484,\n",
       "  'snippet': 'escorting Neville to the Hogwarts Express. Xenophilius <span class=\"searchmatch\">Lovegood</span> is Pandora <span class=\"searchmatch\">Lovegood</span>\\'s husband, <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span>\\'s father, and the editor of The Quibbler, a magazine',\n",
       "  'timestamp': '2018-10-22T17:36:29Z',\n",
       "  'title': 'List of supporting Harry Potter characters',\n",
       "  'wordcount': 12755},\n",
       " {'ns': 0,\n",
       "  'pageid': 670407,\n",
       "  'size': 94795,\n",
       "  'snippet': 'she attacks and insults them. Harry, Hermione, Ron, Ginny Weasley, <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span>, and Neville Longbottom fly to the Ministry of Magic on Thestrals in',\n",
       "  'timestamp': '2018-10-25T09:55:53Z',\n",
       "  'title': 'Harry Potter and the Order of the Phoenix (film)',\n",
       "  'wordcount': 9674},\n",
       " {'ns': 0,\n",
       "  'pageid': 45100,\n",
       "  'size': 48610,\n",
       "  'snippet': 'that Hermione is an only child. Rowling has described the character of <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> as the &quot;anti-Hermione&quot; as they are so different. Hermione\\'s nemesis',\n",
       "  'timestamp': '2018-10-20T15:44:28Z',\n",
       "  'title': 'Hermione Granger',\n",
       "  'wordcount': 5898},\n",
       " {'ns': 0,\n",
       "  'pageid': 667601,\n",
       "  'size': 29845,\n",
       "  'snippet': 'Dumbledore\\'s Army. <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> – Ravenclaw student one year below Harry, a member of Dumbledore\\'s Army. Xenophilius <span class=\"searchmatch\">Lovegood</span> – Father of <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span>, and editor',\n",
       "  'timestamp': '2018-10-23T05:01:51Z',\n",
       "  'title': 'List of Harry Potter characters',\n",
       "  'wordcount': 3585},\n",
       " {'ns': 0,\n",
       "  'pageid': 827110,\n",
       "  'size': 6141,\n",
       "  'snippet': 'Hallows, a character named Xenophilius <span class=\"searchmatch\">Lovegood</span> (the father of one of Harry Potter\\'s more eccentric friends, <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span>) is characterized by his interest',\n",
       "  'timestamp': '2018-06-23T13:19:23Z',\n",
       "  'title': 'Xenophilia',\n",
       "  'wordcount': 848},\n",
       " {'ns': 0,\n",
       "  'pageid': 10093585,\n",
       "  'size': 17042,\n",
       "  'snippet': 'Quincey in Atonement has been called &quot;impressive&quot;. She auditioned to play <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> in Harry Potter and the Order of the Phoenix, though the role ultimately',\n",
       "  'timestamp': '2018-10-03T06:52:19Z',\n",
       "  'title': 'Juno Temple',\n",
       "  'wordcount': 1108},\n",
       " {'ns': 0,\n",
       "  'pageid': 2298654,\n",
       "  'size': 8444,\n",
       "  'snippet': 'Bonnie Wright (Ginny Weasley), Katie Leung (Cho Chang), Evanna Lynch (<span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span>), and Scarlett Byrne (Pansy Parkinson). Azad and Shefali Chowdhury (who',\n",
       "  'timestamp': '2018-09-20T08:31:42Z',\n",
       "  'title': 'Afshan Azad',\n",
       "  'wordcount': 597},\n",
       " {'ns': 0,\n",
       "  'pageid': 241855,\n",
       "  'size': 15333,\n",
       "  'snippet': 'Hoppy, Milly and Mauler. He had a grandson named Rolf, who married <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> (a character who appeared in the last three Harry Potter novels) some',\n",
       "  'timestamp': '2018-10-20T22:08:41Z',\n",
       "  'title': 'Fantastic Beasts and Where to Find Them',\n",
       "  'wordcount': 1704},\n",
       " {'ns': 0,\n",
       "  'pageid': 9834441,\n",
       "  'size': 75188,\n",
       "  'snippet': 'the trio awkwardly attempt to leave but are stopped by <span class=\"searchmatch\">Lovegood</span>. He reveals that <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> has been kidnapped before summoning the Death Eaters, intending',\n",
       "  'timestamp': '2018-10-23T17:40:58Z',\n",
       "  'title': 'Harry Potter and the Deathly Hallows – Part 1',\n",
       "  'wordcount': 6698},\n",
       " {'ns': 0,\n",
       "  'pageid': 48543224,\n",
       "  'size': 36067,\n",
       "  'snippet': 'the striking similarities between Emily and her former character of <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> in the Harry Potter films: There are a lot of similarities, I think',\n",
       "  'timestamp': '2018-08-01T10:48:49Z',\n",
       "  'title': 'My Name Is Emily',\n",
       "  'wordcount': 4655},\n",
       " {'ns': 0,\n",
       "  'pageid': 771537,\n",
       "  'size': 59133,\n",
       "  'snippet': 'Laughland Lee Jordan Luke Youngblood \\xa0 Neville Longbottom Matthew Lewis <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> \\xa0 Evanna Lynch Ernie Macmillan \\xa0 Louis Doyle \\xa0 Louis Doyle \\xa0 Draco Malfoy',\n",
       "  'timestamp': '2018-10-02T10:52:47Z',\n",
       "  'title': 'List of Harry Potter cast members',\n",
       "  'wordcount': 1246},\n",
       " {'ns': 0,\n",
       "  'pageid': 271050,\n",
       "  'size': 10253,\n",
       "  'snippet': 'George Hotz, hacker and founder of comma.ai. Evanna Lynch, who portrays <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> in the Harry Potter movies, attended the Irish Centre For Talented Youth',\n",
       "  'timestamp': '2018-07-30T17:39:03Z',\n",
       "  'title': 'Center for Talented Youth',\n",
       "  'wordcount': 589},\n",
       " {'ns': 0,\n",
       "  'pageid': 855389,\n",
       "  'size': 62366,\n",
       "  'snippet': '(Discworld) Alice Longbottom (Harry Potter) Augusta Longbottom (Harry Potter) <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> (Harry Potter) Brenda Loveknot (Discworld) Lythande (Thieves\\' World',\n",
       "  'timestamp': '2018-10-16T00:47:44Z',\n",
       "  'title': 'List of fictional witches',\n",
       "  'wordcount': 5662},\n",
       " {'ns': 0,\n",
       "  'pageid': 32402649,\n",
       "  'size': 31962,\n",
       "  'snippet': 'his love for Hermione, but she shoots him down. However, Draco meets <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span>, and the two bond over Pigfarts, the wizarding school on Mars. In Dumbledore\\'s',\n",
       "  'timestamp': '2018-09-29T07:35:20Z',\n",
       "  'title': 'A Very Potter Sequel',\n",
       "  'wordcount': 3539},\n",
       " {'ns': 0,\n",
       "  'pageid': 3833378,\n",
       "  'size': 8540,\n",
       "  'snippet': 'resident. Evanna Lynch, actress, starred in the Harry Potter films as <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> Arthur Mathews, co-writer of sitcom Father Ted, was a resident. Peter',\n",
       "  'timestamp': '2018-09-27T15:22:48Z',\n",
       "  'title': 'Termonfeckin',\n",
       "  'wordcount': 909},\n",
       " {'ns': 0,\n",
       "  'pageid': 5032848,\n",
       "  'size': 20087,\n",
       "  'snippet': 'These sets have newly decorated mini-figures and animals including <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span>, Bellatrix Lestrange, new owls and new pigs. Also announced was a Lego',\n",
       "  'timestamp': '2018-10-07T14:36:24Z',\n",
       "  'title': 'Lego Harry Potter',\n",
       "  'wordcount': 1092},\n",
       " {'ns': 0,\n",
       "  'pageid': 34085574,\n",
       "  'size': 4611,\n",
       "  'snippet': 'Potter and the Order of the Phoenix (<span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> (Evanna Lynch)) Harry Potter and the Half-Blood Prince (<span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> (Evanna Lynch)) Horrid Henry: The',\n",
       "  'timestamp': '2018-08-13T14:21:43Z',\n",
       "  'title': 'Shira Naor',\n",
       "  'wordcount': 443},\n",
       " {'ns': 0,\n",
       "  'pageid': 44334,\n",
       "  'size': 34448,\n",
       "  'snippet': 'even more. Harry is left stranded on the train until Nymphadora Tonks (<span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> in the film adaptation) rescues him. Harry spends much of the year trailing',\n",
       "  'timestamp': '2018-09-22T05:46:20Z',\n",
       "  'title': 'Draco Malfoy',\n",
       "  'wordcount': 4395},\n",
       " {'ns': 0,\n",
       "  'pageid': 21419143,\n",
       "  'size': 16180,\n",
       "  'snippet': 'goblin Griphook\\'s legs are broken in the escape from Malfoy Manor. <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> uses Skele-Gro to repair them. Sleekeazy\\'s Hair Potion is a beauty treatment',\n",
       "  'timestamp': '2018-10-23T18:33:17Z',\n",
       "  'title': 'Potions in Harry Potter',\n",
       "  'wordcount': 2101},\n",
       " {'ns': 0,\n",
       "  'pageid': 19156567,\n",
       "  'size': 41338,\n",
       "  'snippet': 'Death Eaters alongside Harry, Hermione, Ginny, Neville Longbottom and <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> at the Department of Mysteries. He is injured in the fight, but makes',\n",
       "  'timestamp': '2018-10-07T16:35:30Z',\n",
       "  'title': 'Ron Weasley',\n",
       "  'wordcount': 4990},\n",
       " {'ns': 0,\n",
       "  'pageid': 906492,\n",
       "  'size': 76774,\n",
       "  'snippet': 'Matthew Lewis (the actor who portrays Neville Longbottom), Evanna Lynch (<span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span>), Jamie Waylett (Vincent Crabbe), Rupert Grint (Ron Weasley), Chris',\n",
       "  'timestamp': '2018-10-16T04:54:32Z',\n",
       "  'title': 'Harry Potter fandom',\n",
       "  'wordcount': 8107},\n",
       " {'ns': 0,\n",
       "  'pageid': 59572,\n",
       "  'size': 19811,\n",
       "  'snippet': '&quot;fiendish glee&quot; writing this scene, which features memorable commentary by <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span>. In 2014 Rowling started publishing a series of match reports from the',\n",
       "  'timestamp': '2018-09-26T22:29:29Z',\n",
       "  'title': 'Quidditch',\n",
       "  'wordcount': 2505},\n",
       " {'ns': 0,\n",
       "  'pageid': 156489,\n",
       "  'size': 29708,\n",
       "  'snippet': 'including Neville Longbottom, Fred and George Weasley, Ginny Weasley and <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span>. Struggling to find a place to practise, Dobby the house elf tells him',\n",
       "  'timestamp': '2018-10-21T03:34:28Z',\n",
       "  'title': 'Harry Potter and the Order of the Phoenix',\n",
       "  'wordcount': 3268},\n",
       " {'ns': 0,\n",
       "  'pageid': 156587,\n",
       "  'size': 61973,\n",
       "  'snippet': 'on Voldemort\\'s orders, preparing to capture Harry, but is stunned by <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> after touching her Dark Mark to summon Voldemort. Amycus, after seeing',\n",
       "  'timestamp': '2018-10-13T05:30:42Z',\n",
       "  'title': 'Death Eater',\n",
       "  'wordcount': 7104},\n",
       " {'ns': 0,\n",
       "  'pageid': 33976749,\n",
       "  'size': 15741,\n",
       "  'snippet': '7, 2016. The announcement featured actress Evanna Lynch, who played <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> in the Harry Potter films. However, the soft opening was on March 31',\n",
       "  'timestamp': '2018-10-02T11:15:44Z',\n",
       "  'title': 'The Wizarding World of Harry Potter (Universal Studios Hollywood)',\n",
       "  'wordcount': 1264},\n",
       " {'ns': 0,\n",
       "  'pageid': 776049,\n",
       "  'size': 93552,\n",
       "  'snippet': 'help of Dobby the house-elf they escape along with fellow prisoners <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span>, Mr. Ollivander, Dean Thomas, and the goblin Griphook. During the escape',\n",
       "  'timestamp': '2018-10-18T23:38:23Z',\n",
       "  'title': 'Harry Potter and the Deathly Hallows',\n",
       "  'wordcount': 9661},\n",
       " {'ns': 0,\n",
       "  'pageid': 667658,\n",
       "  'size': 81208,\n",
       "  'snippet': 'prisoners are being kept in the basement on Voldemort\\'s orders, including <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span>, Dean Thomas, Griphook the Goblin, and Mr. Ollivander. When Snatchers',\n",
       "  'timestamp': '2018-10-21T03:12:48Z',\n",
       "  'title': 'Places in Harry Potter',\n",
       "  'wordcount': 10751},\n",
       " {'ns': 0,\n",
       "  'pageid': 858575,\n",
       "  'size': 98258,\n",
       "  'snippet': 'but is spotted and petrified by Malfoy. Harry is found and saved by <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span>. At Hogwarts, Harry and Ron borrow textbooks for Slughorn\\'s Potions',\n",
       "  'timestamp': '2018-10-21T21:19:48Z',\n",
       "  'title': 'Harry Potter and the Half-Blood Prince (film)',\n",
       "  'wordcount': 9182},\n",
       " {'ns': 0,\n",
       "  'pageid': 284598,\n",
       "  'size': 68965,\n",
       "  'snippet': 'for Magic&quot;. Pottermore. Retrieved 5 May 2017. Rowling, J.\\xa0K. (2003). &quot;<span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span>&quot;. Harry Potter and the Order of the Phoenix. Bloomsbury. ISBN\\xa00747551006',\n",
       "  'timestamp': '2018-10-19T08:13:25Z',\n",
       "  'title': 'Ministry of Magic',\n",
       "  'wordcount': 8228},\n",
       " {'ns': 0,\n",
       "  'pageid': 17763993,\n",
       "  'size': 6306,\n",
       "  'snippet': 'Retrieved 7 January 2011.\\xa0   Andy (7 October 2010). &quot;Evanna Lynch talks <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> on the set of \\'Deathly Hallows\\'&quot;. Harry Potter Fan Zone. Retrieved 7',\n",
       "  'timestamp': '2018-09-15T04:43:24Z',\n",
       "  'title': 'Harry Potter Fan Zone',\n",
       "  'wordcount': 501},\n",
       " {'ns': 0,\n",
       "  'pageid': 28373862,\n",
       "  'size': 37822,\n",
       "  'snippet': 'her teammates along the way. Dawn is mostly based on the character, <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> from the Harry Potter series. Voiced by Caitlynne Medrek. Jo, known',\n",
       "  'timestamp': '2018-10-10T15:29:59Z',\n",
       "  'title': 'Total Drama: Revenge of the Island',\n",
       "  'wordcount': 4352},\n",
       " {'ns': 0,\n",
       "  'pageid': 54465045,\n",
       "  'size': 201532,\n",
       "  'snippet': 'Archived from the original on 10 July 2017. Retrieved 6 July 2017. &quot;<span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> role has been cast&quot;. CBBC Newsround. 2 February 2006. Archived from',\n",
       "  'timestamp': '2018-10-21T21:05:26Z',\n",
       "  'title': 'Wizarding World',\n",
       "  'wordcount': 11476},\n",
       " {'ns': 0,\n",
       "  'pageid': 31941988,\n",
       "  'size': 89883,\n",
       "  'snippet': 'to a duel. McGonagall gathers the Hogwarts community for battle. At <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span>\\'s insistence, Harry speaks to Helena Ravenclaw\\'s ghost, who reveals',\n",
       "  'timestamp': '2018-10-23T13:09:36Z',\n",
       "  'title': 'Harry Potter and the Deathly Hallows – Part 2',\n",
       "  'wordcount': 7804},\n",
       " {'ns': 0,\n",
       "  'pageid': 779883,\n",
       "  'size': 124700,\n",
       "  'snippet': 'Hallows, Neville Longbottom uses the DA\\'s coins to alert people such as <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> and Ginny Weasley that Harry, Ron and Hermione have returned to Hogwarts',\n",
       "  'timestamp': '2018-10-02T19:41:43Z',\n",
       "  'title': 'Magical objects in Harry Potter',\n",
       "  'wordcount': 18571},\n",
       " {'ns': 0,\n",
       "  'pageid': 27511982,\n",
       "  'size': 52160,\n",
       "  'snippet': 'Weasley Wand Scabbers  4  Draco Malfoy Wand Golden Snitch Broomstick  5  <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> Wand Quibbler 2x3  6  Neville Longbottom Wand Mandrake Cauldron  7 ',\n",
       "  'timestamp': '2018-08-25T15:23:22Z',\n",
       "  'title': 'Lego Minifigures (theme)',\n",
       "  'wordcount': 1575},\n",
       " {'ns': 0,\n",
       "  'pageid': 32446375,\n",
       "  'size': 4465,\n",
       "  'snippet': 'Chance Max in The Suite Life of Zack &amp; Cody Bridgette Dubois in Medium <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> in Harry Potter Violet Beauregarde in Charlie and the Chocolate Factory',\n",
       "  'timestamp': '2018-08-10T04:08:47Z',\n",
       "  'title': 'Kitty Kántor',\n",
       "  'wordcount': 338},\n",
       " {'ns': 0,\n",
       "  'pageid': 12976982,\n",
       "  'size': 83013,\n",
       "  'snippet': 'Jaquish wrote two additional scenes which added Peeves, Moaning Myrtle, <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> and the Hogwarts Kitchen House Elves. The show first played at the Firehouse',\n",
       "  'timestamp': '2018-10-03T15:42:45Z',\n",
       "  'title': 'Parodies of Harry Potter',\n",
       "  'wordcount': 10133},\n",
       " {'ns': 0,\n",
       "  'pageid': 3381404,\n",
       "  'size': 11798,\n",
       "  'snippet': '- Actress who plays Ginny Weasley. Evanna Lynch - Actress who plays <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span>. (episode 69) Chris Rankin - Actor who plays Percy Weasley. Crew members',\n",
       "  'timestamp': '2018-10-14T09:01:44Z',\n",
       "  'title': 'PotterCast',\n",
       "  'wordcount': 1474},\n",
       " {'ns': 0,\n",
       "  'pageid': 12294600,\n",
       "  'size': 12016,\n",
       "  'snippet': '&quot;Stick it to Dolores&quot;2:3210.&quot;S.P.E.W.&quot;1:0111.&quot;The Human Hosepipe&quot;3:3312.&quot;<span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> is OK&quot;2:2013.&quot;The Godfather: Part II&quot;2:2514.&quot;The Weapon&quot;4:27Total length:41:58',\n",
       "  'timestamp': '2018-07-16T03:35:48Z',\n",
       "  'title': \"Voldemort Can't Stop the Rock!\",\n",
       "  'wordcount': 1084},\n",
       " {'ns': 0,\n",
       "  'pageid': 24870495,\n",
       "  'size': 15929,\n",
       "  'snippet': 'Silk Spectre Evanna Lynch, Harry Potter and the Half-Blood Prince as <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> Shirley Manson, Terminator: The Sarah Connor Chronicles as Catherine',\n",
       "  'timestamp': '2018-03-05T11:16:37Z',\n",
       "  'title': '2009 Scream Awards',\n",
       "  'wordcount': 1352},\n",
       " {'ns': 0,\n",
       "  'pageid': 20997070,\n",
       "  'size': 7748,\n",
       "  'snippet': 'Chile Evanna Lynch (born 1991), Irish actress, known for her role as <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> in the Harry Potter films Fionán Lynch, Irish politician Genevieve Springston',\n",
       "  'timestamp': '2018-09-13T01:10:46Z',\n",
       "  'title': 'List of people with surname Lynch',\n",
       "  'wordcount': 936},\n",
       " {'ns': 0,\n",
       "  'pageid': 29494722,\n",
       "  'size': 9907,\n",
       "  'snippet': 'Cho Chang Charlie Weasley (uncredited) Arielle Goldman Lily Potter <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> Hedwig Fred Weasley Brian Rosenthal Seamus Finnigan James Potter Past',\n",
       "  'timestamp': '2018-09-29T07:35:24Z',\n",
       "  'title': 'A Very Potter Sequel (EP)',\n",
       "  'wordcount': 326},\n",
       " {'ns': 0,\n",
       "  'pageid': 29960484,\n",
       "  'size': 6402,\n",
       "  'snippet': 'au/fashion/wrap/vraw_livestyle2010 https://bowienews.wordpress.com/2012/06/04/from-<span class=\"searchmatch\">luna</span>-<span class=\"searchmatch\">lovegood</span>-to-looking-lovely-in-bowie/ http://www.stylebistro',\n",
       "  'timestamp': '2018-10-06T15:03:49Z',\n",
       "  'title': 'Bowie Wong',\n",
       "  'wordcount': 768},\n",
       " {'ns': 0,\n",
       "  'pageid': 45256977,\n",
       "  'size': 10437,\n",
       "  'snippet': 'Eleanor Morton (Ginny Weasley/Molly Weasley); Matt Tedford (Ron Weasley/<span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span>); Marny Godden (Hermione Granger/Uncle Albert); Jon Brittain (Hagrid);',\n",
       "  'timestamp': '2018-05-24T12:08:53Z',\n",
       "  'title': 'Weirdos Comedy Club',\n",
       "  'wordcount': 1405},\n",
       " {'ns': 0,\n",
       "  'pageid': 43136825,\n",
       "  'size': 37894,\n",
       "  'snippet': 'comic strip character Lone Ranger (2009) character from The Lone Ranger <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> (2013) character from Harry Potter Lyra (2005) harpist or the harp from',\n",
       "  'timestamp': '2018-01-22T12:31:15Z',\n",
       "  'title': 'List of fictional characters on stamps of the United States',\n",
       "  'wordcount': 2818},\n",
       " {'ns': 0,\n",
       "  'pageid': 947583,\n",
       "  'size': 72122,\n",
       "  'snippet': 'noted for her cleverness and creativity, and was described by Xenophilius <span class=\"searchmatch\">Lovegood</span> as beautiful. The Sorting Hat introduces her as &quot;Fair Ravenclaw, from glen&quot;',\n",
       "  'timestamp': '2018-10-24T10:40:37Z',\n",
       "  'title': 'Hogwarts staff',\n",
       "  'wordcount': 8384},\n",
       " {'ns': 0,\n",
       "  'pageid': 29494707,\n",
       "  'size': 14359,\n",
       "  'snippet': 'Cho Chang Charlie Weasley (uncredited) Arielle Goldman Lily Potter <span class=\"searchmatch\">Luna</span> <span class=\"searchmatch\">Lovegood</span> Hedwig Fred Weasley Brian Rosenthal Seamus Finnigan James Potter Past',\n",
       "  'timestamp': '2018-09-29T07:36:43Z',\n",
       "  'title': 'A Very StarKid Album',\n",
       "  'wordcount': 572},\n",
       " {'ns': 0,\n",
       "  'pageid': 2228434,\n",
       "  'size': 68305,\n",
       "  'snippet': 'the Potters) or Tinworth in Cornwall. The all-wizard Weasley, Diggory, <span class=\"searchmatch\">Lovegood</span>, and Fawcett families live near the Muggle village of Ottery St Catchpole',\n",
       "  'timestamp': '2018-09-10T16:49:22Z',\n",
       "  'title': 'Fictional universe of Harry Potter',\n",
       "  'wordcount': 9196}]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data['query']['search'] # check out this structure! We can keep chaining these."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hmm, that looks like a lot of stuff and we know that it's a list of data. So let's check the length."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "55"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(data['query']['search'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Yay! There's our 55!\n",
    "\n",
    "Let's save this to a variable and call it records."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "records = data['query']['search']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**HOMEWORK HINT!** The data structure that you'll be working with on your homework is a simplified version of this file and represents only the results coming in from this key.  This means that your data variable coming in from the json parser will be a list of dictionaries and not a dictionary like the start of this lecture. \n",
    "\n",
    "This means that you should **start here** if you are looking for more direct hints for how things should be working.\n",
    "\n",
    "Now that we have a new data structure, let's repeat this process of inspection of what we have."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'list'>\n"
     ]
    }
   ],
   "source": [
    "print(type(records))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "55\n"
     ]
    }
   ],
   "source": [
    "print(len(records))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So we have a list of what?  Let's inspect the first one in this and see what's going on.  We can use regular slicing syntax here to get out the first element."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "entry = records[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'dict'>\n",
      "7\n"
     ]
    }
   ],
   "source": [
    "print(type(entry))\n",
    "print(len(entry))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So this is a dictionary of 7 key/value pairs.  Let's look at the keys and their datatypes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ns <class 'int'>\n",
      "title <class 'str'>\n",
      "pageid <class 'int'>\n",
      "size <class 'int'>\n",
      "wordcount <class 'int'>\n",
      "snippet <class 'str'>\n",
      "timestamp <class 'str'>\n"
     ]
    }
   ],
   "source": [
    "for key, value in entry.items():\n",
    "    print(key, type(value))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So all of these values are either a string of an int, which means that we don't have any more drilling down to do.\n",
    "\n",
    "Generally, once you get down to non-aggregated data types you will be able to start analyzing the data and generating results.\n",
    "\n",
    "Let's check that all of the entries in this list are of type dictionary.  We're going to use a few things here.  First, we'll use a list accumulator to gather up all the data types.  Then we can do a few things to that list to inspect how many of what do we have."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "types = [] \n",
    "\n",
    "for item in records:\n",
    "    types.append(type(item))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[<class 'dict'>, <class 'dict'>, <class 'dict'>, <class 'dict'>, <class 'dict'>, <class 'dict'>, <class 'dict'>, <class 'dict'>, <class 'dict'>, <class 'dict'>, <class 'dict'>, <class 'dict'>, <class 'dict'>, <class 'dict'>, <class 'dict'>, <class 'dict'>, <class 'dict'>, <class 'dict'>, <class 'dict'>, <class 'dict'>, <class 'dict'>, <class 'dict'>, <class 'dict'>, <class 'dict'>, <class 'dict'>, <class 'dict'>, <class 'dict'>, <class 'dict'>, <class 'dict'>, <class 'dict'>, <class 'dict'>, <class 'dict'>, <class 'dict'>, <class 'dict'>, <class 'dict'>, <class 'dict'>, <class 'dict'>, <class 'dict'>, <class 'dict'>, <class 'dict'>, <class 'dict'>, <class 'dict'>, <class 'dict'>, <class 'dict'>, <class 'dict'>, <class 'dict'>, <class 'dict'>, <class 'dict'>, <class 'dict'>, <class 'dict'>, <class 'dict'>, <class 'dict'>, <class 'dict'>, <class 'dict'>, <class 'dict'>]\n"
     ]
    }
   ],
   "source": [
    "print(types)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There's a lot here, and we can try to eyeball it but we can also use python to look at stuff here.\n",
    "\n",
    "What we really care about here are any unique values within the list.  There are a few ways we can do this.  First, we can try to look at the distinct values.  \n",
    "\n",
    "The `set()` datatype in python will be good for this.  The set object is a little little like a list, where it will hold values.  This is actually a really fun mathematical tool that we can use, but set containers are really valuable for looking at combinations of data in lists.\n",
    "\n",
    "Most important to know about sets is that they can only hold unique values.  This means that when you recast as list to a set it collapses all the values down to just the unique values.  So maybe you can see how handy this is?\n",
    "\n",
    "Let's take a look."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{<class 'dict'>}\n"
     ]
    }
   ],
   "source": [
    "print(set(types))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We only have one result here, which means that all the values are the same across the entire original list.  This also means that all the items in the records list have the same data type.\n",
    "\n",
    "We could use this same method to check how long each dictionary is within the list of records and check that they are uniform. **HOMEWORK HINT**.\n",
    "\n",
    "Let's now look at the words used in the snippet. About how long is each snippet? How uniform are they? Knowing that might influence what we choose to do with this text. We're going to clean out all the punctuation from each and then split to get a rough approximation of how long each is. This should give us a range.\n",
    "\n",
    "While we're in there, we might as well count all the words, right? Let's do that too since we already have the words split up.\n",
    "\n",
    "We're going to use a nested for loop to first grab and split the text, and then the word count pattern to count all the words seen across all the records."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\n",
    "import string\n",
    "\n",
    "def clean_line(line_str):\n",
    "    cleanline = line_str\n",
    "    for punc in string.punctuation:\n",
    "        cleanline = cleanline.replace(punc, \"\")\n",
    "    return cleanline\n",
    "\n",
    "wordcount = {}\n",
    "lengthcount = []\n",
    "\n",
    "for item in records:\n",
    "    snippet = item['snippet']\n",
    "    clean_text = clean_line(snippet).lower()\n",
    "    words = clean_text.split()\n",
    "    # let's add the length of the words\n",
    "    lengthcount.append(len(words))\n",
    "    # now that we have the words we can loop over them and count them\n",
    "    for word in words:\n",
    "        if word in wordcount:\n",
    "            wordcount[word] += 1\n",
    "        else:\n",
    "            wordcount[word] = 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at the lengths of the snippets first and see how uniform the lengths might be."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[26, 29, 27, 27, 29, 27, 28, 27, 26, 26, 30, 26, 28, 24, 28, 28, 27, 23, 27, 21, 29, 26, 25, 28, 29, 27, 26, 23, 26, 27, 26, 27, 26, 25, 27, 26, 27, 27, 27, 24, 28, 25, 29, 27, 25, 16, 24, 27, 23, 5, 20, 27, 25, 23, 24]\n"
     ]
    }
   ],
   "source": [
    "print(lengthcount)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This looks pretty uniform, but remember that we can use set to see how many unique values there are."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{5, 16, 20, 21, 23, 24, 25, 26, 27, 28, 29, 30}\n"
     ]
    }
   ],
   "source": [
    "print(set(lengthcount))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Interesting!  There's one really short one.  Let's put a pin in that."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Now that we have all these counts, we can try and look at the total spread.  Let's look at the unique counts of this dictionary accumulator. This will give us a sense of how the data is spread around. \n",
    "\n",
    "We can get the list of values from this dictionary with the .values() method, then pass that into a set to see all the unique values.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 16, 17, 26, 27, 28, 32, 40, 57, 59, 83, 116}\n"
     ]
    }
   ],
   "source": [
    "uniquecounts = set(wordcount.values())\n",
    "# let's sort it so we can see things better\n",
    "print(uniquecounts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks like the top word has 116 counts and the lowest is 1. Given that we have only 55 results, that's a word that must appear just more than twice for each snippet!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('span', 116),\n",
       " ('the', 83),\n",
       " ('classsearchmatchlovegoodspan', 59),\n",
       " ('classsearchmatchlunaspan', 57),\n",
       " ('and', 40),\n",
       " ('harry', 32),\n",
       " ('in', 28),\n",
       " ('of', 27),\n",
       " ('potter', 26),\n",
       " ('to', 17),\n",
       " ('a', 16),\n",
       " ('weasley', 15),\n",
       " ('lynch', 14),\n",
       " ('is', 12),\n",
       " ('evanna', 12),\n",
       " ('as', 11),\n",
       " ('who', 11),\n",
       " ('character', 10),\n",
       " ('her', 9),\n",
       " ('longbottom', 9),\n",
       " ('on', 9),\n",
       " ('from', 9),\n",
       " ('for', 8),\n",
       " ('neville', 8),\n",
       " ('by', 8),\n",
       " ('father', 7),\n",
       " ('ginny', 7),\n",
       " ('xenophilius', 6),\n",
       " ('hogwarts', 6),\n",
       " ('hermione', 6),\n",
       " ('was', 5),\n",
       " ('she', 5),\n",
       " ('series', 5),\n",
       " ('films', 5),\n",
       " ('at', 5),\n",
       " ('role', 5),\n",
       " ('he', 5),\n",
       " ('ron', 5),\n",
       " ('are', 5),\n",
       " ('but', 5),\n",
       " ('actress', 5),\n",
       " ('7', 5),\n",
       " ('quibbler', 4),\n",
       " ('played', 4),\n",
       " ('order', 4),\n",
       " ('phoenix', 4),\n",
       " ('has', 4),\n",
       " ('malfoy', 4),\n",
       " ('retrieved', 4),\n",
       " ('wand', 4),\n",
       " ('magazine', 3),\n",
       " ('film', 3),\n",
       " ('his', 3),\n",
       " ('editor', 3),\n",
       " ('classsearchmatchlovegoodspans', 3),\n",
       " ('that', 3),\n",
       " ('rowling', 3),\n",
       " ('dumbledores', 3),\n",
       " ('been', 3),\n",
       " ('cho', 3),\n",
       " ('chang', 3),\n",
       " ('draco', 3),\n",
       " ('irish', 3),\n",
       " ('him', 3),\n",
       " ('including', 3),\n",
       " ('goblin', 3),\n",
       " ('escape', 3),\n",
       " ('fred', 3),\n",
       " ('2017', 3),\n",
       " ('charlie', 3),\n",
       " ('plays', 3),\n",
       " ('when', 2),\n",
       " ('last', 2),\n",
       " ('original', 2),\n",
       " ('2010', 2),\n",
       " ('wizarding', 2),\n",
       " ('eccentric', 2),\n",
       " ('same', 2),\n",
       " ('during', 2),\n",
       " ('auditioned', 2),\n",
       " ('play', 2),\n",
       " ('them', 2),\n",
       " ('described', 2),\n",
       " ('they', 2),\n",
       " ('army', 2),\n",
       " ('–', 2),\n",
       " ('ravenclaw', 2),\n",
       " ('one', 2),\n",
       " ('year', 2),\n",
       " ('hallows', 2),\n",
       " ('named', 2),\n",
       " ('potters', 2),\n",
       " ('more', 2),\n",
       " ('reveals', 2),\n",
       " ('death', 2),\n",
       " ('eaters', 2),\n",
       " ('similarities', 2),\n",
       " ('matthew', 2),\n",
       " ('lewis', 2),\n",
       " ('louis', 2),\n",
       " ('doyle', 2),\n",
       " ('george', 2),\n",
       " ('portrays', 2),\n",
       " ('discworld', 2),\n",
       " ('however', 2),\n",
       " ('two', 2),\n",
       " ('resident', 2),\n",
       " ('have', 2),\n",
       " ('new', 2),\n",
       " ('halfblood', 2),\n",
       " ('prince', 2),\n",
       " ('uses', 2),\n",
       " ('actor', 2),\n",
       " ('chris', 2),\n",
       " ('which', 2),\n",
       " ('dobby', 2),\n",
       " ('house', 2),\n",
       " ('voldemorts', 2),\n",
       " ('orders', 2),\n",
       " ('after', 2),\n",
       " ('along', 2),\n",
       " ('prisoners', 2),\n",
       " ('mr', 2),\n",
       " ('ollivander', 2),\n",
       " ('dean', 2),\n",
       " ('thomas', 2),\n",
       " ('griphook', 2),\n",
       " ('5', 2),\n",
       " ('quotspan', 2),\n",
       " ('known', 2),\n",
       " ('archived', 2),\n",
       " ('july', 2),\n",
       " ('6', 2),\n",
       " ('uncredited', 2),\n",
       " ('arielle', 2),\n",
       " ('goldman', 2),\n",
       " ('lily', 2),\n",
       " ('hedwig', 2),\n",
       " ('brian', 2),\n",
       " ('rosenthal', 2),\n",
       " ('seamus', 2),\n",
       " ('finnigan', 2),\n",
       " ('james', 2),\n",
       " ('past', 2),\n",
       " ('lone', 2),\n",
       " ('ranger', 2),\n",
       " ('or', 2),\n",
       " ('editorinchief', 1),\n",
       " ('strange', 1),\n",
       " ('claiming', 1),\n",
       " ('supposedly', 1),\n",
       " ('untrue', 1),\n",
       " ('things', 1),\n",
       " ('believe', 1),\n",
       " ('podcast', 1),\n",
       " ('host', 1),\n",
       " ('narrator', 1),\n",
       " ('rose', 1),\n",
       " ('prominence', 1),\n",
       " ('portrayal', 1),\n",
       " ('appearing', 1),\n",
       " ('four', 1),\n",
       " ('semihumans', 1),\n",
       " ('greatly', 1),\n",
       " ('attractive', 1),\n",
       " ('wood', 1),\n",
       " ('nymphs', 1),\n",
       " ('pupil', 1),\n",
       " ('profess', 1),\n",
       " ('belief', 1),\n",
       " ('number', 1),\n",
       " ('magical', 1),\n",
       " ('creatures', 1),\n",
       " ('shrek', 1),\n",
       " ('super', 1),\n",
       " ('slam', 1),\n",
       " ('maximoff', 1),\n",
       " ('supporting', 1),\n",
       " ('marvel', 1),\n",
       " ('comics', 1),\n",
       " ('universe', 1),\n",
       " ('mothews', 1),\n",
       " ('teenage', 1),\n",
       " ('company', 1),\n",
       " ('reprise', 1),\n",
       " ('script', 1),\n",
       " ('xenophillius', 1),\n",
       " ('interview', 1),\n",
       " ('miniserial', 1),\n",
       " ('proof', 1),\n",
       " ('time', 1),\n",
       " ('ronan', 1),\n",
       " ('eventually', 1),\n",
       " ('escorting', 1),\n",
       " ('express', 1),\n",
       " ('pandora', 1),\n",
       " ('husband', 1),\n",
       " ('attacks', 1),\n",
       " ('insults', 1),\n",
       " ('fly', 1),\n",
       " ('ministry', 1),\n",
       " ('magic', 1),\n",
       " ('thestrals', 1),\n",
       " ('an', 1),\n",
       " ('only', 1),\n",
       " ('child', 1),\n",
       " ('quotantihermionequot', 1),\n",
       " ('so', 1),\n",
       " ('different', 1),\n",
       " ('hermiones', 1),\n",
       " ('nemesis', 1),\n",
       " ('student', 1),\n",
       " ('below', 1),\n",
       " ('member', 1),\n",
       " ('friends', 1),\n",
       " ('characterized', 1),\n",
       " ('interest', 1),\n",
       " ('quincey', 1),\n",
       " ('atonement', 1),\n",
       " ('called', 1),\n",
       " ('quotimpressivequot', 1),\n",
       " ('though', 1),\n",
       " ('ultimately', 1),\n",
       " ('bonnie', 1),\n",
       " ('wright', 1),\n",
       " ('katie', 1),\n",
       " ('leung', 1),\n",
       " ('scarlett', 1),\n",
       " ('byrne', 1),\n",
       " ('pansy', 1),\n",
       " ('parkinson', 1),\n",
       " ('azad', 1),\n",
       " ('shefali', 1),\n",
       " ('chowdhury', 1),\n",
       " ('hoppy', 1),\n",
       " ('milly', 1),\n",
       " ('mauler', 1),\n",
       " ('had', 1),\n",
       " ('grandson', 1),\n",
       " ('rolf', 1),\n",
       " ('married', 1),\n",
       " ('appeared', 1),\n",
       " ('three', 1),\n",
       " ('novels', 1),\n",
       " ('some', 1),\n",
       " ('trio', 1),\n",
       " ('awkwardly', 1),\n",
       " ('attempt', 1),\n",
       " ('leave', 1),\n",
       " ('stopped', 1),\n",
       " ('kidnapped', 1),\n",
       " ('before', 1),\n",
       " ('summoning', 1),\n",
       " ('intending', 1),\n",
       " ('striking', 1),\n",
       " ('between', 1),\n",
       " ('emily', 1),\n",
       " ('former', 1),\n",
       " ('there', 1),\n",
       " ('lot', 1),\n",
       " ('i', 1),\n",
       " ('think', 1),\n",
       " ('laughland', 1),\n",
       " ('lee', 1),\n",
       " ('jordan', 1),\n",
       " ('luke', 1),\n",
       " ('youngblood', 1),\n",
       " ('ernie', 1),\n",
       " ('macmillan', 1),\n",
       " ('hotz', 1),\n",
       " ('hacker', 1),\n",
       " ('founder', 1),\n",
       " ('commaai', 1),\n",
       " ('movies', 1),\n",
       " ('attended', 1),\n",
       " ('centre', 1),\n",
       " ('talented', 1),\n",
       " ('youth', 1),\n",
       " ('alice', 1),\n",
       " ('augusta', 1),\n",
       " ('brenda', 1),\n",
       " ('loveknot', 1),\n",
       " ('lythande', 1),\n",
       " ('thieves', 1),\n",
       " ('world', 1),\n",
       " ('love', 1),\n",
       " ('shoots', 1),\n",
       " ('down', 1),\n",
       " ('meets', 1),\n",
       " ('bond', 1),\n",
       " ('over', 1),\n",
       " ('pigfarts', 1),\n",
       " ('school', 1),\n",
       " ('mars', 1),\n",
       " ('starred', 1),\n",
       " ('arthur', 1),\n",
       " ('mathews', 1),\n",
       " ('cowriter', 1),\n",
       " ('sitcom', 1),\n",
       " ('ted', 1),\n",
       " ('peter', 1),\n",
       " ('these', 1),\n",
       " ('sets', 1),\n",
       " ('newly', 1),\n",
       " ('decorated', 1),\n",
       " ('minifigures', 1),\n",
       " ('animals', 1),\n",
       " ('bellatrix', 1),\n",
       " ('lestrange', 1),\n",
       " ('owls', 1),\n",
       " ('pigs', 1),\n",
       " ('also', 1),\n",
       " ('announced', 1),\n",
       " ('lego', 1),\n",
       " ('horrid', 1),\n",
       " ('henry', 1),\n",
       " ('even', 1),\n",
       " ('left', 1),\n",
       " ('stranded', 1),\n",
       " ('train', 1),\n",
       " ('until', 1),\n",
       " ('nymphadora', 1),\n",
       " ('tonks', 1),\n",
       " ('adaptation', 1),\n",
       " ('rescues', 1),\n",
       " ('spends', 1),\n",
       " ('much', 1),\n",
       " ('trailing', 1),\n",
       " ('griphooks', 1),\n",
       " ('legs', 1),\n",
       " ('broken', 1),\n",
       " ('manor', 1),\n",
       " ('skelegro', 1),\n",
       " ('repair', 1),\n",
       " ('sleekeazys', 1),\n",
       " ('hair', 1),\n",
       " ('potion', 1),\n",
       " ('beauty', 1),\n",
       " ('treatment', 1),\n",
       " ('alongside', 1),\n",
       " ('department', 1),\n",
       " ('mysteries', 1),\n",
       " ('injured', 1),\n",
       " ('fight', 1),\n",
       " ('makes', 1),\n",
       " ('jamie', 1),\n",
       " ('waylett', 1),\n",
       " ('vincent', 1),\n",
       " ('crabbe', 1),\n",
       " ('rupert', 1),\n",
       " ('grint', 1),\n",
       " ('quotfiendish', 1),\n",
       " ('gleequot', 1),\n",
       " ('writing', 1),\n",
       " ('this', 1),\n",
       " ('scene', 1),\n",
       " ('features', 1),\n",
       " ('memorable', 1),\n",
       " ('commentary', 1),\n",
       " ('2014', 1),\n",
       " ('started', 1),\n",
       " ('publishing', 1),\n",
       " ('match', 1),\n",
       " ('reports', 1),\n",
       " ('struggling', 1),\n",
       " ('find', 1),\n",
       " ('place', 1),\n",
       " ('practise', 1),\n",
       " ('elf', 1),\n",
       " ('tells', 1),\n",
       " ('preparing', 1),\n",
       " ('capture', 1),\n",
       " ('stunned', 1),\n",
       " ('touching', 1),\n",
       " ('dark', 1),\n",
       " ('mark', 1),\n",
       " ('summon', 1),\n",
       " ('voldemort', 1),\n",
       " ('amycus', 1),\n",
       " ('seeing', 1),\n",
       " ('2016', 1),\n",
       " ('announcement', 1),\n",
       " ('featured', 1),\n",
       " ('soft', 1),\n",
       " ('opening', 1),\n",
       " ('march', 1),\n",
       " ('31', 1),\n",
       " ('help', 1),\n",
       " ('houseelf', 1),\n",
       " ('with', 1),\n",
       " ('fellow', 1),\n",
       " ('being', 1),\n",
       " ('kept', 1),\n",
       " ('basement', 1),\n",
       " ('snatchers', 1),\n",
       " ('spotted', 1),\n",
       " ('petrified', 1),\n",
       " ('found', 1),\n",
       " ('saved', 1),\n",
       " ('borrow', 1),\n",
       " ('textbooks', 1),\n",
       " ('slughorns', 1),\n",
       " ('potions', 1),\n",
       " ('magicquot', 1),\n",
       " ('pottermore', 1),\n",
       " ('may', 1),\n",
       " ('j', 1),\n",
       " ('k', 1),\n",
       " ('2003', 1),\n",
       " ('classsearchmatchlovegoodspanquot', 1),\n",
       " ('bloomsbury', 1),\n",
       " ('isbn', 1),\n",
       " ('0747551006', 1),\n",
       " ('january', 1),\n",
       " ('2011', 1),\n",
       " ('andy', 1),\n",
       " ('october', 1),\n",
       " ('quotevanna', 1),\n",
       " ('talks', 1),\n",
       " ('set', 1),\n",
       " ('deathly', 1),\n",
       " ('hallowsquot', 1),\n",
       " ('fan', 1),\n",
       " ('zone', 1),\n",
       " ('teammates', 1),\n",
       " ('way', 1),\n",
       " ('dawn', 1),\n",
       " ('mostly', 1),\n",
       " ('based', 1),\n",
       " ('voiced', 1),\n",
       " ('caitlynne', 1),\n",
       " ('medrek', 1),\n",
       " ('jo', 1),\n",
       " ('10', 1),\n",
       " ('castquot', 1),\n",
       " ('cbbc', 1),\n",
       " ('newsround', 1),\n",
       " ('2', 1),\n",
       " ('february', 1),\n",
       " ('2006', 1),\n",
       " ('duel', 1),\n",
       " ('mcgonagall', 1),\n",
       " ('gathers', 1),\n",
       " ('community', 1),\n",
       " ('battle', 1),\n",
       " ('insistence', 1),\n",
       " ('speaks', 1),\n",
       " ('helena', 1),\n",
       " ('ravenclaws', 1),\n",
       " ('ghost', 1),\n",
       " ('das', 1),\n",
       " ('coins', 1),\n",
       " ('alert', 1),\n",
       " ('people', 1),\n",
       " ('such', 1),\n",
       " ('returned', 1),\n",
       " ('scabbers', 1),\n",
       " ('4', 1),\n",
       " ('golden', 1),\n",
       " ('snitch', 1),\n",
       " ('broomstick', 1),\n",
       " ('2x3', 1),\n",
       " ('mandrake', 1),\n",
       " ('cauldron', 1),\n",
       " ('chance', 1),\n",
       " ('max', 1),\n",
       " ('suite', 1),\n",
       " ('life', 1),\n",
       " ('zack', 1),\n",
       " ('amp', 1),\n",
       " ('cody', 1),\n",
       " ('bridgette', 1),\n",
       " ('dubois', 1),\n",
       " ('medium', 1),\n",
       " ('violet', 1),\n",
       " ('beauregarde', 1),\n",
       " ('chocolate', 1),\n",
       " ('factory', 1),\n",
       " ('jaquish', 1),\n",
       " ('wrote', 1),\n",
       " ('additional', 1),\n",
       " ('scenes', 1),\n",
       " ('added', 1),\n",
       " ('peeves', 1),\n",
       " ('moaning', 1),\n",
       " ('myrtle', 1),\n",
       " ('kitchen', 1),\n",
       " ('elves', 1),\n",
       " ('show', 1),\n",
       " ('first', 1),\n",
       " ('firehouse', 1),\n",
       " ('episode', 1),\n",
       " ('69', 1),\n",
       " ('rankin', 1),\n",
       " ('percy', 1),\n",
       " ('crew', 1),\n",
       " ('members', 1),\n",
       " ('quotstick', 1),\n",
       " ('it', 1),\n",
       " ('doloresquot23210quotspewquot10111quotthe', 1),\n",
       " ('human', 1),\n",
       " ('hosepipequot33312quotspan', 1),\n",
       " ('okquot22013quotthe', 1),\n",
       " ('godfather', 1),\n",
       " ('part', 1),\n",
       " ('iiquot22514quotthe', 1),\n",
       " ('weaponquot427total', 1),\n",
       " ('length4158', 1),\n",
       " ('silk', 1),\n",
       " ('spectre', 1),\n",
       " ('shirley', 1),\n",
       " ('manson', 1),\n",
       " ('terminator', 1),\n",
       " ('sarah', 1),\n",
       " ('connor', 1),\n",
       " ('chronicles', 1),\n",
       " ('catherine', 1),\n",
       " ('chile', 1),\n",
       " ('born', 1),\n",
       " ('1991', 1),\n",
       " ('fionán', 1),\n",
       " ('politician', 1),\n",
       " ('genevieve', 1),\n",
       " ('springston', 1),\n",
       " ('aufashionwrapvrawlivestyle2010', 1),\n",
       " ('httpsbowienewswordpresscom20120604fromspan', 1),\n",
       " ('classsearchmatchlunaspanspan', 1),\n",
       " ('classsearchmatchlovegoodspantolookinglovelyinbowie', 1),\n",
       " ('httpwwwstylebistro', 1),\n",
       " ('eleanor', 1),\n",
       " ('morton', 1),\n",
       " ('weasleymolly', 1),\n",
       " ('matt', 1),\n",
       " ('tedford', 1),\n",
       " ('weasleyspan', 1),\n",
       " ('marny', 1),\n",
       " ('godden', 1),\n",
       " ('grangeruncle', 1),\n",
       " ('albert', 1),\n",
       " ('jon', 1),\n",
       " ('brittain', 1),\n",
       " ('hagrid', 1),\n",
       " ('comic', 1),\n",
       " ('strip', 1),\n",
       " ('2009', 1),\n",
       " ('2013', 1),\n",
       " ('lyra', 1),\n",
       " ('2005', 1),\n",
       " ('harpist', 1),\n",
       " ('harp', 1),\n",
       " ('noted', 1),\n",
       " ('cleverness', 1),\n",
       " ('creativity', 1),\n",
       " ('beautiful', 1),\n",
       " ('sorting', 1),\n",
       " ('hat', 1),\n",
       " ('introduces', 1),\n",
       " ('quotfair', 1),\n",
       " ('glenquot', 1),\n",
       " ('tinworth', 1),\n",
       " ('cornwall', 1),\n",
       " ('allwizard', 1),\n",
       " ('diggory', 1),\n",
       " ('fawcett', 1),\n",
       " ('families', 1),\n",
       " ('live', 1),\n",
       " ('near', 1),\n",
       " ('muggle', 1),\n",
       " ('village', 1),\n",
       " ('ottery', 1),\n",
       " ('st', 1),\n",
       " ('catchpole', 1)]"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def sortbycount(pair):\n",
    "    return pair[1]\n",
    "\n",
    "pairs = list(wordcount.items())\n",
    "pairs.sort(key = sortbycount, reverse = True)\n",
    "pairs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So maybe our interests have shifted and we want to create a data file that we can open up in excel and play around with.  We will need to create a CSV file for this.\n",
    "\n",
    "# Creating CSV files\n",
    "\n",
    "Now let's switch gears and think about reconfiguring this data into another format.  JSON is useful, but not something that we can easily open and inspect. Let's get this file into a rectangular format, like something that we can open in excel.\n",
    "\n",
    "To do this, we need to change these records into things that resemble rows we might see in a spreadsheet.  These rows are stored as a list, with each element in the list representing the individual cell values within that row.  This means you're going to do a double list accumulator pattern. The outside accumulator will hold all the lists that represent the rows. The for loop will iterate over the records in your data and access all the desired values and assemble the lists for each row.  You will only need a single for loop here, as you can extract the values from the record dictionary directly without looping over the values.\n",
    "\n",
    "We need to start with our desired data values.\n",
    "\n",
    "Let's say we want the following headers for our csv:\n",
    "\n",
    "pageid, snippetwordcount\n",
    "\n",
    "Two of these values are coming in directly from our dictionaries, and one we can compute on the snippet during a loop.\n",
    "\n",
    "Here's some psuedocode for how to do this:\n",
    "\n",
    "```\n",
    "# set your primary list accumulator base, called allrows, to be an empty list\n",
    "\n",
    "# loop over all the data records (indented comments go in your for loop)\n",
    "\n",
    "    # set your row accumulator base to an empty list (call the variable row)\n",
    "\n",
    "    # extract all the values you need from each record dict, saving them to variable\n",
    "    # names that match the content. \n",
    "\n",
    "    # use append to add each value you've extracted into the row accumulator\n",
    "\n",
    "    # add an assert statement that the length of row should be 4. \n",
    "    # Set the error message to say \"The length of the row list isn't 4\".\n",
    "\n",
    "    # at the very end, append that row to your allrows list\n",
    "```\n",
    "\n",
    "I'm also going to move the word count stuff into a function, so we don't clutter up out for loop."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import string\n",
    "\n",
    "def clean_line(line_str):\n",
    "    cleanline = line_str\n",
    "    for punc in string.punctuation:\n",
    "        cleanline = cleanline.replace(punc, \"\")\n",
    "    return cleanline\n",
    "\n",
    "def count_words(bigstring):\n",
    "    clean_text = clean_line(bigstring).lower()\n",
    "    words = clean_text.split()\n",
    "    return len(words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can loop through out records and grab the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "allrows = []\n",
    "\n",
    "for record in records:\n",
    "    row = []\n",
    "    pageid = record['pageid']\n",
    "    snippetcount = count_words(record['snippet'])\n",
    "    row.append(pageid)\n",
    "    row.append(snippetcount)\n",
    "    assert len(row) == 2, \"The length of the row list isn't 2\"\n",
    "    allrows.append(row)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's write an assert statement that our results list is 55 items long."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "assert len(allrows) == 55, \"The len isn't 55, you're missing data!\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[391371, 26],\n",
       " [3931304, 29],\n",
       " [874930, 27],\n",
       " [23483380, 27],\n",
       " [38290297, 29],\n",
       " [891112, 27],\n",
       " [11061022, 28],\n",
       " [3512033, 27],\n",
       " [670407, 26],\n",
       " [45100, 26],\n",
       " [667601, 30],\n",
       " [827110, 26],\n",
       " [10093585, 28],\n",
       " [2298654, 24],\n",
       " [241855, 28],\n",
       " [9834441, 28],\n",
       " [48543224, 27],\n",
       " [771537, 23],\n",
       " [271050, 27],\n",
       " [855389, 21],\n",
       " [32402649, 29],\n",
       " [3833378, 26],\n",
       " [5032848, 25],\n",
       " [34085574, 28],\n",
       " [44334, 29],\n",
       " [21419143, 27],\n",
       " [19156567, 26],\n",
       " [906492, 23],\n",
       " [59572, 26],\n",
       " [156489, 27],\n",
       " [156587, 26],\n",
       " [33976749, 27],\n",
       " [776049, 26],\n",
       " [667658, 25],\n",
       " [858575, 27],\n",
       " [284598, 26],\n",
       " [17763993, 27],\n",
       " [28373862, 27],\n",
       " [54465045, 27],\n",
       " [31941988, 24],\n",
       " [779883, 28],\n",
       " [27511982, 25],\n",
       " [32446375, 29],\n",
       " [12976982, 27],\n",
       " [3381404, 25],\n",
       " [12294600, 16],\n",
       " [24870495, 24],\n",
       " [20997070, 27],\n",
       " [29494722, 23],\n",
       " [29960484, 5],\n",
       " [45256977, 20],\n",
       " [43136825, 27],\n",
       " [947583, 25],\n",
       " [29494707, 23],\n",
       " [2228434, 24]]"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "allrows"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have the list of rows, we can go through the process of writing out the CSV.\n",
    "\n",
    "First, you need to make a list that contains the headers you want to appear in your file.  Each header should be a string with the text value.  Your list should be the same length as the internal lists for each row (so 2 in this case.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "headers = ['pageid', 'wordcount']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can go through our boilerplate code for writing out the CSV.  This uses a different structure for opening a fileio object, but you still will see an outfile object.  Importantly, there's no need to close the outfile because this pattern takes care of that for you.\n",
    "\n",
    "``` python\n",
    "import csv\n",
    "\n",
    "with open('youroutfilename.csv', 'w', newline = '') as outfile:\n",
    "    csvout = csv.writer(outfile)\n",
    "    csvout.writerow(headers)\n",
    "    csvout.writerows(allrows)\n",
    "```\n",
    "\n",
    "First we import the csv module.  This gives us specific functions for parsing CSVs, including writing them.\n",
    "\n",
    "Then, we are using the with block keyword to open our file.  This structure will automatically close the file we have opened as soon as the with block is exited from.\n",
    "\n",
    "This is one that you won't really understand right now, just note the things that you'll need to change.  Let's finish walking through it and then I'll tell you what you need to change.\n",
    "\n",
    "There are three lines of code within this with statement.  We do have a regular outfile object, just opened differently.  This fileio object is passed into the `csv.writer()` function.  That function transforms the fileio object into a csvwriter object.  This is a special kind of io object that knows how to write csv data via special methods.  This is all you need to understand about how this is working.\n",
    "\n",
    "The `newline = ''` bit is necessary for windows machines to create the file correctly, but is benign on macs.  Otherwise, csvs created this way an a windows machine will have extra lines.\n",
    "\n",
    "The next two lines are demonstrations of these special methods.  Each requires a different kind of data structure.\n",
    "\n",
    "`.writerow(headers)` takes a single list of strings.  That row will be written as one line of the file, with the elements within it written separated by commas.  \n",
    "\n",
    "`.writerows(allrows)` takes a list of lists.  Each of those sublists needs to have strings, and the length of each list need to be the same.  This will write out each sublist as a separate row, with each string in that list separated by a comma.\n",
    "\n",
    "What you need to provide?\n",
    "\n",
    "* the file name in the `open()` function\n",
    "* the name of the outfile variable you want (outfile is usually fine)\n",
    "* the csvout variable you want (usually csvoutfile is fine)\n",
    "* the variable name with the headers list goes within `.writerow()`\n",
    "* the variable name with the list of lists goes within `.writerows()`.\n",
    "\n",
    "And that's it!  Execute the code and look at your file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "import csv\n",
    "\n",
    "headers = ['pageid', 'snippetcount']\n",
    "\n",
    "with open('lunasnippetcount.csv', 'w', newline = '') as outfile:\n",
    "    csvout = csv.writer(outfile)\n",
    "    csvout.writerow(headers)\n",
    "    csvout.writerows(allrows)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can see that the pageid of 29960484 has the shortest snippet.  Let's find that in the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'ns': 0, 'title': 'Bowie Wong', 'pageid': 29960484, 'size': 6402, 'wordcount': 768, 'snippet': 'au/fashion/wrap/vraw_livestyle2010 https://bowienews.wordpress.com/2012/06/04/from-<span class=\"searchmatch\">luna</span>-<span class=\"searchmatch\">lovegood</span>-to-looking-lovely-in-bowie/ http://www.stylebistro', 'timestamp': '2018-10-06T15:03:49Z'}\n"
     ]
    }
   ],
   "source": [
    "for entry in records:\n",
    "    if entry['pageid'] == 29960484: # these are stored as ints in the dicts\n",
    "        print(entry)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Reading in csvs\n",
    "\n",
    "\n",
    "We're roughly going to repeat the process of writing things out, creating really similar objects that we started with.  We'll read in the headers into a single list, then read in all the data rows as a list of lists.  You'll note that the boilerplate is pretty similar, but all the things have been flipped for file reading, and we have assignment statements along with our read methods.\n",
    "\n",
    "You'll also see that we have a list accumulator happening to collect all the lines.  This is actually different from the standard boilerplate, as that would use something that you've never really seen before.\n",
    "\n",
    "Here's the boilerplate.\n",
    "\n",
    "``` python\n",
    "import csv\n",
    "\n",
    "allrows = []\n",
    "\n",
    "with open('yourexistingfilename.csv', 'r', newline = '') as infile:\n",
    "    csvin = csv.reader(infile)\n",
    "    headers = next(csvin)\n",
    "    for line in csvin:\n",
    "        allrows.append(line)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import csv\n",
    "\n",
    "lunadata = []\n",
    "\n",
    "with open('lunasnippetcount.csv', 'r', newline = '') as infile:\n",
    "    csvin = csv.reader(infile)\n",
    "    headers = next(csvin)\n",
    "    for line in csvin:\n",
    "        lunadata.append(line)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can inspect our structures and see what we have."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['pageid', 'snippetcount']"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "headers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "55\n"
     ]
    }
   ],
   "source": [
    "print(len(lunadata))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Now we can us all our normal stuff for interacting with this to get out data and look at values.  Let's use this structure to sort our values so we can see the pageids with the pagecount.  We're going to use the sorting method that we learned last week."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['12294600', '16'],\n",
       " ['45256977', '20'],\n",
       " ['855389', '21'],\n",
       " ['771537', '23'],\n",
       " ['906492', '23'],\n",
       " ['29494722', '23'],\n",
       " ['29494707', '23'],\n",
       " ['2298654', '24'],\n",
       " ['31941988', '24'],\n",
       " ['24870495', '24'],\n",
       " ['2228434', '24'],\n",
       " ['5032848', '25'],\n",
       " ['667658', '25'],\n",
       " ['27511982', '25'],\n",
       " ['3381404', '25'],\n",
       " ['947583', '25'],\n",
       " ['391371', '26'],\n",
       " ['670407', '26'],\n",
       " ['45100', '26'],\n",
       " ['827110', '26'],\n",
       " ['3833378', '26'],\n",
       " ['19156567', '26'],\n",
       " ['59572', '26'],\n",
       " ['156587', '26'],\n",
       " ['776049', '26'],\n",
       " ['284598', '26'],\n",
       " ['874930', '27'],\n",
       " ['23483380', '27'],\n",
       " ['891112', '27'],\n",
       " ['3512033', '27'],\n",
       " ['48543224', '27'],\n",
       " ['271050', '27'],\n",
       " ['21419143', '27'],\n",
       " ['156489', '27'],\n",
       " ['33976749', '27'],\n",
       " ['858575', '27'],\n",
       " ['17763993', '27'],\n",
       " ['28373862', '27'],\n",
       " ['54465045', '27'],\n",
       " ['12976982', '27'],\n",
       " ['20997070', '27'],\n",
       " ['43136825', '27'],\n",
       " ['11061022', '28'],\n",
       " ['10093585', '28'],\n",
       " ['241855', '28'],\n",
       " ['9834441', '28'],\n",
       " ['34085574', '28'],\n",
       " ['779883', '28'],\n",
       " ['3931304', '29'],\n",
       " ['38290297', '29'],\n",
       " ['32402649', '29'],\n",
       " ['44334', '29'],\n",
       " ['32446375', '29'],\n",
       " ['667601', '30'],\n",
       " ['29960484', '5']]"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def getcount(row):\n",
    "    return row[1]\n",
    "\n",
    "lunadata.sort(key = getcount)\n",
    "\n",
    "lunadata"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That didn't seem to do the trick.  What happened?  Look at how this is sorting.  This is sorting by ascii value and not integer.  We don't actually need to change the values in our data to fix this, we can return an int value in the getcount function to have it sort by the int value.  Let's make that change and take a look."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['29960484', '5'],\n",
       " ['12294600', '16'],\n",
       " ['45256977', '20'],\n",
       " ['855389', '21'],\n",
       " ['771537', '23'],\n",
       " ['906492', '23'],\n",
       " ['29494722', '23'],\n",
       " ['29494707', '23'],\n",
       " ['2298654', '24'],\n",
       " ['31941988', '24'],\n",
       " ['24870495', '24'],\n",
       " ['2228434', '24'],\n",
       " ['5032848', '25'],\n",
       " ['667658', '25'],\n",
       " ['27511982', '25'],\n",
       " ['3381404', '25'],\n",
       " ['947583', '25'],\n",
       " ['391371', '26'],\n",
       " ['670407', '26'],\n",
       " ['45100', '26'],\n",
       " ['827110', '26'],\n",
       " ['3833378', '26'],\n",
       " ['19156567', '26'],\n",
       " ['59572', '26'],\n",
       " ['156587', '26'],\n",
       " ['776049', '26'],\n",
       " ['284598', '26'],\n",
       " ['874930', '27'],\n",
       " ['23483380', '27'],\n",
       " ['891112', '27'],\n",
       " ['3512033', '27'],\n",
       " ['48543224', '27'],\n",
       " ['271050', '27'],\n",
       " ['21419143', '27'],\n",
       " ['156489', '27'],\n",
       " ['33976749', '27'],\n",
       " ['858575', '27'],\n",
       " ['17763993', '27'],\n",
       " ['28373862', '27'],\n",
       " ['54465045', '27'],\n",
       " ['12976982', '27'],\n",
       " ['20997070', '27'],\n",
       " ['43136825', '27'],\n",
       " ['11061022', '28'],\n",
       " ['10093585', '28'],\n",
       " ['241855', '28'],\n",
       " ['9834441', '28'],\n",
       " ['34085574', '28'],\n",
       " ['779883', '28'],\n",
       " ['3931304', '29'],\n",
       " ['38290297', '29'],\n",
       " ['32402649', '29'],\n",
       " ['44334', '29'],\n",
       " ['32446375', '29'],\n",
       " ['667601', '30']]"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def getcount(row):\n",
    "    return int(row[1])\n",
    "\n",
    "lunadata.sort(key = getcount)\n",
    "\n",
    "lunadata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}