scripts_&_analysis/1_lyricsgenius_requests.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Downloading Lyrical Discographies from Genius Lyrics\n",
    "\n",
    "Taking inspiration from [Alexus Brown's 2021 Project](https://github.com/Data-Science-for-Linguists-2021/Rapper_Topic_Modeling), I'm using the `lyricsgenius` package tutorial by [John W Miller](https://github.com/johnwmillr/LyricsGenius).  This file has been separated out so that API access won't be reiterated multiple times."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from lyricsgenius import Genius\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Access Token has been redacted.  Genius API account required.\n",
    "genius = Genius(\"XXXXXXXXXXXXXXXXXXXXX\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "genius.remove_section_headers = True\n",
    "genius.verbose = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test Run with Crispin Glover's Discography"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# request Crispin Glover's full discography\n",
    "artist = genius.search_artist(\"Crispin Glover\", sort=\"title\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Clowny Clown Clown LyricsI was walking on the ground\n",
      "I didn't make a sound\n",
      "Then, I turned around\n",
      "And I saw a clown\n",
      "Had a frown\n",
      "Stood on a mound\n",
      "Started barking like a hound\n",
      "Huh her ha hur ur\n",
      "Clowny clown clown\n",
      "When I came to what I found\n",
      "He showed me something that was brown\n",
      "So we became great friends and\n",
      "Late in life, he got sick...\n",
      "I gave him some soup, but he got\n",
      "Worse and asked for its purse\n",
      "It got it, but it was empty\n",
      "So it cried a plenty\n",
      "I wondered what to do\n",
      "I didn't know what to think\n",
      "So I got a drink\n",
      "And then I showed it\n",
      "Something that was round\n",
      "And it died, smiled\n",
      "And fell on the ground\n",
      "Clowny clown clown\n",
      "Thinking back about those days with the clown\n",
      "I get teary-eyed and really snide\n",
      "I think that deep down\n",
      "I hated that clown\n",
      "But not as much as Mr. Farr\n",
      "I'm going to go smoke a cigar\n",
      "Clowny clown clown\n",
      "I was walking on the ground\n",
      "I didn't make a sound\n",
      "And then I turned around\n",
      "I saw a clown\n",
      "Clown\n",
      "Clowny Clown clown\n",
      "Ha ha!\n",
      "I hate you clown\n",
      "Your ugly frown\n",
      "Smiley lips\n",
      "Think I'll clip you across the nose, clown\n",
      "Ha ha!\n",
      "Want a cigar?\n",
      "It'll get you real far ha ha\n",
      "Like Mr. Farr, get it?\n",
      "Ha ha Mr. Farr?\n",
      "Clown\n",
      "Clown\n",
      "(coughs) See what a cigar'll do? He he!\n",
      "Clown\n",
      "Ugly Clown1Embed\n"
     ]
    }
   ],
   "source": [
    "# example lyrics: Crispin Glover's \"Clowny Clown Clown\"\n",
    "print(artist.song(\"Clowny Clown Clown\").lyrics)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Lyrics_CrispinGlover.json already exists. Overwrite?\n",
      "(y/n): y\n",
      "Wrote Lyrics_CrispinGlover.json.\n"
     ]
    }
   ],
   "source": [
    "# save the lyrics to a JSON file\n",
    "artist.save_lyrics()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# test reopening the JSON file\n",
    "with open(\"Lyrics_CrispinGlover.json\") as file:\n",
    "    cgFileLoad = json.load(file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "## This would show the full 'songs' value for Crispin Glover\n",
    "# cgFileLoad['songs']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a dataframe that contains Crispin Glover's songs and their full data\n",
    "testdf = pd.DataFrame(cgFileLoad['songs'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "# reduce the dataframe to just the necessary identifying information and the lyrics\n",
    "testdf = testdf[['artist', 'title', 'lyrics']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>artist</th>\n",
       "      <th>title</th>\n",
       "      <th>lyrics</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>Crispin Glover</td>\n",
       "      <td>Auto-Manipulator</td>\n",
       "      <td>Auto-Manipulator Lyrics\\nWomen are sweet\\nAnd ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>Crispin Glover</td>\n",
       "      <td>Clowny Clown Clown</td>\n",
       "      <td>Clowny Clown Clown LyricsI was walking on the ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>Crispin Glover</td>\n",
       "      <td>Never Say “Never” To Always</td>\n",
       "      <td>Never Say “Never” To Always LyricsAlways is al...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>Crispin Glover</td>\n",
       "      <td>New Clean Song</td>\n",
       "      <td>New Clean Song Lyrics\\nYesterday I had my birt...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>Crispin Glover</td>\n",
       "      <td>Overture</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>Crispin Glover</td>\n",
       "      <td>Selected Readings From Oak Mot Part I</td>\n",
       "      <td>Selected Readings From Oak Mot Part I LyricsOa...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>Crispin Glover</td>\n",
       "      <td>Selected Readings From Oak Mot Part II</td>\n",
       "      <td>Selected Readings From Oak Mot Part II Lyrics\"...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>Crispin Glover</td>\n",
       "      <td>Selected Readings From Oak Mot Part III</td>\n",
       "      <td>Selected Readings From Oak Mot Part III Lyrics...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>Crispin Glover</td>\n",
       "      <td>Selected Readings From Oak Mot Part IV</td>\n",
       "      <td>Selected Readings From Oak Mot Part IV Lyrics“...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>Crispin Glover</td>\n",
       "      <td>Selected Readings From Rat Catching</td>\n",
       "      <td>Selected Readings From Rat Catching LyricsRat ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>Crispin Glover</td>\n",
       "      <td>The Daring Young Man On The Flying Trapeze</td>\n",
       "      <td>The Daring Young Man On The Flying Trapeze Lyr...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>Crispin Glover</td>\n",
       "      <td>These Boots Are Made For Walking</td>\n",
       "      <td>These Boots Are Made For Walking Lyrics\\nAlway...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>Crispin Glover</td>\n",
       "      <td>Untitled Bonus Track 1</td>\n",
       "      <td>Untitled Bonus Track 1 LyricsAll I had to do w...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>Crispin Glover</td>\n",
       "      <td>Untitled Bonus Track 2</td>\n",
       "      <td>Untitled Bonus Track 2 LyricsChapter 13: In wh...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>Crispin Glover</td>\n",
       "      <td>Untitled Bonus Track 3</td>\n",
       "      <td>Untitled Bonus Track 3 LyricsO Fortuna velut l...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            artist                                       title  \\\n",
       "0   Crispin Glover                            Auto-Manipulator   \n",
       "1   Crispin Glover                          Clowny Clown Clown   \n",
       "2   Crispin Glover                 Never Say “Never” To Always   \n",
       "3   Crispin Glover                              New Clean Song   \n",
       "4   Crispin Glover                                    Overture   \n",
       "5   Crispin Glover       Selected Readings From Oak Mot Part I   \n",
       "6   Crispin Glover      Selected Readings From Oak Mot Part II   \n",
       "7   Crispin Glover     Selected Readings From Oak Mot Part III   \n",
       "8   Crispin Glover      Selected Readings From Oak Mot Part IV   \n",
       "9   Crispin Glover         Selected Readings From Rat Catching   \n",
       "10  Crispin Glover  The Daring Young Man On The Flying Trapeze   \n",
       "11  Crispin Glover            These Boots Are Made For Walking   \n",
       "12  Crispin Glover                      Untitled Bonus Track 1   \n",
       "13  Crispin Glover                      Untitled Bonus Track 2   \n",
       "14  Crispin Glover                      Untitled Bonus Track 3   \n",
       "\n",
       "                                               lyrics  \n",
       "0   Auto-Manipulator Lyrics\\nWomen are sweet\\nAnd ...  \n",
       "1   Clowny Clown Clown LyricsI was walking on the ...  \n",
       "2   Never Say “Never” To Always LyricsAlways is al...  \n",
       "3   New Clean Song Lyrics\\nYesterday I had my birt...  \n",
       "4                                                      \n",
       "5   Selected Readings From Oak Mot Part I LyricsOa...  \n",
       "6   Selected Readings From Oak Mot Part II Lyrics\"...  \n",
       "7   Selected Readings From Oak Mot Part III Lyrics...  \n",
       "8   Selected Readings From Oak Mot Part IV Lyrics“...  \n",
       "9   Selected Readings From Rat Catching LyricsRat ...  \n",
       "10  The Daring Young Man On The Flying Trapeze Lyr...  \n",
       "11  These Boots Are Made For Walking Lyrics\\nAlway...  \n",
       "12  Untitled Bonus Track 1 LyricsAll I had to do w...  \n",
       "13  Untitled Bonus Track 2 LyricsChapter 13: In wh...  \n",
       "14  Untitled Bonus Track 3 LyricsO Fortuna velut l...  "
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# When I clean the data, I'll have to remove lyric-less and \"title Lyrics\"\n",
    "testdf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# initialize empty list\n",
    "musicians = []\n",
    "\n",
    "# load musicians from wiki_musicians.py into a list\n",
    "with open(\"0_wiki_musicians.txt\") as mfile:\n",
    "    for line in mfile:\n",
    "        musicians.append(line.strip('\\n'))\n",
    "mfile.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Hasil Adkins',\n",
       " 'Eden ahbez',\n",
       " 'Ajdar',\n",
       " 'Leona Anderson',\n",
       " 'Brittany Anjou',\n",
       " 'Nathaniel Ayers',\n",
       " 'Syd Barrett',\n",
       " 'Tryphosa Bates-Batcheller',\n",
       " 'Leila Bela',\n",
       " 'The Better Beatles',\n",
       " 'Y. Bhekhirst',\n",
       " 'Button King',\n",
       " 'Captain Beefheart',\n",
       " 'Cherry Sisters',\n",
       " 'Corn Mo']"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Show a bit of the musicians list\n",
    "musicians[:15]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Download the Full Musicians List\n",
    "[This 49 Years of Lyrics Project](https://towardsdatascience.com/49-years-of-lyrics-why-so-angry-1adf0a3fa2b4) used a try-except code block, that I used somewhat similarly below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wrote Lyrics_HasilAdkins.json.\n",
      "Wrote Lyrics_EdenAhbez.json.\n",
      "Wrote Lyrics_Ajdar.json.\n",
      "Wrote Lyrics_LeonaAnderson.json.\n",
      "Wrote Lyrics_BrittanyAnjou.json.\n",
      "Wrote Lyrics_NatAyer.json.\n",
      "Wrote Lyrics_SydBarrett.json.\n",
      "Wrote Lyrics_LeiaBLACKSWAN.json.\n",
      "Wrote Lyrics_CaptainBeefheart.json.\n",
      "Wrote Lyrics_CornMo.json.\n",
      "Wrote Lyrics_LesCompagnonsDeLaChanson.json.\n",
      "Wrote Lyrics_DavidCronenbergsWife.json.\n",
      "Wrote Lyrics_Dr.Demento.json.\n",
      "Wrote Lyrics_DIVINE.json.\n",
      "Wrote Lyrics_ROE.json.\n",
      "Wrote Lyrics_RokyErickson.json.\n",
      "Wrote Lyrics_DamiãoExperiença.json.\n",
      "Wrote Lyrics_ExtraditionOrder.json.\n",
      "Wrote Lyrics_JadFair.json.\n",
      "Wrote Lyrics_SteveFarnie.json.\n",
      "Wrote Lyrics_WildManFischer.json.\n",
      "Wrote Lyrics_JohnFrusciante.json.\n",
      "Lyrics_CrispinGlover.json already exists. Overwrite?\n",
      "(y/n): y\n",
      "Wrote Lyrics_CrispinGlover.json.\n",
      "Wrote Lyrics_MarkGormley.json.\n",
      "Wrote Lyrics_RobertGraettinger.json.\n",
      "Wrote Lyrics_PeterGrudzien.json.\n",
      "Wrote Lyrics_LeslieHall.json.\n",
      "Wrote Lyrics_NaomiHale.json.\n",
      "Wrote Lyrics_DavidLiebeHart.json.\n",
      "Wrote Lyrics_PaulHawkins.json.\n",
      "Wrote Lyrics_JohnHinckleyJr..json.\n",
      "Wrote Lyrics_WilliamHung.json.\n",
      "Wrote Lyrics_IceJJFish.json.\n",
      "Wrote Lyrics_BobbyJameson.json.\n",
      "Wrote Lyrics_Jandek.json.\n",
      "Wrote Lyrics_AbnerJay.json.\n",
      "Wrote Lyrics_DanielJohnston.json.\n",
      "Wrote Lyrics_TheKidsofWidneyHigh.json.\n",
      "Wrote Lyrics_LegendaryStardustCowboy.json.\n",
      "Wrote Lyrics_LeslieandtheLYs.json.\n",
      "Wrote Lyrics_LilPeep.json.\n",
      "Wrote Lyrics_VennuMallesh.json.\n",
      "Wrote Lyrics_CharlesManson.json.\n",
      "Wrote Lyrics_MiguelitoAngelMasso.json.\n",
      "Wrote Lyrics_JohnMaus.json.\n",
      "Wrote Lyrics_Mrs.Miller.json.\n",
      "Wrote Lyrics_SextonMing.json.\n",
      "Wrote Lyrics_Moondog.json.\n",
      "Wrote Lyrics_R.StevieMoore.json.\n",
      "Wrote Lyrics_NervousNorvus.json.\n",
      "Wrote Lyrics_GlobalNetworkSophiaAmato.json.\n",
      "Wrote Lyrics_OgdenEdsl.json.\n",
      "Wrote Lyrics_MichaelOShea.json.\n",
      "Wrote Lyrics_JohnOtway.json.\n",
      "Wrote Lyrics_BrucePalmer.json.\n",
      "Wrote Lyrics_LuciaPamela.json.\n",
      "Wrote Lyrics_HarryPartch.json.\n",
      "Wrote Lyrics_WeirdPaulPetroskey.json.\n",
      "Wrote Lyrics_PhilemonArthurAndTheDung.json.\n",
      "Wrote Lyrics_Eilertpilarm.json.\n",
      "Wrote Lyrics_ArielPink.json.\n",
      "Wrote Lyrics_PortsmouthSinfonia.json.\n",
      "Wrote Lyrics_SandraPhillips.json.\n",
      "Wrote Lyrics_LesRallizesDénudés.json.\n",
      "Wrote Lyrics_TheRedKrayola.json.\n",
      "Wrote Lyrics_SkySaxon.json.\n",
      "Wrote Lyrics_TheShaggs.json.\n",
      "Wrote Lyrics_WilliamShatner.json.\n",
      "Wrote Lyrics_SmellyEllie.json.\n",
      "Wrote Lyrics_CraigSmith.json.\n",
      "Wrote Lyrics_TheSpaceLady.json.\n",
      "Wrote Lyrics_AlexanderSkipSpence.json.\n",
      "Wrote Lyrics_TheTillers.json.\n",
      "Wrote Lyrics_TinyTim.json.\n",
      "Wrote Lyrics_Tonetta.json.\n",
      "Wrote Lyrics_TheTrachtenburgFamilySlideshowPlayers.json.\n",
      "Wrote Lyrics_BobbTrimble.json.\n",
      "Wrote Lyrics_VargVikernes.json.\n",
      "Wrote Lyrics_Viper.json.\n",
      "Wrote Lyrics_WesleyWillis.json.\n",
      "Wrote Lyrics_BrianWilson.json.\n",
      "Wrote Lyrics_GaryWilson.json.\n",
      "Wrote Lyrics_JPEGMAFIA.json.\n",
      "Wrote Lyrics_JohnReuben.json.\n",
      "Wrote Lyrics_TayZonday.json.\n"
     ]
    }
   ],
   "source": [
    "# initialize a list for artists that aren't found on Genius\n",
    "unsuccessful = []\n",
    "\n",
    "# access Genius for all musicians on the list\n",
    "for m in musicians:\n",
    "    try:\n",
    "        artist = genius.search_artist(m, sort=\"title\")\n",
    "        artist.save_lyrics()\n",
    "    except:\n",
    "        unsuccessful.append(m)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the output above, the following files absolutely need verification of the artist: (Marked with F if not verified, T if verified)\n",
    "- (F) Nat Ayer (Nathaniel Ayers)\n",
    "- (F) Leia BLACKSWAN (Leila Bela) - this is K-Pop\n",
    "- (F) LesCompagnonsDeLaChanson (Cromagnon)\n",
    "- (F) DIVINE (Divine) - intended to be Pink Flamingos Divine\n",
    "- (F) ROE (Marian Dora?)\n",
    "- (F) Naomi Hale (Naomi Hall) - really wish I had these lyrics though\n",
    "- (F) Lil Peep (Lil B) - This is popular Rap\n",
    "- (F) Miguelito Angel Masso (Angela Masson)\n",
    "- (F) Global Network Sophia Amato (Amaro Neto)\n",
    "- (F) Sandra Phillips (Sondra Prill)\n",
    "- (F) Smelly Ellie (Smelly)\n",
    "- (T) Alexander Skip Spence (Skip Spence)\n",
    "- (F) The Tillers (The Tinklers)\n",
    "- (F) JPEGMAFIA (Wing)\n",
    "- (F) John Reuben (Zapoppin')\n",
    "\n",
    "In summary, all but Alexander Skip Spence are the wrong file. To handle this, I've put all the correct files in a folder in my private directory and left the extraneous files elsewhere."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "26 Artists could not be found on Genius, not including those that were found incorrectly.\n"
     ]
    }
   ],
   "source": [
    "print(len(unsuccessful), \"Artists could not be found on Genius, not including those that were found incorrectly.\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}