beet lyric plugin ends with "urllib2.HTTPError: HTTP Error 403: Forbidden"

[avma]

Problem

hi

while running beet lyrics, after about 5 minutes, it stops with "urllib2.HTTPError: HTTP Error 403: Forbidden" without completing a full scan of my files

Running this command in verbose (-vv) mode:

beet lyrics

Led to this problem:

lyrics: lyrics not found: Scott Bradlee - Mashups by Candlelight, Volume Two - S                                                                                                                                                             tepping Out Night and Day
lyrics: lyrics not found: Scott Bradlee - Mashups by Candlelight, Volume Two - G                                                                                                                                                             ame of Thrones Piano Improvisation
lyrics: lyrics not found: Brenda Boykin  - NINE -  Hard Swing Travellin' Man (Cl                                                                                                                                                             ub Des Belugas remix)
Traceback (most recent call last):
  File "/usr/local/bin/beet", line 9, in <module>
    load_entry_point('beets==1.4.3', 'console_scripts', 'beet')()
  File "/usr/local/lib/python2.7/dist-packages/beets/ui/__init__.py", line 1209,                                                                                                                                                              in main
    _raw_main(args)
  File "/usr/local/lib/python2.7/dist-packages/beets/ui/__init__.py", line 1196,                                                                                                                                                              in _raw_main
    subcommand.func(lib, suboptions, subargs)
  File "/usr/local/lib/python2.7/dist-packages/beetsplug/lyrics.py", line 692, i                                                                                                                                                             n func
    opts.force_refetch or self.config['force'],
  File "/usr/local/lib/python2.7/dist-packages/beetsplug/lyrics.py", line 718, i                                                                                                                                                             n fetch_item_lyrics
    lyrics = [self.get_lyrics(artist, title) for title in titles]
  File "/usr/local/lib/python2.7/dist-packages/beetsplug/lyrics.py", line 751, i                                                                                                                                                             n get_lyrics
    lyrics = backend.fetch(artist, title)
  File "/usr/local/lib/python2.7/dist-packages/beetsplug/lyrics.py", line 566, i                                                                                                                                                             n fetch
    data = urllib.request.urlopen(url)
  File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 437, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 550, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 475, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 558, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 403: Forbidden

Here's a link to the music files that trigger the bug (if relevant):

Setup

• OS: Raspbian GNU/Linux 8 (jessie)
• Python version: 2.7.9
• beets version: 1.4.3
• Turning off plugins made problem go away (yes/no):

My configuration (output of beet config) is:

# Beets configuration --------------------------------------------------------

directory: /mnt/media/music/mymusic
library:   /mnt/media/data/beets.blb
import:
    # write metadata to music files
    write: yes

    # move imported files from source to the music directory
    move: yes
    copy: no
    delete: no

    # use auto-tagging where possible
    # do not require confirmation on strong matches
    autotag: yes
    timid: no

    resume: ask
    incremental: no
    none_rec_action: ask
    log:

    quiet: no  # enable with command line option
    quiet_fallback: skip
    default_action: apply

    singletons: no
    languages: []
    detail: no
    flat: no

# use the release-date of the original (first) release of an album?
original_date: no

# on multi-disk releases, assign track numbers for the whole album.
# If "per disk", make sure tracknames do not collide ("paths" setting).
per_disc_numbering: no

# files matching these patterns are deleted from source after import
clutter: ["Thumbs.DB", ".DS_Store", "*.m3u", "*.m3u8", ".pls", "*.jpg"]

# files/directories matching one of these patterns are ignored during import
ignore: [".*", "*~", "System Volume Information"]


# Paths ----------------------------------------------------------------------

# Paths and filenames for music files
# relative to music directory
paths:
    default: %asciify{$albumartist}/%asciify{$album}%aunique{}/$track. %asciify{$title}
    singleton: Non-Album/%asciify{$artist}/%asciify{$title}
    comp: Compilations/%asciify{$album}%aunique{}/$track. %asciify{$title}
replace:
    '[\\/]': _
    '^\.': _
    '[\x00-\x1f]': _
    '[<>:"\?\*\|]': _
    '\.$': _
    '\s+$': ''

path_sep_replace: _

# filename for the album art
art_filename: cover  # results in "cover.jpg"
max_filename_length: 0  # unlimited

# General --------------------------------------------------------------------

# use mutliple threads during import
threaded: yes
timeout: 5.0
verbose: no


# User Interface -------------------------------------------------------------

color: yes
list_format_item: %upper{$artist} - $album - $track. $title
list_format_album: %upper{$albumartist} - $album
time_format: '%d-%m-%Y %H:%M:%S'
terminal_encoding: utf8

ui:
    terminal_width: 80
    length_diff_thresh: 10.0

# Auto Tagger ----------------------------------------------------------------

match:
    strong_rec_thresh: 0.1      # match 90% or better for auto import
    medium_rec_thresh: 0.25
    rec_gap_thresh: 0.25
    max_rec:
        missing_tracks: medium
        unmatched_tracks: medium
    distance_weights:
        source: 2.0
        artist: 3.0
        album: 3.0
        media: 1.0
        mediums: 1.0
        year: 1.0
        country: 0.5
        label: 0.5
        catalognum: 0.5
        albumdisambig: 0.5
        album_id: 5.0
        tracks: 2.0
        missing_tracks: 0.9
        unmatched_tracks: 0.6
    #mpdstats,
    missing,
    duplicates,
    copyartifacts,
    lastimport,
    importadded,
    web
]
pluginpath: []

# Plugins Config -------------------------------------------------------------

lyrics:
    auto: yes
    import.wrire: yes
    fallback: ''
    sources: google lyricwiki lyrics.com musixmatch genius.com
    google_API_key: XXX

echonest_tempo:
    auto: yes

lastgenre:
    auto: yes
    whitelist: yes   # ~/.config/beets/genres.txt
    canoncical: yes  #~/.config/beets/genres-tree.yaml
    fallback: ''
    source: album    #artist track
    separator: '/'

fetchart:
    auto: yes
    #maxwidth: 300
    cautious: true
    cover_names: cover folder back cd artist fold inside inlay booklet

embedart:
    auto: yes
    #maxwidth: 300

replaygain:
    auto: yes
    overwrite: yes
    albumgain: yes

scrub:
    auto: yes

copyartifacts:
    extensions: .*

# required for mpdstats
#mpd:
 #   host: localhost
 #   port: 6600
 #   password:  # none

#mpdstats:
    #rating: False
    # two ratings are calculated:
    # "rolling" based on recent development
    # "stable" based on all-time development
    # rating mix 0.0 is all "rolling", 1.0 is all "stable"
    #rating_mix: 0.75

missing:
    format: $albumartist - $album - $track - $title
    count: yes
    total: yes

duplicates:
    checksum: no  # expensive

lastfm:
    user: 

importadded:
    preserve_mtimes: yes
    preserve_write_mtimes: yes

web:
    host: 0.0.0.0

[jackwilsdon]

Looks like this is a duplicate of #2437, which was fixed in 8087e828914a917a864dad32bd4ce06ed286b527. You can find instructions on how to run the latest source in the [FAQ](http://beets.readthedocs.io/en/latest/faq.html).

[sampsyo]

Thanks, @jackwilsdon!

[avma]

although when i tried to update request, i'm getting:Requirement already satisfied: "requests in /usr/local/lib/python2.7/dist-packages"

[jackwilsdon]

You need to update beets itself to the Git version or just disable the google lyrics source by removing it from your config until a new version of beets comes out (which I'd recommend doing instead of using the Git version).

[avma]

i did that too (ran "sudo pip install -U beets" and it saya: Requirement already up-to-date)...
is there a way to remove and reinstall lyrics.py?
i did try to manually edit lyrics.py as suggested in the #2437 fix, but then started getting more(other) errors:
lyrics: failed to fetch: http://lyrics.wikia.com/Devon_Allman:Find_Ourselves (404)
lyrics: failed to fetch: https://www.musixmatch.com/lyrics/Devon-Allman/Find-Ourselves (404)
lyrics: lyrics not found: Devon Allman - Ride or Die - Find Ourselves
^CTraceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/beets/ui/init.py", line 1209, in main
_raw_main(args)
File "/usr/local/lib/python2.7/dist-packages/beets/ui/init.py", line 1196, in _raw_main
subcommand.func(lib, suboptions, subargs)
File "/usr/local/lib/python2.7/dist-packages/beetsplug/lyrics.py", line 698, in func
opts.force_refetch or self.config['force'],
File "/usr/local/lib/python2.7/dist-packages/beetsplug/lyrics.py", line 724, in fetch_item_lyrics
lyrics = [self.get_lyrics(artist, title) for title in titles]
File "/usr/local/lib/python2.7/dist-packages/beetsplug/lyrics.py", line 757, in get_lyrics
lyrics = backend.fetch(artist, title)
File "/usr/local/lib/python2.7/dist-packages/beetsplug/lyrics.py", line 566, in fetch
data = self.fetch_url(url)
File "/usr/local/lib/python2.7/dist-packages/beetsplug/lyrics.py", line 216, in fetch_url
'User-Agent': USER_AGENT,
File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 70, in get
return request('get', url, params=params, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 56, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 488, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 609, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 423, in send
timeout=timeout
File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py", line 600, in urlopen
chunked=chunked)
File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py", line 345, in _make_request
self._validate_conn(conn)
File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py", line 844, in validate_conn
conn.connect()
File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connection.py", line 326, in connect
ssl_context=context)
File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/util/ssl.py", line 324, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "/usr/lib/python2.7/ssl.py", line 350, in wrap_socket
_context=self)
File "/usr/lib/python2.7/ssl.py", line 566, in init
self.do_handshake()
File "/usr/lib/python2.7/ssl.py", line 788, in do_handshake
self._sslobj.do_handshake()
KeyboardInterrupt

[sampsyo]

Did you take a look at the FAQ entry @jackwilsdon referred to above? It shows how to run from source: http://beets.readthedocs.io/en/latest/faq.html

Or, we'll do an official release very soon!

[avma]

thanks a lot...
yes i did, i think i'll just wait for the new release, for now i just commented out the google_api_key.