Refactor distributor modules to classes

[mmmaisel]

Refactoring as discussed in issue #242. All regressions should be fixed by now.
Random samples from the tests directory (test2.xml, test3.xml, Decoder.xml) work without errors.

This closes issues #242, #174, #215

[hildogjr]

We still have to checking the Digikey module/class (I don't now if this is issue act in the others class).
Using https://github.com/xesscorp/KiCost/blob/master/tests/acquire-PWM.xml (one personal project mine when was developed the multi-part...) I realize that just have success scrape with direct part number (e.g. https://www.digikey.com/products/en?keywords=MCP4922-E%2FP-ND), part numbers that return a table (e.g. https://www.digikey.com/products/en?keywords=C0805C334K5RACTU) does not populate the spreadsheet.

[mmmaisel]

@hildogjr The digikey issue should be fixed with my last commit.

[hildogjr]

Understood the change. But not fixed yet, I am debug and understanding the changes now.
acquire-PWM.xlsx

[anderwm]

I spoke to soon... with the last commit I get
No currency/country configuration for mouser.
Traceback (most recent call last):
File "/home/andersonm/kicad5_pro/KiCost/kicost/distributors/digikey/digikey.py", line 122, in dist_define_locale_currency
html = self.browser.scrape_URL(url)
File "/home/andersonm/kicad5_pro/KiCost/kicost/distributors/fake_browser.py", line 208, in scrape_URL
raise ValueError('No page')
ValueError: No page

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/andersonm/kicad5_pro/ve_kicost/bin/kicost", line 11, in
load_entry_point('kicost', 'console_scripts', 'kicost')()
File "/home/andersonm/kicad5_pro/KiCost/kicost/main.py", line 274, in main
local_currency=args.currency)
File "/home/andersonm/kicad5_pro/KiCost/kicost/kicost.py", line 257, in kicost
d, instance = result.get()
File "/usr/lib/python3.6/multiprocessing/pool.py", line 644, in get
raise self._value
File "/usr/lib/python3.6/multiprocessing/pool.py", line 119, in worker
result = (True, func(*args, **kwds))
File "/home/andersonm/kicad5_pro/KiCost/kicost/kicost.py", line 244, in mt_init_dist
instance.define_locale_currency(local_currency)
File "/home/andersonm/kicad5_pro/KiCost/kicost/distributors/distributor.py", line 105, in define_locale_currency
self.dist_define_locale_currency(locale, currency)
File "/home/andersonm/kicad5_pro/KiCost/kicost/distributors/digikey/digikey.py", line 125, in dist_define_locale_currency
raise PartHtmlError
kicost.globals.PartHtmlError

Commit 5a90494 works but not for Digikey

[hildogjr]

@anderwm by the HISTORY, currency/locale is just implemented for Digikey yet. The Digikey module is in validation now, if you just want Mouser result, you could use --include mouser or --exclude digikey for exclude Digikey.

[mmmaisel]

@hildogjr The cause of the digikey error was that digikey detected and banned the scraper. I implemented a user-agent switch in fake_browser if KiCost is detected. This solves the digikey error and future related errors.

@anderwm I could not reproduce your error. Does this still happen with the latest version? If yes, please run KiCost with the -d3 switch to enable debugging output.

[hildogjr]

I am closing all the issue fix with this new branch and do some procedure to full features tests and merge.

From 558 Digikey parts, just missed 2: "5001" ("36-5001-ND") and "A-DS 09 PP/Z".
The first is a different Digikey response page (the work around is use "36-5001-ND test"), for the second part I am in doubt. But the response is now very conscious.

[hildogjr]

Founded the source of the issue using the level 8 debug:
Looking for the "A-DS 09 PP/Z" , the link reported is https://www.digikey.com/products/en?keywords=A-DS+09+PP%2FZ+ and not https://www.digikey.com/products/en?keywords=A-DS%2009%20PP%2FZ.
Appear that the spaces (%20 code) have bing parsed as "add parameter".
To reproduce: Create a CSV file with "A-DS 09 PP/Z" and Scrape using KiCost.
Checking if there is some kind of link interpretation before the scrape.

[mmmaisel]

@hildogjr The trailing space bug was already there before the refactoring. It was added by urlquote(pn + ' ' + extra_search_terms line even if extra_search_terms was empty.
I fixed this behaviour in the last commit.

[hildogjr]

I will check this minor issues before release on PyPi.