Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Refactor distributor modules to classes #262

Merged
merged 29 commits into from
Jun 7, 2018
Merged

Refactor distributor modules to classes #262

merged 29 commits into from
Jun 7, 2018

Commits on Jun 4, 2018

  1. Started implementation of fake_browser and distributor base classes.

    @mmmaisel
    mmmaisel committed Jun 4, 2018
    Configuration menu
    Copy the full SHA
    4e3c7ec View commit details
    Browse the repository at this point in the history

  2. Started refactoring as discussed in issue hildogjr#242. 

    Goals are:
- Add stateful user-agent and cookie handling in fake_browser
- Scrape parts in distrtibutor then part order
- Use a class inheritance approach to distributor modules, this allows
  adding state information and reduces the amount of variabled passed around.
- One scraping thread per distributor, simplify locking

Implemented in this commit:
- Used class approach to distributors and fake_browser
- Parts are scraped in distributor -> part order
- One (IO limited) python thread per distributor
- Simplified locking
    @mmmaisel
    mmmaisel committed Jun 4, 2018
    Configuration menu
    Copy the full SHA
    54e1a8d View commit details
    Browse the repository at this point in the history

  3. Implemented dist_mouser class.

    @mmmaisel
    mmmaisel committed Jun 4, 2018
    Configuration menu
    Copy the full SHA
    f58cca9 View commit details
    Browse the repository at this point in the history

  4. Implemented dist_farnell class.

    @mmmaisel
    mmmaisel committed Jun 4, 2018
    Configuration menu
    Copy the full SHA
    817b136 View commit details
    Browse the repository at this point in the history

  5. Implemented dist_local class.

    @mmmaisel
    mmmaisel committed Jun 4, 2018
    Configuration menu
    Copy the full SHA
    aa1f175 View commit details
    Browse the repository at this point in the history

  6. Implemented dist_digikey class.

    @mmmaisel
    mmmaisel committed Jun 4, 2018
    Configuration menu
    Copy the full SHA
    0ec9ee1 View commit details
    Browse the repository at this point in the history

  7. Implemented dist_newark class.

    @mmmaisel
    mmmaisel committed Jun 4, 2018
    Configuration menu
    Copy the full SHA
    4e10f2b View commit details
    Browse the repository at this point in the history

  8. Implemented dist_rs class.

    @mmmaisel
    mmmaisel committed Jun 4, 2018
    Configuration menu
    Copy the full SHA
    7fcc91f View commit details
    Browse the repository at this point in the history

  9. Implemented dist_tme class.

    @mmmaisel
    mmmaisel committed Jun 4, 2018
    Configuration menu
    Copy the full SHA
    5c7be6a View commit details
    Browse the repository at this point in the history

  10. fake_browser: Replace "urllib" with "requests" to remove problematic … 

    …"Connection: close" http header.
    @mmmaisel
    mmmaisel committed Jun 4, 2018
    Configuration menu
    Copy the full SHA
    0b68823 View commit details
    Browse the repository at this point in the history

  11. Removed now unused file web_routines.py.

    @mmmaisel
    mmmaisel committed Jun 4, 2018
    Configuration menu
    Copy the full SHA
    0bbfe98 View commit details
    Browse the repository at this point in the history

  12. Local distributor improvements and fixed custom local distributor nam… 

    …es regression.
    @mmmaisel
    mmmaisel committed Jun 4, 2018
    Configuration menu
    Copy the full SHA
    8cccfc7 View commit details
    Browse the repository at this point in the history

  13. Fixed logger initialization, log messages from kicost.py should now s… 

    …how up correctly.
    @mmmaisel
    mmmaisel committed Jun 4, 2018
    Configuration menu
    Copy the full SHA
    7182bfe View commit details
    Browse the repository at this point in the history

  14. Added new debugging level DEBUG_HTTP_RESPONSES and added per thread t… 

    …iming traces.
    @mmmaisel
    mmmaisel committed Jun 4, 2018
    Configuration menu
    Copy the full SHA
    071f5d0 View commit details
    Browse the repository at this point in the history

  15. Limit num_processes to distributor count, fixed name error.

    @mmmaisel
    mmmaisel committed Jun 4, 2018
    Configuration menu
    Copy the full SHA
    253346b View commit details
    Browse the repository at this point in the history

  16. Default throttling_delay to 5 seconds to avoid getting banned.

    @mmmaisel
    mmmaisel committed Jun 4, 2018
    Configuration menu
    Copy the full SHA
    53bd362 View commit details
    Browse the repository at this point in the history

  17. Moved throttling_delay handling to fake_browser.

    @mmmaisel
    mmmaisel committed Jun 4, 2018
    Configuration menu
    Copy the full SHA
    9f38aab View commit details
    Browse the repository at this point in the history

  18. Revised python2/3 import differences as suggested by hildogr.

    @mmmaisel
    mmmaisel committed Jun 4, 2018
    Configuration menu
    Copy the full SHA
    6ac3ec2 View commit details
    Browse the repository at this point in the history

  19. Implemented multithreaded distributor initialisation and reduced mt_s… 

    …crape_part argument count.
    @mmmaisel
    mmmaisel committed Jun 4, 2018
    Configuration menu
    Copy the full SHA
    dd69881 View commit details
    Browse the repository at this point in the history

  20. Added "requests" dependency to setup.py.

    @mmmaisel
    mmmaisel committed Jun 4, 2018
    Configuration menu
    Copy the full SHA
    0718fd3 View commit details
    Browse the repository at this point in the history

  21. Moved CurrencyConverter instance to globals.

    @mmmaisel
    mmmaisel committed Jun 4, 2018
    Configuration menu
    Copy the full SHA
    500116e View commit details
    Browse the repository at this point in the history

  22. Fixed remaining digikey refactoring issues.

    @mmmaisel
    mmmaisel committed Jun 4, 2018
    Configuration menu
    Copy the full SHA
    34c34fa View commit details
    Browse the repository at this point in the history

  23. Fixed remaining RS refactoring issues and updated URL.

    @mmmaisel
    mmmaisel committed Jun 4, 2018
    Configuration menu
    Copy the full SHA
    08412e3 View commit details
    Browse the repository at this point in the history

  24. TME fix and clean-up imports

    @hildogjr @mmmaisel
    hildogjr authored and mmmaisel committed Jun 4, 2018
    Configuration menu
    Copy the full SHA
    5a90494 View commit details
    Browse the repository at this point in the history

Commits on Jun 5, 2018

  1. Fixed syntax/naming error in digikey part number from table function.

    @mmmaisel
    mmmaisel committed Jun 5, 2018
    Configuration menu
    Copy the full SHA
    bcfbe7d View commit details
    Browse the repository at this point in the history

Commits on Jun 6, 2018

  1. Fixed compiler errors in digikey dist_define_locale_currency, 

    removed unneccessary non-logger debug print in kicost.py.
    @mmmaisel
    mmmaisel committed Jun 6, 2018
    Configuration menu
    Copy the full SHA
    d1902b3 View commit details
    Browse the repository at this point in the history

  2. Implemented session recreation strategy if scraper gets detected.

    @mmmaisel
    mmmaisel committed Jun 6, 2018
    Configuration menu
    Copy the full SHA
    3aa0659 View commit details
    Browse the repository at this point in the history

  3. Removed unused "log_level" parameter in distributor class. 

    This commit finishes the refactoring.
    @mmmaisel
    mmmaisel committed Jun 6, 2018
    Configuration menu
    Copy the full SHA
    fa1d98f View commit details
    Browse the repository at this point in the history

Commits on Jun 7, 2018

  1. Remove trailing space from search URLs.

    @mmmaisel
    mmmaisel committed Jun 7, 2018
    Configuration menu
    Copy the full SHA
    51ba35d View commit details
    Browse the repository at this point in the history