This is a set of spiders based on the scrapy library that crawl dealer websites and extract data about vehicles in stock. It currently has BMW X3 hardcoded as that's what I was looking for, but it should theoretically work for any make and model as long as the dealer uses one of the three supported platforms - dealeron, dealer.com and dealerinspire.
The goal of this project is to democratize data on available vehicle inventory and serve as a free tool in your search for your next car.
The easiest way to set this up is to use a python virtual environment. These instructions have been tested on OSX with python 3:
python3 -m venv env
source env/bin/activate
pip install -r requirements.txt
cd dealers
scrapy crawl bimmer_dealerinspire -o dealerinspire_inventory.json
scrapy crawl bimmer_dealercom -o dealercom_inventory.json
scrapy crawl bimmer_dealeron -o dealeron_inventory.json
I've found it easy to browse inventory from the command line by using jq. You can sort and filter, for example:
cat dealerinspire_inventory.json | jq 'sort_by(.msrp)' | less
- Onboard all dealers across the US. This should be as easy as copy/pasting their websites in the spider code.
- Auto-discover dealers. This can be done using a spider crawling https://bmwusa.com/
- Add links to vehicles.
- Follow vehicle links and scrape options and colors.
- Extend to other brands.
- Preserve output in a database. Elasticsearch should work very well.
- Build a UI.
- Decode VIN or at least add a decoder link.