Skip to content

Latest commit

 

History

History
86 lines (59 loc) · 2.14 KB

README.md

File metadata and controls

86 lines (59 loc) · 2.14 KB
Raw

MovieRater

GitHub Actions status codecov
Crawl and merge ptt/imdb/yahoo movie data, help easy search high rating movie in Chinese and English.

Quick Start

  • nvm use
  • npm install -g yarn
  • docker compose up
  • yarn
  • yarn build && yarn start then open http://localhost:3003

Get the latest Data locally

  1. yarn setup
  2. yarn mergedata

Developement

Please open three command line:
For UI developement, server run at http://localhost:3004:

  1. docker compose up
  2. yarn start
  3. yarn webpack

For server developement, server run at http://localhost:3003:

  1. docker compose up
  2. yarn tsc:w
  3. TZ=Asia/Taipei yarn nodemon

Test

yarn test

Debug Test in Vscode

  1. yarn tsc:w
  2. Edit Debug Test section in ./.vscode/launch.json, for Example, if you want to debug netflixCrawler.ts file, modify the args as the following:
"name": "Debug Test",
....
"args": [
    "dist/test/netflixCrawler.test.js",
    "--no-timeouts"
],
  1. Set the breakpoint and click the Debug Test in Vscode Run and Debug debug test

Application flow

  1. Server start
  2. Load data in cache, include recent movie list, all merged data
  3. Start scheduler for crawl yahoo/imdb/ptt system diagram

Crawler

  • To manually run a single crawler you could reference firstTimeSetup.ts, and run yarn setup

Reference

The project UI is using Material-UI

Deployment

Terraform

gcloud auth application-default login
cd terraform
terraform init
terraform plan

Heroku

It is automatically built and deploy on heroku.