Skip to content

Latest commit

 

History

History
340 lines (301 loc) · 15.5 KB

README.md

File metadata and controls

340 lines (301 loc) · 15.5 KB
Raw

redisearchStock

A stock ticker solution typeahead solution based on downloaded stock files. Uses Python redisearch for an API and jquery with bootstrap ajax typeahead plugin.

app screen

Outline

Overview

Uses python to load stock market daily values from download flat files. Using a redisearch index, a web browser application written in jquery uses typeahead prioritizing stocks with the highest volume to provide type ahead suggestions. When a stock is selected, more detail is displayed on the stock history.

Important Links/Notes

Bootstrap ajax typeahead

This plugin is in the repository so no setting is needed. To learn more-follow the directions in the bootstrap-ajax-typahead github

Instructions

Create environment

Clone the github

get clone https://github.com/jphaugla/redisearchStock.git

Download the datafiles

  • Download stock files here Stooq stock files
  • Once downloaded, move the file (it should be a directory called data) to the main redisearchStock directory
    • Can combine the various stooq files at the daily level by including world, us, etc under this daily directory
    • There is a separate file for each stock or currency with a long history of data. See instructions below for setting the environment variables to limit history load
    • To simplify things, best to remove the spaces in the file directory such as "nasdaq stocks". The spaces are dealt with in docker-compose but never cleaned this up in k8s. Just easier to eliminate the spaces.

Set environment variables

The docker compose file has the environment variables set for the redis connection and the location of the data files. In k8s, the environment is set in the configmap. This code uses redisearch. The redis database must have redisearch installed. In docker, the redis stack image contains all the modules. In k8s, redisearch is added in the database yaml file. Check the environment variables for appropriateness. Especially check the TICKER_DATA_LOCATION because loading all of the US tickers with all of the history can be a lot of data on a laptop. Here is an explanation of the environment variables. Modify these values in docker-compose.yml or in the configmap for k8s. If outside of a container, there is a file created with environment variable at scripts/app.env.

variable Original Value Desccription
REDIS_HOST redis The name of the redis docker container
REDIS_PASSWORD Redis Password
REDIS_PORT 6379 redis port
REDIS_INDEX Ticker redis port
PROCESSES 6 On larger machines, increasing this will increase load speed
WRITE_JSON false flag to use JSON instead of Hash structures
PIPELINE true flag to use pipeline for each of the ticker files (only works on python)
OLDEST_VALUE 20220101 Skip any records older than this date
CURRENT_VALUE 20220414 Use this as current value. This also sets mostrecent flag
PROCESS_RECENTS false will set most recent flag for specified keys back to false (requires creation of specific redis set)
---------------------- ----------------- ------------------------------------------------------------------------------------------------------

Multiple options for creating the environment:

  • run with python docker-compose using a flask and redis container
  • installing for mac os
  • running on linux (probably in the cloud)
  • running on kubernetes (example uses GKE)
  • run using java instead of python This java implemention is in a subfolder

Docker Compose Python

Build just needs to be done initially. NOTE: if building a new image for k8s, ensure the Dockerfile is doing a copy of the src directory into the image and not relying on docker-compose mount of the src directory. Additionally, docker can be run with the Java application. See java readme

cd python
docker-compose build
docker-compose up -d

Deploy Python

Can be issues with running flask on linux at the time of installing requirements files

create python environment

cd src
python3 -m venv venv
source venv/bin/activate
  • Use an environment file for locations
  • Need to make sure the data location variables are set correctly
  • Can also set the number of concurrent processes for the client using the "PROCESSES" environment variable
source scripts/app.env

Prepare Ticker on Python

cd src
pip install -r requirements.txt

Kubernetes

This example is showing GKE steps-adjust accordingly for other versions

Install Redis Enterprise k8s

  • Get to K8 namespace directory
cd k8s
  • Follow Redis Enterprise k8s installation instructions all the way through to step 4. Use the demo namespace as instructed.
  • For Step 5
    • the admission controller steps are not necessary
    • but the webhook instructions are not necessary
  • Don't do Step 6 as the databases for this github are in the k8s subdirectory of this github
Create redis enterprise database.
Verify cluster is ready
  • check health using kubectl
kubectl get all
kubectl get rec
kubectl get pods
  • check enterprise node cluster ui (optional steps just for demonstration purposes)
./getClusterUnPw.sh
  • port forward so can access the redis cluster
kubectl port-forward service/rec-ui 8443
Create redis database
kubectl apply -f redis-enterprise-database.yml
verify database
  • check for database in management ui
  • check database yaml output
  • get port and password for database
  • port-forward to the database
kubectl get redb/redis-enterprise-database -o yaml
./getDatabasePw.sh
kubectl port-forward service/redis-enterprise-database 10740
  • connect to redis-cli - use the password and port from the output of ./getDatabasePw.sh
redis-cli -p 16379 -a rhliu76
  • log into redis enterprise node and use rladmin
kubectl exec -it rec-0 -- bash
rladmin status extra all
change database
  • can make changes to redis database from controller and verify in either rladmin or in the GUI
  • use https://localhost:8443 to view the changes
  • modify redis-enterprise-database.yml to enable replication and to increase database size
edit redis-enterprise-database.yml
kubectl apply -f redis-enterprise-database.yml
  • will see these changes in the management ui and in rladmin

Add redisinsights

These instructions are based on Install RedisInsights on k8s   The above instructions have two options for installing redisinights, this uses the second option to install without a service (avoids creating a load balancer)

  • copy the yml file above into a file named redisinsight.yml (this file is already in the k8s directory)
  • create redisinsights
kubectl apply -f redisinsight.yml
kubectl port-forward deployment/redisinsight 8001
  • from chrome or firefox open the browser using http://localhost:8001
  • Click "I already have a database"
  • Click "Connect to Redis Database"
  • Create Connection to target redis database with following parameter entries
Key Value
host redis-enterprise-database.demo
port 18154 (get from ./getDatabasepw.sh above)
name TargetDB
Username (leave blank)
Password DrCh7J31 (from ./getDatabasepw.sh above)
  • click ok

Deploy redis-searchStock on Kubernetes

docker login
  • modify, create the environmental variables by editing configmap.yml
    • can find the IP addresses and ports for each of the databases by running kubectl get services
    • In the example below the IP address for the REDIS_HOST in the configmap.yaml is 10.28.16.188 services
    • get the database password and port by running getDatabasePw. Put the returned password the configmap REDIS_PASSWORD and REDIS_PORT
  • apply the configuration map
cd k8s
# change REDIS_PASSWORD and REDIS_PORT based on ./getDatabasePw
vi configmap.yaml
kubectl apply -f configmap.yaml
  • deploy the redis-searchstock
  • can switch between the java version and python version by changing image used in stock.yaml. Jedis version is commented out.
kubectl apply -f pvc.yaml
kubectl apply -f stock.yaml
kubectl get pods
  • port forward and continue with testing of the APIs
    • NOTE: get exact name use kubectl get pods
kubectl port-forward redis-searchstock-c568d9b6b-5t8v6 5000

Run search stock

  • get the data files from stooq
  • copy the stooq files into place. Easiest to zip them before copying using tar -cvf all_tar.tgz data/daily
    • can be issue with hidden files caused by running tar on Mac and deploying on linux see troubleshooting
kubectl cp all_tar.tgz redis-searchstock-c568d9b6b-5t8v6:/data
Publish docker image

The docker image used in k8s/stock.yaml is jphaugla/redis-searchstock:latest If building own image, have docker login, change to personal docker account and change the stock.yaml To build different version of the docker image: docker-compose build, docker tag, docker publish

cd python
docker login
docker-compose build
docker image tag redis-searchstock:latest jphaugla/redis-searchstock:latest
docker image push jphaugla/redis-searchstock:latest

Memtier benchmark

Can also optionally deploy memtier benchmark as a pod in the cluster Before adding memtier.yml, must have extra node and then label the node

kubectl label nodes gke-jph-k8s-cluster-default-pool-6ecc6b17-zllk app=memtier
kubectl apply -f memtier.yml

Deploy redis-sql

Can deploy redis-sql to run SQL on top of redisearch

  • Edit the URI string for connection to redisenterprise using ./getDatabasePw.sh
  • deploy configmap and yaml for trino
cd k8s
kubectl apply -f trino-configmap.yaml
kubectl apply -f trino.yaml

Use redisinsights or the management UI to observe the benchmark results

Use the application

Create Index

There is python running in the flask container (appy.py) listening for API calls. One of the API calls will recreate the index. Use the following script to create the index

cd scripts
./redoIndex.sh

Prepare the load

* If PROCESS_RECENTS is set, set list of recent dates to specifically set the MostRecent flag to false
  * This is needed when loading the next set of values.  E.g.  Current data is 20220315 and want to ensure three previous dates are false for MostRecent.  
```bash
docker exec -it redis redis-cli
sadd remove_current 20220314 20220313 20220312

Start Ticker Load

  • verify the directory location specified in the script
cd scripts
./loadFile.sh

Can observe the load progress by watching the load for each file

docker exec -it redis redis-cli 
hgetall ticker_load
  • THIS IS HOW to start flask app server
  • However, it is already running as part of the flask container
docker exec -it flask bash -c "python appy.py"

Go to the stock type ahead page and find the desired stock

These are a group of sample redis-cli queries to see

redic-cli -f scripts/searchQueries.txt

There are scripts in the scripts directory in addition to loadFile.sh and redoIndex.sh. Check these out for additional API calls available such as adding a ticker, deleting a ticker, getting one ticker field, removing one ticker field, etc.

Troubleshooting

Moving the Stooq datafiles from apple (BSD) to Linux can cause an odd read errors in the loading file program delete all these erroneous files. I have now added some code to ignore hidden files in python version of code.

find data/daily -type f -iname "._*.*" -ls -delete