Give redis-box a restart policy

[vringar]

When I wanted to scale down the cluster to a single node to get the jobs redistributed, the node that the redis-box pod was running on got killed and it just didn't spin back up but accepted it's deletion leaving me unable to access the hosted redis. Maybe that pod should have a restart policy

[birdsarah]

I haven't had this problem. I think I usually leave the redis box alive and scale my crawl cluster to 0 or 1 notes. And then when I want to get going again, I used this redis flush command to make sure the crawl is clear.

  docker exec -it $(docker-compose ps -q redis) sh -c "redis-cli flushall"

[vringar]

I was describing a scenario using Kubernetes on GCP and from the commands you are using I'm guessing you're running locally using docker-compose

[vringar]

Perhaps my approach of just scaling down the cluster to a single node was just semantically wrong and I should have instead reduced the parallelism of the openwpm-crawl job. But even so from my understanding the redis-box should always be running until the cluster is deleted.

[birdsarah]

Sorry, in that particular case I was but I do most of my crawls on GCP

[birdsarah]

No, I regularly scale down my kubernetes cluster as you described. It's normal for me to want to run a series of crawls. Setting up all the infrastructure over and over again is a pain. So as a cost compromise, I will scale up and down resources as you describe so I'm not just needlessly leaving lots of nodes on for an extended period of time, but I save my self from repeating the same setup over and over again.

[vringar]

Hmm, interesting. Maybe I was just too impatient and the redis box would have popped up again, but the cluster claimed it was healthy and I was unable to do kubectl exec -it redis-box -- sh to reach the redis-box. So I figured it must have been killed and not spun back up again.

[birdsarah]

well the redis box is seperate. if you run kubectl delete -f redis-box.yaml you have to go through the process to reset it up and update the crawl.yaml etc.

[vringar]

Hmm, this is the error message I got: Error from server (NotFound): pods "gke-etp2-cluster-default-pool-b7496c00-mld5" not found if I ever get it again, I'll try to explore this more. I already nuked the cluster.

[birdsarah]

yep. happy to chat here or on matrix.