Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(docker): Update docker-compose to use lily worker pattern #1044

Merged
merged 3 commits into from
Aug 29, 2022
Merged

feat(docker): Update docker-compose to use lily worker pattern #1044

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
4ee032c
feat(docker): Update docker-compose to use lily worker pattern
placer14 Aug 23, 2022
194d34e
fix: Snapshot envvar; prom host; env quotes
placer14 Aug 24, 2022
04c637a
chore: Address comments
placer14 Aug 29, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion build/grafana/provisioning/datasources/datasource.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ datasources:
# <int> org id. will default to orgId 1 if not specified
orgId: 1
# <string> url
url: http://localhost:9090
url: http://prometheus:9090
# <string> database password, if used
password:
# <string> database user, if used
Expand Down
15 changes: 9 additions & 6 deletions build/lily/docker_init.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,17 @@ set -exo pipefail

mkdir -p ${LILY_REPO}/keystore

if [[ ! -z "${_LILY_DOCKER_INIT_IMPORT_MAINNET_SNAPSHOT}" ]]; then
# import snapshot when _LILY_DOCKER_INIT_IMPORT_MAINNET_SNAPSHOT is set

if [[ ! -z "${LILY_DOCKER_INIT_IMPORT_MAINNET_SNAPSHOT}" ]]; then
# set default snapshot path if not already defined
snapshot="${LILY_DOCKER_INIT_IMPORT_SNAPSHOT_PATH:-https://snapshots.mainnet.filops.net/minimal/latest}"

# import snapshot when LILY_DOCKER_INIT_IMPORT_MAINNET_SNAPSHOT is set
if [[ -f "${LILY_REPO}/datastore/_imported" ]]; then
echo "Skipping import, found ${LILY_REPO}/datastore/_imported file."
else
echo "Importing snapshot from https://fil-chain-snapshots-fallback.s3.amazonaws.com/mainnet/minimal_finality_stateroots_latest.car..."
lily init --import-snapshot="https://fil-chain-snapshots-fallback.s3.amazonaws.com/mainnet/minimal_finality_stateroots_latest.car"

echo "Importing snapshot from ${snapshot}"
lily init --import-snapshot=${snapshot}
status=$?
if [ $status -eq 0 ]; then
touch "/var/lib/lily/datastore/_imported"
Expand All @@ -22,6 +25,6 @@ else
lily init
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't comment on the line, but above on line 11, could we add an elif with something like:

elif [[ ! -z "${LILY_DOCKER_INIT_IMPORT_SNAPSHOT_PATH}" ]]; then
    echo "Importing snapshot from ${LILY_DOCKER_INIT_IMPORT_SNAPSHOT_PATH}"
    lily init --import-snapshot=${LILY_DOCKER_INIT_IMPORT_SNAPSHOT_PATH}

then we could optionally specify locally mounted snapshots in the compose file: https://github.com/filecoin-project/lily/pull/1044/files#diff-e45e45baeda1c1e73482975a664062aa56f20c03dd9d64a827aba57775bed0d3R86

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I made the envvar get a default assignment if found to be empty.

fi

chmod 0600 ${LILY_REPO}/keystore/*
chmod -R 0600 ${LILY_REPO}/keystore

lily $@
12 changes: 8 additions & 4 deletions build/lily/docker.env → build/lily/notifier.env
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,16 +5,20 @@ LILY_REPO=/var/lib/lily
#LILY_CONFIG=/var/lib/lily/config.toml

# Postgres URL may be overridden
#LILY_STORAGE_POSTGRESQL_DB_URL=postgres://user:pass@host:port/database?options
placer14 marked this conversation as resolved.
Show resolved Hide resolved
LILY_STORAGE_POSTGRESQL_DB_URL=postgres://postgres@timescaledb:5432/postgres?sslmode=disable
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

need the password field: postgres:password@timescaledb:5432/postgres?sslmode=disable


# Enable IMPORT_SNAPSHOT below to use snapshot on lily startup
#_LILY_DOCKER_INIT_IMPORT_MAINNET_SNAPSHOT=true
#LILY_DOCKER_INIT_IMPORT_MAINNET_SNAPSHOT=true
#LILY_DOCKER_INIT_IMPORT_SNAPSHOT_PATH=

# Debugging options
#LILY_TRACING=true
LILY_PROMETHEUS_PORT="prometheus:9090"
LILY_PROMETHEUS_PORT=0.0.0.0:9090

# Logging options
GOLOG_LOG_FMT=json
GOLOG_LOG_LEVEL=info
LILY_LOG_LEVEL_NAMED="vm:error,badgerbs:error"
LILY_LOG_LEVEL_NAMED=vm:error,badgerbs:error

# Queue configuration
LILY_REDIS_ADDR=redis-queue:6379
14 changes: 14 additions & 0 deletions build/lily/notifier_config.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
[API]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

needs a storage config

ListenAddress = "/ip4/0.0.0.0/tcp/1234/http"
[Libp2p]
ListenAddresses = ["/ip4/0.0.0.0/tcp/1347"]
ConnMgrLow = 400
ConnMgrHigh = 500
ConnMgrGrace = "5m0s"
[Queue]
[Queue.Notifiers]
[Queue.Notifiers.Notifier1]
Network = "tcp"
Addr = "redis-queue:6379"
DB = 0
PoolSize = 0
2 changes: 2 additions & 0 deletions build/lily/redis-queue.env
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
# FOR DEV ONLY: disabled authentication
ALLOW_EMPTY_PASSWORD=yes
24 changes: 24 additions & 0 deletions build/lily/worker.env
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
# Path for lily state should be on a persisted volume
placer14 marked this conversation as resolved.
Show resolved Hide resolved
LILY_REPO=/var/lib/lily

# Config path default is below but may be customized
#LILY_CONFIG=/var/lib/lily/config.toml

# Postgres URL may be overridden
LILY_STORAGE_POSTGRESQL_DB_URL=postgres://postgres@timescaledb:5432/postgres?sslmode=disable

# Enable IMPORT_SNAPSHOT below to use snapshot on lily startup
#LILY_DOCKER_INIT_IMPORT_MAINNET_SNAPSHOT=true
#LILY_DOCKER_INIT_IMPORT_SNAPSHOT_PATH=

# Debugging options
#LILY_TRACING=true
LILY_PROMETHEUS_PORT=0.0.0.0:9091

# Logging options
GOLOG_LOG_FMT=json
GOLOG_LOG_LEVEL=info
LILY_LOG_LEVEL_NAMED=vm:error,badgerbs:error

# Queue configuration
LILY_REDIS_ADDR=redis-queue:6379
32 changes: 32 additions & 0 deletions build/lily/worker_config.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
[API]
ListenAddress = "/ip4/0.0.0.0/tcp/1234/http"
[Libp2p]
ListenAddresses = ["/ip4/0.0.0.0/tcp/1347"]
ConnMgrLow = 400
ConnMgrHigh = 500
ConnMgrGrace = "5m0s"
[Queue]
[Queue.Workers]
[Queue.Workers.Worker1]
[Queue.Workers.Worker1.RedisConfig]
Network = "tcp"
Addr = "redis-queue:6379"
DB = 0
PoolSize = 0
[Queue.Workers.Worker1.WorkerConfig]
Concurrency = 1
LoggerLevel = "debug"
WatchQueuePriority = 5
FillQueuePriority = 3
IndexQueuePriority = 1
WalkQueuePriority = 1
StrictPriority = false
ShutdownTimeout = 30000000000
[Storage]
[Storage.Postgresql]
[Storage.Postgresql.postgres]
SchemaName = "lily"
URLEnv = "LILY_STORAGE_POSTGRESQL_DB_URL"
ApplicationName = "lily"
PoolSize = 20
AllowUpsert = false
2 changes: 1 addition & 1 deletion build/prometheus/prometheus.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,4 @@ scrape_configs:
scrape_interval: 5s

static_configs:
- targets: ['localhost:9991']
- targets: ['notifier:9090', 'worker:9091']
placer14 marked this conversation as resolved.
Show resolved Hide resolved
87 changes: 63 additions & 24 deletions docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ services:
- '--web.console.templates=/usr/share/prometheus/consoles'
ports:
- 9090:9090
network_mode: "host"
#network_mode: "host"
restart: always

grafana:
placer14 marked this conversation as resolved.
Show resolved Hide resolved
Expand All @@ -47,38 +47,77 @@ services:
- prometheus
ports:
- 3000:3000
network_mode: "host"
#network_mode: "host"
volumes:
- grafana_data:/var/lib/grafana
- ./build/grafana/provisioning/:/etc/grafana/provisioning/
env_file:
- ./build/grafana/config.monitoring
restart: always

#lily:
#container_name: lily
## Select (only one) image
#image: filecoin/lily:v0.8.0
#image: filecoin/lily:v0.8.0-calibnet
#image: filecoin/lily:v0.8.0-calibnet-dev
#env_file:
## Check envvars for configurable options
#- ./build/lily/docker.env
#depends_on:
#- prometheus
#- timescaledb
#- jaeger
#ports:
#- 1234:1234
#volumes:
#- lily_data:/var/lib/lily
#- ./build/lily/docker_init.sh:/usr/bin/docker_init.sh
#entrypoint: /usr/bin/docker_init.sh
#command:
#- daemon
redis-queue:
image: bitnami/redis:7.0
env_file:
# Check envvars for configurable options
- ./build/lily/redis-queue.env
ports:
- 6379:6379
restart: always

notifier:
image: filecoin/lily:v0.11.0
env_file:
# Check envvars for configurable options
- ./build/lily/notifier.env
depends_on:
- prometheus
- timescaledb
- jaeger
- redis-queue
ports:
- 1234:1234
volumes:
# holds lily datastore repo
- lily_notifier_data:/var/lib/lily
# persist params through restarts
- lily_notifier_tmp:/var/tmp/filecoin-proof-parameters
# notifier-specific config
- ./build/lily/notifier_config.toml:/var/lib/lily/config.toml
- ./build/lily/docker_init.sh:/usr/bin/docker_init.sh
entrypoint: /usr/bin/docker_init.sh
command:
- "daemon --bootstrap=false"
placer14 marked this conversation as resolved.
Show resolved Hide resolved

worker:
image: filecoin/lily:v0.11.0
env_file:
# Check envvars for configurable options
- ./build/lily/worker.env
depends_on:
- prometheus
- timescaledb
- jaeger
- redis-queue
- notifier
ports:
- 1234
volumes:
# holds lily datastore repo
- lily_worker_data:/var/lib/lily
# persist params through restarts
- lily_worker_tmp:/var/tmp/filecoin-proof-parameters
# notifier-specific config
- ./build/lily/worker_config.toml:/var/lib/lily/config.toml
- ./build/lily/docker_init.sh:/usr/bin/docker_init.sh
entrypoint: /usr/bin/docker_init.sh
command:
- "daemon --bootstrap=false"
placer14 marked this conversation as resolved.
Show resolved Hide resolved

volumes:
timescaledb: {}
prometheus_data: {}
grafana_data: {}
lily_data: {}
lily_notifier_data: {}
lily_worker_data: {}
lily_notifier_tmp: {}
lily_worker_tmp: {}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The lily_*_tmp volumes will persist the params so you don't have to keep downloading them. <3

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If it's not already, could the daemon's repo also be persisted? (so we don't redownload the snapshots)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both worker and notifier have their repo's persisted.