Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

lotus bundle chart #84

Merged
merged 14 commits into from
Jun 29, 2021
6 changes: 6 additions & 0 deletions charts/lotus-bundle/Chart.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
apiVersion: v2
name: lotus-bundle
description: bundle your application with lotus
type: application
version: 0.0.1
appVersion: 0.0.1
22 changes: 22 additions & 0 deletions charts/lotus-bundle/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# Lotus Bundle

If you are writing a lotus application, bundle your application with a lotus node.


## Default behaviors:

Lotus will run as a sidecar to your application. You can access lotus at 127.0.0.1:1234.
There is no authentication, and lotus will not be accessible outside of your pod.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you mean by no authentication? To use any write APIs the user will still need to be able to read the $LOTUS_PATH/token, this is required for wallet interaction so I think we will need to find a way to ensure access to the token. On first review I don't see there is any access.

Copy link
Contributor

@travisperson travisperson Jun 22, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This might be as simple as

  volumes:
  - name: lotus-path
    mountPath: /var/lib/lotus
    readOnly: true
    
  ...
  env:  
  - name: LOTUS_PATH
    value: /var/lib/lotus

This would require that the backing volume support multiple readers and I don't remember if this is common or not. This is the same as your shared volume, which is required for this chart at the moment, so it could work. If the share volume ends up being optional and this feature isnt' common we may need to find another approach.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I want to say that because they are in the same pod this isn't an issue, and should just work as long as there is only one writer.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right. Adding the lotus-path into the application container as read-only.


Lotus wallets are managed by kubernetes secrets. Any lotus wallet found in the secret
provided will be imported and available for use by your application.

TODO: Configure lotus to use a wallet service.


## Options:

lite-mode: Run lotus lite.
lite-backend: if running in lite-mode, use this service backend. By default, api.chain.love.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe add a note that api.chain.love is a lotus-gateway service and has a limited api surface area.


See values.yaml for examples.
11 changes: 11 additions & 0 deletions charts/lotus-bundle/templates/NOTES.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
{{ .Values.application.name }} with lotus sidecars

Release ---------- {{ .Release.Name }}
Namespace -------- {{ .Release.Namespace }}
Application ------ {{ .Values.application.container.image }}

The following following initial wallets will be imported:

{{ range $wallet := .Values.wallets }}
$wallet.address
{{ end }}
10 changes: 10 additions & 0 deletions charts/lotus-bundle/templates/configmaps.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
{{- range $cm := .Values.application.configMaps }}
{{- if not $cm.external }}
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ $.Release.Name }}-{{ $cm.name }}
data:
{{ .toYaml .keys | nindent 2 }}
{{- end }}
{{- end }}
23 changes: 23 additions & 0 deletions charts/lotus-bundle/templates/ingress.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
{{- if .Values.application.ingress.enabled }}
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: {{ .Release.Name }}-{{ .Values.application.name }}
annotations:
kubernetes.io/ingress.class: {{ .Values.application.ingress.class }}
{{- with .Values.application.ingress.annotations }}
{{ toYaml . | nindent 4 }}
{{- end }}
spec:
rules:
{{- range $rule := .Values.application.ingress.httpRules }}
- host: {{ $rule.host }}
http:
paths:
- path: {{ $rule.path }}
backend:
serviceName: {{ .Release.Name }}-{{ .Values.application.name }}
servicePort: $rule.servicePort
{{- end }}
{{- end }}
10 changes: 10 additions & 0 deletions charts/lotus-bundle/templates/secrets.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
{{- range $sec := .Values.application.secrets }}
{{- if not $sec.external }}
apiVersion: v1
kind: Secret
metadata:
name: {{ $.Release.Name }}-{{ $sec.name }}
data:
{{ toYaml .keys | nindent 2 }}
{{- end }}
{{- end }}
20 changes: 20 additions & 0 deletions charts/lotus-bundle/templates/service-monitor.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
{{- if .Values.prometheus.serviceMonitor }}
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: {{ .Release.Name }}-{{ .Values.application.name }}
namespace: {{ .Release.Namespace }}
spec:
namespaceSelector:
matchNames:
- {{ .Release.Namespace }}
selector:
matchLabels:
app: {{ .Values.application.name }}
release: {{ .Release.Name }}
endpoints:
- port: {{ .Values.prometheus.port }}
path: {{ .Values.prometheus.path }}
interval: 30s
{{- end }}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wondering if we should include a service monitor for lotus, don't know what it exports in lite mode, but it would help with at least basic go metrics, and would be beneficial when not running in lite mode.

23 changes: 23 additions & 0 deletions charts/lotus-bundle/templates/service.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
{{ if .Values.application.service.enabled }}
---
apiVersion: v1
kind: Service
metadata:
name: {{ .Release.Name }}-{{ .Values.application.name }}
namespace: {{ .Release.Namespace }}
labels:
app: {{ .Values.application.name }}
release: {{ .Release.Name }}
{{- with .Values.application.labels }}
{{ toYaml . | nindent 4 }}
{{- end }}
spec:
type: {{ .Values.application.service.type }}
selector:
app: {{ .Values.application.name }}
release: {{ .Release.Name }}
ports:
{{- with .Values.application.service.ports }}
{{ toYaml . | nindent 4 }}
{{- end }}
{{ end }}
170 changes: 170 additions & 0 deletions charts/lotus-bundle/templates/statefulset.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,170 @@
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: {{ .Release.Name }}-{{ .Values.application.name }}
labels:
app: {{ .Values.application.name }}
{{- with .Values.application.labels }}
{{ toYaml . | indent 4 }}
{{- end }}
spec:
replicas: {{ .Values.application.replicas }}
serviceName: {{ .Values.application.name }}
selector:
matchLabels:
app: {{ .Values.application.name }}
release: {{ .Release.Name }}
volumeClaimTemplates:
- metadata:
name: lotus-path
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 2000Gi
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe this size should have two values depending on if lite mode is enabled. There is very little to store when using a remote node. I think the only consumer of space might be for creating storage deals as a temporary hold over of the dag. I'm not sure if that is automatically cleaned up yet.

We can leave this as 2000Gi for now, but we should look into make this smaller so that this chart is more approachable. On most platforms the cost is 0.10/Gi which would be $200 a month in storage costs alone.

- metadata:
name: parameter-cache
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 500Gi
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Assuming this isn't going to be used to run a miner, the parameters are a tad under 300MB, I'd say set this to 1Gi. With a miner they are only currently 102Gi. I think we can assume though that this chart will not be used for running a miner so 1Gi should be fine.

- metadata:
name: shared-volume
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: {{ .Values.application.storage.size }}
template:
metadata:
labels:
app: {{ .Values.application.name }}
chart: {{ .Chart.Name }}-{{ .Chart.Version }}
release: {{ .Release.Name }}
{{- with .Values.application.labels }}
{{ toYaml . | indent 8 }}
{{- end }}
spec:
securityContext:
fsGroup: 532
volumes:
# The wallet, mounted by the wallet importer
- name: wallets-secret-volume
secret:
secretName: {{ .Release.Name }}-wallets
defaultMode: 0600
{{- range $sec := .Values.application.secrets }}
# secrets and configmaps, mounted by the application
- name: {{ $sec.name }}-volume
secret:
{{- if $sec.external }}
secretName: {{ $sec.name }}
{{- else }}
secretName: {{ $.Release.Name }}-{{ $sec.name }}
{{ end }}
defaultMode: 0600
{{- end }}
{{- range $cm := .Values.application.configMaps }}
- name: {{ $cm.name }}-volume
configMap:
{{- if $cm.external }}
name: {{ $cm.name }}
{{- else }}
name: {{ .Release.Name }}-{{ $cm.name }}
{{- end }}
{{- end }}

containers:
- name: {{ .Values.application.container.name }}
image: {{ .Values.application.container.image }}
imagePullPolicy: {{ .Values.application.container.imagePullPolicy }}
command: {{ .Values.application.container.command }}
args: {{ .Values.application.container.args }}
env:
- name: FULLNODE_API_INFO
value: "/ip4/127.0.0.1/tcp/1234/http"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any reason to force http?

This will limit the use of the API and not allow the use of any method that returns channels, as they require websockets to work. We should probably either force websockets or make it configurable. Http is a lot nicer as it's less prone to networking issues, so I like the idea of allowing it.

{{- with .Values.application.container.env }}
{{- toYaml . | nindent 10 }}
{{- end }}
volumeMounts:
{{- range $sec := .Values.application.secrets }}
- name: {{ $sec.name }}-volume
mountPath: {{ $sec.mount }}
{{- end }}
{{- range $cm := .Values.application.configMaps }}
- name: {{ $cm.name }}-volume
mountPath: {{ $cm.mount }}
{{- end }}
- name: shared-volume
mountPath: {{ .Values.application.storage.mount }}
# lotus sidecar
- name: lotus
image: {{ .Values.lotus.image }}
imagePullPolicy: IfNotPresent
args:
- daemon
{{- if .Values.lotus.lite.enabled }}
- "--lite"
{{- end }}
env:
- name: FILECOIN_PARAMETER_CACHE
value: /var/tmp/filecoin-proof-parameters
- name: LOTUS_PATH
value: /var/lib/lotus
- name: LOTUS_JAEGER_AGENT_HOST
value: 127.0.0.1
- name: LOTUS_JAEGER_AGENT_PORT
value: "6831"
{{- if .Values.lotus.lite.enabled }}
- name: FULLNODE_API_INFO
value: {{ .Values.lotus.lite.backend }}
- name: DOCKER_LOTUS_IMPORT_SNAPSHOT
value: ""
{{- else }}
- name: DOCKER_LOTUS_IMPORT_SNAPSHOT
value: https://fil-chain-snapshots-fallback.s3.amazonaws.com/mainnet/minimal_finality_stateroots_latest.car
{{- end }}
resources:
{{- if .Values.lotus.lite.enabled }}
requests:
memory: 16Gi
cpu: 8
limits:
memory: 32Gi
cpu: 16
{{- else }}
requests:
memory: 64Gi
cpu: 8
limits:
memory: 96Gi
cpu: 16
{{- end }}
volumeMounts:
- name: lotus-path
mountPath: /var/lib/lotus
- name: parameter-cache
mountPath: /var/tmp/filecoin-proof-parameters
- name: shared-volume
mountPath: {{ .Values.application.storage.mount }}
readOnly: true
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should check with the Dealbot team, but I think this might have to exist within certain constraints to $LOTUS_PATH. Eg when using lotus locally in $HOME/.lotus, you can't import data outside of $HOME.

I haven't verified this recently, but I know this was a thing a while back, not sure if the dealbot team is working around this in some other way.

# wallet importer
# TODO: switch to a wallet server
- name: wallet-importer
image: {{ .Values.lotus.image }}
imagePullPolicy: IfNotPresent
command: [ "bash", "-c" ]
args:
- 'while sleep 60; do for key in /wallets/*; do lotus wallet import "${key}" || true; done; done'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've filed a ticket to address this || true. This is fine for now.

filecoin-project/lotus#6562

env:
- name: FULLNODE_API_INFO
value: "/ip4/127.0.0.1/tcp/1234/http"
volumeMounts:
- name: wallets-secret-volume
mountPath: /wallets
readOnly: true
8 changes: 8 additions & 0 deletions charts/lotus-bundle/templates/wallets.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
apiVersion: v1
kind: Secret
metadata:
name: {{ .Release.Name }}-wallets
data:
{{ range $sec := .Values.wallets }}
$sec.address: $sec.exported
{{ end }}
63 changes: 63 additions & 0 deletions charts/lotus-bundle/values.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
# required:
# Your application. Replace with your own details.
application:
name: "nginx-example"
labels: []
replicas:
container:
name: application
image: "nginx"
imagePullPolicy: IfNotPresent
command: []
args: []
env: {}
resources: {}
secrets:
- name: example-secret # helm will create this secret
keys:
password: extraspecial
mount: /here
- name: external-secret # a pre-existing secret
external: true
mount: /there
configMaps: [] # same format as secrets
service:
enabled: true
type: ClusterIP
ports:
- protocol: TCP
port: 80
targetPort: 80
name: http
ingress:
enabled: false
class: nginx
annotations: {}
httpRules:
- host: example.com
path: /
servicePort: http
storage: # storage volume shared between application and lotus node
size: 1Gi
mount: /shared

# Wallets are added to the wallet secret.
# if you don't want to specify wallets in values.yaml,
# you can add them later by editing the secret.
# wallets:
# - address: f3xxxyyy
# exported: aabbccdd
wallets: []

# lotus configuration
lotus:
image: filecoin/lotus:latest
lite:
enabled: false
backend: wss://api.chain.love

# Do you want prometheus monitoring with prometheus-operator?
prometheus:
serviceMonitor: false
path: /metrics
port: ""