-
Notifications
You must be signed in to change notification settings - Fork 7
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
lotus bundle chart #84
Changes from 4 commits
a6e5a6a
0821f1e
0662246
7d38159
4f7fa3a
7814e57
8ae0f16
9b98fc4
da43c72
ac35927
14487be
b2e607f
385f595
6eef741
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
apiVersion: v2 | ||
name: lotus-bundle | ||
description: bundle your application with lotus | ||
type: application | ||
version: 0.0.1 | ||
appVersion: 0.0.1 |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,22 @@ | ||
# Lotus Bundle | ||
|
||
If you are writing a lotus application, bundle your application with a lotus node. | ||
|
||
|
||
## Default behaviors: | ||
|
||
Lotus will run as a sidecar to your application. You can access lotus at 127.0.0.1:1234. | ||
There is no authentication, and lotus will not be accessible outside of your pod. | ||
|
||
Lotus wallets are managed by kubernetes secrets. Any lotus wallet found in the secret | ||
provided will be imported and available for use by your application. | ||
|
||
TODO: Configure lotus to use a wallet service. | ||
|
||
|
||
## Options: | ||
|
||
lite-mode: Run lotus lite. | ||
lite-backend: if running in lite-mode, use this service backend. By default, api.chain.love. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Maybe add a note that api.chain.love is a lotus-gateway service and has a limited api surface area. |
||
|
||
See values.yaml for examples. |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,11 @@ | ||
{{ .Values.application.name }} with lotus sidecars | ||
|
||
Release ---------- {{ .Release.Name }} | ||
Namespace -------- {{ .Release.Namespace }} | ||
Application ------ {{ .Values.application.container.image }} | ||
|
||
The following following initial wallets will be imported: | ||
|
||
{{ range $wallet := .Values.wallets }} | ||
$wallet.address | ||
{{ end }} |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,10 @@ | ||
{{- range $cm := .Values.application.configMaps }} | ||
{{- if not $cm.external }} | ||
apiVersion: v1 | ||
kind: ConfigMap | ||
metadata: | ||
name: {{ $.Release.Name }}-{{ $cm.name }} | ||
data: | ||
{{ .toYaml .keys | nindent 2 }} | ||
{{- end }} | ||
{{- end }} |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,23 @@ | ||
{{- if .Values.application.ingress.enabled }} | ||
--- | ||
apiVersion: networking.k8s.io/v1beta1 | ||
kind: Ingress | ||
metadata: | ||
name: {{ .Release.Name }}-{{ .Values.application.name }} | ||
annotations: | ||
kubernetes.io/ingress.class: {{ .Values.application.ingress.class }} | ||
{{- with .Values.application.ingress.annotations }} | ||
{{ toYaml . | nindent 4 }} | ||
{{- end }} | ||
spec: | ||
rules: | ||
{{- range $rule := .Values.application.ingress.httpRules }} | ||
- host: {{ $rule.host }} | ||
http: | ||
paths: | ||
- path: {{ $rule.path }} | ||
backend: | ||
serviceName: {{ .Release.Name }}-{{ .Values.application.name }} | ||
servicePort: $rule.servicePort | ||
{{- end }} | ||
{{- end }} |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,10 @@ | ||
{{- range $sec := .Values.application.secrets }} | ||
{{- if not $sec.external }} | ||
apiVersion: v1 | ||
kind: Secret | ||
metadata: | ||
name: {{ $.Release.Name }}-{{ $sec.name }} | ||
data: | ||
{{ toYaml .keys | nindent 2 }} | ||
{{- end }} | ||
{{- end }} |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,20 @@ | ||
{{- if .Values.prometheus.serviceMonitor }} | ||
--- | ||
apiVersion: monitoring.coreos.com/v1 | ||
kind: ServiceMonitor | ||
metadata: | ||
name: {{ .Release.Name }}-{{ .Values.application.name }} | ||
namespace: {{ .Release.Namespace }} | ||
spec: | ||
namespaceSelector: | ||
matchNames: | ||
- {{ .Release.Namespace }} | ||
selector: | ||
matchLabels: | ||
app: {{ .Values.application.name }} | ||
release: {{ .Release.Name }} | ||
endpoints: | ||
- port: {{ .Values.prometheus.port }} | ||
path: {{ .Values.prometheus.path }} | ||
interval: 30s | ||
{{- end }} | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Wondering if we should include a service monitor for lotus, don't know what it exports in lite mode, but it would help with at least basic go metrics, and would be beneficial when not running in lite mode. |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,23 @@ | ||
{{ if .Values.application.service.enabled }} | ||
--- | ||
apiVersion: v1 | ||
kind: Service | ||
metadata: | ||
name: {{ .Release.Name }}-{{ .Values.application.name }} | ||
namespace: {{ .Release.Namespace }} | ||
labels: | ||
app: {{ .Values.application.name }} | ||
release: {{ .Release.Name }} | ||
{{- with .Values.application.labels }} | ||
{{ toYaml . | nindent 4 }} | ||
{{- end }} | ||
spec: | ||
type: {{ .Values.application.service.type }} | ||
selector: | ||
app: {{ .Values.application.name }} | ||
release: {{ .Release.Name }} | ||
ports: | ||
{{- with .Values.application.service.ports }} | ||
{{ toYaml . | nindent 4 }} | ||
{{- end }} | ||
{{ end }} |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,170 @@ | ||
--- | ||
apiVersion: apps/v1 | ||
kind: StatefulSet | ||
metadata: | ||
name: {{ .Release.Name }}-{{ .Values.application.name }} | ||
labels: | ||
app: {{ .Values.application.name }} | ||
{{- with .Values.application.labels }} | ||
{{ toYaml . | indent 4 }} | ||
{{- end }} | ||
spec: | ||
replicas: {{ .Values.application.replicas }} | ||
serviceName: {{ .Values.application.name }} | ||
selector: | ||
matchLabels: | ||
app: {{ .Values.application.name }} | ||
release: {{ .Release.Name }} | ||
volumeClaimTemplates: | ||
- metadata: | ||
name: lotus-path | ||
spec: | ||
accessModes: | ||
- ReadWriteOnce | ||
resources: | ||
requests: | ||
storage: 2000Gi | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Maybe this size should have two values depending on if lite mode is enabled. There is very little to store when using a remote node. I think the only consumer of space might be for creating storage deals as a temporary hold over of the dag. I'm not sure if that is automatically cleaned up yet. We can leave this as 2000Gi for now, but we should look into make this smaller so that this chart is more approachable. On most platforms the cost is 0.10/Gi which would be $200 a month in storage costs alone. |
||
- metadata: | ||
name: parameter-cache | ||
spec: | ||
accessModes: | ||
- ReadWriteOnce | ||
resources: | ||
requests: | ||
storage: 500Gi | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Assuming this isn't going to be used to run a miner, the parameters are a tad under 300MB, I'd say set this to 1Gi. With a miner they are only currently 102Gi. I think we can assume though that this chart will not be used for running a miner so 1Gi should be fine. |
||
- metadata: | ||
name: shared-volume | ||
spec: | ||
accessModes: | ||
- ReadWriteOnce | ||
resources: | ||
requests: | ||
storage: {{ .Values.application.storage.size }} | ||
template: | ||
metadata: | ||
labels: | ||
app: {{ .Values.application.name }} | ||
chart: {{ .Chart.Name }}-{{ .Chart.Version }} | ||
release: {{ .Release.Name }} | ||
{{- with .Values.application.labels }} | ||
{{ toYaml . | indent 8 }} | ||
{{- end }} | ||
spec: | ||
securityContext: | ||
fsGroup: 532 | ||
volumes: | ||
# The wallet, mounted by the wallet importer | ||
- name: wallets-secret-volume | ||
secret: | ||
secretName: {{ .Release.Name }}-wallets | ||
defaultMode: 0600 | ||
{{- range $sec := .Values.application.secrets }} | ||
# secrets and configmaps, mounted by the application | ||
- name: {{ $sec.name }}-volume | ||
secret: | ||
{{- if $sec.external }} | ||
secretName: {{ $sec.name }} | ||
{{- else }} | ||
secretName: {{ $.Release.Name }}-{{ $sec.name }} | ||
{{ end }} | ||
defaultMode: 0600 | ||
{{- end }} | ||
{{- range $cm := .Values.application.configMaps }} | ||
- name: {{ $cm.name }}-volume | ||
configMap: | ||
{{- if $cm.external }} | ||
name: {{ $cm.name }} | ||
{{- else }} | ||
name: {{ .Release.Name }}-{{ $cm.name }} | ||
{{- end }} | ||
{{- end }} | ||
|
||
containers: | ||
- name: {{ .Values.application.container.name }} | ||
image: {{ .Values.application.container.image }} | ||
imagePullPolicy: {{ .Values.application.container.imagePullPolicy }} | ||
command: {{ .Values.application.container.command }} | ||
args: {{ .Values.application.container.args }} | ||
env: | ||
- name: FULLNODE_API_INFO | ||
value: "/ip4/127.0.0.1/tcp/1234/http" | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Any reason to force http? This will limit the use of the API and not allow the use of any method that returns channels, as they require websockets to work. We should probably either force websockets or make it configurable. Http is a lot nicer as it's less prone to networking issues, so I like the idea of allowing it. |
||
{{- with .Values.application.container.env }} | ||
{{- toYaml . | nindent 10 }} | ||
{{- end }} | ||
volumeMounts: | ||
{{- range $sec := .Values.application.secrets }} | ||
- name: {{ $sec.name }}-volume | ||
mountPath: {{ $sec.mount }} | ||
{{- end }} | ||
{{- range $cm := .Values.application.configMaps }} | ||
- name: {{ $cm.name }}-volume | ||
mountPath: {{ $cm.mount }} | ||
{{- end }} | ||
- name: shared-volume | ||
mountPath: {{ .Values.application.storage.mount }} | ||
# lotus sidecar | ||
- name: lotus | ||
image: {{ .Values.lotus.image }} | ||
imagePullPolicy: IfNotPresent | ||
args: | ||
- daemon | ||
{{- if .Values.lotus.lite.enabled }} | ||
- "--lite" | ||
{{- end }} | ||
env: | ||
- name: FILECOIN_PARAMETER_CACHE | ||
value: /var/tmp/filecoin-proof-parameters | ||
- name: LOTUS_PATH | ||
value: /var/lib/lotus | ||
- name: LOTUS_JAEGER_AGENT_HOST | ||
value: 127.0.0.1 | ||
- name: LOTUS_JAEGER_AGENT_PORT | ||
value: "6831" | ||
{{- if .Values.lotus.lite.enabled }} | ||
- name: FULLNODE_API_INFO | ||
value: {{ .Values.lotus.lite.backend }} | ||
- name: DOCKER_LOTUS_IMPORT_SNAPSHOT | ||
value: "" | ||
{{- else }} | ||
- name: DOCKER_LOTUS_IMPORT_SNAPSHOT | ||
value: https://fil-chain-snapshots-fallback.s3.amazonaws.com/mainnet/minimal_finality_stateroots_latest.car | ||
{{- end }} | ||
resources: | ||
{{- if .Values.lotus.lite.enabled }} | ||
requests: | ||
memory: 16Gi | ||
cpu: 8 | ||
limits: | ||
memory: 32Gi | ||
cpu: 16 | ||
{{- else }} | ||
requests: | ||
memory: 64Gi | ||
cpu: 8 | ||
limits: | ||
memory: 96Gi | ||
cpu: 16 | ||
{{- end }} | ||
volumeMounts: | ||
- name: lotus-path | ||
mountPath: /var/lib/lotus | ||
- name: parameter-cache | ||
mountPath: /var/tmp/filecoin-proof-parameters | ||
- name: shared-volume | ||
mountPath: {{ .Values.application.storage.mount }} | ||
readOnly: true | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Should check with the Dealbot team, but I think this might have to exist within certain constraints to $LOTUS_PATH. Eg when using lotus locally in $HOME/.lotus, you can't import data outside of $HOME. I haven't verified this recently, but I know this was a thing a while back, not sure if the dealbot team is working around this in some other way. |
||
# wallet importer | ||
# TODO: switch to a wallet server | ||
- name: wallet-importer | ||
image: {{ .Values.lotus.image }} | ||
imagePullPolicy: IfNotPresent | ||
command: [ "bash", "-c" ] | ||
args: | ||
- 'while sleep 60; do for key in /wallets/*; do lotus wallet import "${key}" || true; done; done' | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I've filed a ticket to address this |
||
env: | ||
- name: FULLNODE_API_INFO | ||
value: "/ip4/127.0.0.1/tcp/1234/http" | ||
volumeMounts: | ||
- name: wallets-secret-volume | ||
mountPath: /wallets | ||
readOnly: true |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,8 @@ | ||
apiVersion: v1 | ||
kind: Secret | ||
metadata: | ||
name: {{ .Release.Name }}-wallets | ||
data: | ||
{{ range $sec := .Values.wallets }} | ||
$sec.address: $sec.exported | ||
{{ end }} |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,63 @@ | ||
# required: | ||
# Your application. Replace with your own details. | ||
application: | ||
name: "nginx-example" | ||
labels: [] | ||
replicas: | ||
container: | ||
name: application | ||
image: "nginx" | ||
imagePullPolicy: IfNotPresent | ||
command: [] | ||
args: [] | ||
env: {} | ||
resources: {} | ||
secrets: | ||
- name: example-secret # helm will create this secret | ||
keys: | ||
password: extraspecial | ||
mount: /here | ||
- name: external-secret # a pre-existing secret | ||
external: true | ||
mount: /there | ||
configMaps: [] # same format as secrets | ||
service: | ||
enabled: true | ||
type: ClusterIP | ||
ports: | ||
- protocol: TCP | ||
port: 80 | ||
targetPort: 80 | ||
name: http | ||
ingress: | ||
enabled: false | ||
class: nginx | ||
annotations: {} | ||
httpRules: | ||
- host: example.com | ||
path: / | ||
servicePort: http | ||
storage: # storage volume shared between application and lotus node | ||
size: 1Gi | ||
mount: /shared | ||
|
||
# Wallets are added to the wallet secret. | ||
# if you don't want to specify wallets in values.yaml, | ||
# you can add them later by editing the secret. | ||
# wallets: | ||
# - address: f3xxxyyy | ||
# exported: aabbccdd | ||
wallets: [] | ||
|
||
# lotus configuration | ||
lotus: | ||
image: filecoin/lotus:latest | ||
lite: | ||
enabled: false | ||
backend: wss://api.chain.love | ||
|
||
# Do you want prometheus monitoring with prometheus-operator? | ||
prometheus: | ||
serviceMonitor: false | ||
path: /metrics | ||
port: "" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What do you mean by no authentication? To use any write APIs the user will still need to be able to read the
$LOTUS_PATH/token
, this is required for wallet interaction so I think we will need to find a way to ensure access to the token. On first review I don't see there is any access.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This might be as simple as
This would require that the backing volume support multiple readers and I don't remember if this is common or not. This is the same as your shared volume, which is required for this chart at the moment, so it could work. If the share volume ends up being optional and this feature isnt' common we may need to find another approach.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I want to say that because they are in the same pod this isn't an issue, and should just work as long as there is only one writer.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You're right. Adding the lotus-path into the application container as read-only.