Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Adding openAPIV3Schema Validation for the Flink application custom resource #33

Merged
merged 3 commits into from
Jun 24, 2019
Merged

Adding openAPIV3Schema Validation for the Flink application custom resource #33

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
c5d5a4f
Adding openAPIV3Schema Validation for the Flink application custom re…
anandswaminathan Jun 21, 2019
fb00e46
Fixes
anandswaminathan Jun 21, 2019
c197993
Address comments
anandswaminathan Jun 24, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
352 changes: 350 additions & 2 deletions deploy/crd.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,9 +11,357 @@ spec:
singular: flinkapplication
shortNames:
- flinkapp

scope: Namespaced
version: v1alpha1
validation:
# openAPIV3Schema is the schema for validating custom objects.
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
image:
type: string
imagePullPolicy:
type: string
enum: [Always, Never, IfNotPresent]
imagePullSecrets:
type: array
items:
type: object
properties:
name:
type: string
jarName:
type: string
programArgs:
type: string
entryClass:
type: string
flinkVersion:
type: string
restartNonce:
type: string
parallelism:
type: integer
minimum: 1
deleteMode:
type: string
enum: [Savepoint, None, ForceCancel]
deploymentMode:
type: string
enum: [Dual]
rpcPort:
type: integer
minimum: 1
maximum: 65535
blobPort:
type: integer
minimum: 1
maximum: 65535
queryPort:
type: integer
minimum: 1
maximum: 65535
metricsQueryPort:
type: integer
minimum: 1
maximum: 65535
flinkConfig:
type: object
properties:
additionalProperties:
type: string
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is too restrictive. In the go definition this is map[string]interface{}, and there are flink configs that are numeric and boolean in addition to string.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem with map[string]interface{} is that it can allow nested structure. On the other end we are appending only strings to the conf.yaml.

I tested with integer values like taskmanager.heap.size: 200 and boolean values and they work. It integer as numeric string and does not fail.

savepointInfo:
type: object
properties:
savepointLocation:
type: string
jobManagerConfig:
type: object
properties:
replicas:
type: integer
minimum: 1
offHeapMemoryFraction:
type: float
nodeSelector:
type: object
properties:
additionalProperties:
type: string
envConfig:
type: object
properties:
env:
items:
properties:
name:
type: string
value:
type: string
valueFrom:
properties:
configMapKeyRef:
properties:
key:
type: string
name:
type: string
optional:
type: boolean
required:
- key
type: object
fieldRef:
properties:
apiVersion:
type: string
fieldPath:
type: string
required:
- fieldPath
type: object
resourceFieldRef:
properties:
containerName:
type: string
divisor: {}
resource:
type: string
required:
- resource
type: object
secretKeyRef:
properties:
key:
type: string
name:
type: string
optional:
type: boolean
required:
- key
type: object
type: object
required:
- name
type: object
type: array
envFrom:
items:
properties:
configMapRef:
properties:
name:
type: string
optional:
type: boolean
type: object
prefix:
type: string
secretRef:
properties:
name:
type: string
optional:
type: boolean
type: object
type: object
type: array
resources:
type: object
properties:
requests:
type: object
properties:
memory:
type: string
pattern: ^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$
cpu:
type: string
pattern: ^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$
storage:
type: string
pattern: ^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$
ephemeral-storage:
type: string
pattern: ^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$
limits:
type: object
properties:
memory:
type: string
pattern: ^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$
cpu:
type: string
pattern: ^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$
storage:
type: string
pattern: ^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$
ephemeral-storage:
type: string
pattern: ^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$
taskManagerConfig:
type: object
properties:
taskSlots:
type: integer
minimum: 1
offHeapMemoryFraction:
type: float
nodeSelector:
type: object
properties:
additionalProperties:
type: string
envConfig:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we factor this out into its own object? That would avoid the need to duplicate it between TM/JM.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried that I am unable to make components work for custom resources. Do you have any working example you can share ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like it's not supported in kubernetes: kubernetes/kubernetes#62872 😞

type: object
properties:
env:
items:
properties:
name:
type: string
value:
type: string
valueFrom:
properties:
configMapKeyRef:
properties:
key:
type: string
name:
type: string
optional:
type: boolean
required:
- key
type: object
fieldRef:
properties:
apiVersion:
type: string
fieldPath:
type: string
required:
- fieldPath
type: object
resourceFieldRef:
properties:
containerName:
type: string
divisor: {}
resource:
type: string
required:
- resource
type: object
secretKeyRef:
properties:
key:
type: string
name:
type: string
optional:
type: boolean
required:
- key
type: object
type: object
required:
- name
type: object
type: array
envFrom:
items:
properties:
configMapRef:
properties:
name:
type: string
optional:
type: boolean
type: object
prefix:
type: string
secretRef:
properties:
name:
type: string
optional:
type: boolean
type: object
type: object
type: array
resources:
type: object
properties:
requests:
type: object
properties:
memory:
type: string
pattern: ^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$
cpu:
type: string
pattern: ^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$
storage:
type: string
pattern: ^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$
ephemeral-storage:
type: string
pattern: ^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$
limits:
type: object
properties:
memory:
type: string
pattern: ^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$
cpu:
type: string
pattern: ^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$
storage:
type: string
pattern: ^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$
ephemeral-storage:
type: string
pattern: ^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$
volumes:
type: array
items:
type: object
properties:
name:
type: string
required:
- name
volumeMounts:
type: array
items:
type: object
properties:
mountPath:
type: string
mountPropagation:
type: string
name:
type: string
readOnly:
type: boolean
subPath:
type: string
subPathExpr:
type: string
required:
- name
- mountPath
required:
- image
- jarName
- parallelism
- entryClass

additionalPrinterColumns:
- name: Phase
Expand Down Expand Up @@ -42,4 +390,4 @@ spec:
JSONPath: .status.jobStatus.jobRestartCount
- name: Age
type: date
JSONPath: .metadata.creationTimestamp
JSONPath: .metadata.creationTimestamp
2 changes: 1 addition & 1 deletion integ/test_app.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ spec:
path: /tmp/checkpoints
type: Directory
flinkVersion: "1.8"
deployType: Dual
deploymentMode: Dual
jarName: "operator-test-app-1.0.0-SNAPSHOT.jar"
parallelism: 3
entryClass: "com.lyft.OperatorTestApp"
Loading