Fractional {job,task}manager.heap.size crashes Flink

[JonnyIncognito]

The offHeapMemoryFraction can cause fractional values to be written to flink-conf.yaml, which crashes Flink on startup:

/opt/flink/bin/config.sh: line 261: [ERROR] >> 20: syntax error: operand expected (error token is "[ERROR] >> 20")
[ERROR] Configured JobManager memory size is not a valid value. Please set 'jobmanager.heap.size' in flink-conf.yaml.

For example, the following FlinkApplication...

  jobManagerConfig:
    resources:
      requests:
        memory: "4Gi"
        cpu: "1.0"
    offHeapMemoryFraction: 0.15
    replicas: 1
  taskManagerConfig:
    taskSlots: 1
    resources:
      requests:
        memory: "4Gi"
        cpu: "1.0"
    offHeapMemoryFraction: 0.3

... causes the following to be generated:

jobmanager.heap.size: 3481.6
taskmanager.heap.size: 2867.2

Above is using v0.2.0

[anandswaminathan]

@JonnyIncognito

Aah yes, it is is float https://github.com/lyft/flinkk8soperator/blob/master/pkg/controller/flink/config.go#L87

Want to contribute a fix ?

[JonnyIncognito]

Sure! I'm a Go newbie but seems like a nice simple task to start with.

[anandswaminathan]

Also we have a slack channel - slack

Feel free to join.

[JonnyIncognito]

Addition to this: I noticed that the memory characteristics weren't correct in our metrics, which eventually led me to realise that the two config parameters are using the wrong units. Same generated example as above:

jobmanager.heap.size: 3481.6
taskmanager.heap.size: 2867.2

The xyz.heap.size config parameters without a unit suffix are interpreted as bytes, not the FlinkK8sOperator's intended MiB. This leads to taskmanager.sh skipping the -Xms...M, -Xmx...M and -XX:MaxDirectMemorySize=... JVM arguments. The generated config values should either have a "m" suffix (therefore string type instead of float64) so that config.sh:parseBytes() can interpret them correctly or the config keys xyz.heap.mb should be used instead.

[anandswaminathan]

^ cc @mwylde

[mwylde]

Sorry for the long delay, I have a fix out in #106. And thanks for the report!

[mwylde]

The fix has been merged. Thanks for reporting!

[JonnyIncognito]

That's great, many thanks! The power of open-source: I filed this fully expecting to and planning to fix it myself some time this coming month, but happy you guys got there first and it's fixed :) I'll upgrade our clusters as soon as I can.