What is the most ideomatic way of rendering inline JSON in CRD ?

[eshepelyuk]

Hello

I have to render CRD looking smth like this

.....
spec:
  forProvider:
    inlinePolicy:
      - name: default
        policy: |
          {
            "Version": "2012-10-17",
            "Statement": [
              {
                "Effect": "Allow",
                "Resource": "*",
                "Action": "ssm:GetParameter*"
              }
            ]
          }

Using naive _HT!{{ .. }} block, I am receiving YAML result and according to docs it's an expected behaviour.
So to render inline JSON - I've declared a new value in hull.config.specific and later used _HT*toPrettyJson|hull.config.specific.inlineJson to render it.

It works OK now, but I'd like to understand

• is this recommended/ideomatic approach
• is this syntax covered in docs, because I only discovered it by checking test cases

Thanks in advance.

[gre9ory]

Hi @eshepelyuk,

again you bring up an interesting topic.

In the further explanations I am focusing on the value of policy in your CR example above.

First off, I think you can easily distinguish the case where you just want to have plain JSON in the value of policy which is not altered by tpl in any way. This of course is served already by your example:

- name: plain-json-string
  policy: |
    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Resource": "*",
          "Action": "ssm:GetParameter*"
        }
      ]
    }

yielding identical output:

- name: plain-json-string
  policy: |-
    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Resource": "*",
          "Action": "ssm:GetParameter*"
        }
      ]
    }

More interesting is of course the possibility of dynamic alteration of policy's value.

Using _HT! around the value will make HULL interpret the policy content and the content in this case is interpreted as flow-style YAML. I have written some explanations about why this is the case here for example. The returned value of _HT! is then a dictionary in this case which is appended to the YAML tree.

This means when using this input:

- name: interpreted-as-flow-style-yaml
  policy: |-
    _HT!
      {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Effect": "{{ printf "%s" "Allow" }}",
            "Resource": "*",
            "Action": "ssm:GetParameter*"
          }
        ]
      }

yields the dictionary in YAML form:

- name: interpreted-flow-style-yaml
  policy:
    Statement:
    - Action: ssm:GetParameter*
      Effect: Allow
      Resource: '*'
    Version: "2012-10-17"

The added benefit is that it allows to make dynamic changes to the content (see "Effect": "{{ printf "%s" "Allow" }}" above as an simple example).

Now lastly, if you want to obtain a serialized value for policy from dictionary input, so far the only option is to define the dictionary in YAML elsewhere and reference it with _HT* and add a serialization instruction as in _HT*toPrettyJson|hull.config.specific.inlineJson.

This method has the advantages:

• the source dictionary can be defined hassle-free in the regular YAML form
• it also allows dynamic manipulation of the content
• the source dictionary can be referenced to in multiple places, potentially using it non-serialized or serialized in different forms for different purposes.

Downside is that you need to define the source dictionary under hull.config.specific seperately which might become a scaling problem. I will leave out this example because I think this is what you got in place, I can provide one however if you like. At the moment this is the idiomatic option to do this.

Documentation around serialization options can be found here if you do a find serialize on the page - but you are correct that it could be more prominently placed.

But I think you are actually after a different thing here: serializing inline definitions without reusage of the content elsewhere. I did a quick implementation test whether I can make a combination of _HT! and serialization instructions work and it should work. So by doing this:

- name: interpreted-and-serialized-flow-style-yaml
  policy: |-
    _HT!toPrettyJson|
      {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Effect": "{{ printf "%s" "Allow" }}",
            "Resource": "*",
            "Action": "ssm:GetParameter*"
          }
        ]
      }

I was able to produce this (yet unpolished) output:

    - name: interpreted-and-serialized-flow-style-yaml
      policy: |2-

          {
            "Version": "2012-10-17",
            "Statement": [
              {
                "Effect": "Allow",
                "Resource": "*",
                "Action": "ssm:GetParameter*"
              }
            ]
          }

If you would prefer such a possilibility I can look into cleanly implementing it as a feature, I do see some usecases for this.

Let me know if you think that is a better option for you than what you have currently and if your questions are answered sufficiently, thank you.

[eshepelyuk]

Hello @gre9ory

Thanks for the great explanation, as you're usually providing.
I do agree my example was a bit misleading, because my use case was more around rendering dynamic inline JSON ( using helm templating). So your assumptions were correct.

But I think you are actually after a different thing here: serializing inline definitions without reusage of the content elsewhere. I did a quick implementation test whether I can make a combination of _HT! and serialization instructions work and it should work.
If you would prefer such a possilibility I can look into cleanly implementing it as a feature, I do see some usecases for this.

Yes, you're right, I was looking for this, how to render inline JSON without defining temporary variables.
But if this syntax works _HT!toPrettyJson| - then no need to implement anything new - just update docs, am I wrong ?

[gre9ory]

Sorry, maybe I wasn't clear. That functionality is not available right now,

I did a quick test implementation to see if it would work and first impression judging on the outcome was that it looked pretty good and should be doable.

So right now you can do these serializations for _HT*toJson|... get and _HT/toJson|... include transformations.

I did not initially think about your use case but as said it makes sense so I will try to integrate it so that _HT!toJson|... etc also works. Gonna take a couple of days probably with tests and all.

[eshepelyuk]

Thank you.

[gre9ory]

This took longer than expected but it made me learn, discover and reevaluate a few things which makes the overall product better so it was worth it.

The problem I had was with returning and serializing arrays but after a while I found the solution: the Helm function fromYamlArray! Maybe it wasn't available last time I worked on this but it is godsend to interpret array input correctly.

Discovering fromYamlArray also resolves a fundamental problem with the current _HT/ include usage. It was not capable of returning arrays directly, you alway needed to wrap them in a dictionary and refer to the key in the _HT/some_key/... fashion.

Now you can just use _HT! and return any array (also including templating of course) directly such as this:

test-tpl-list: |-
  _HT!
    [
      "a",
      "b",
      {"c":"d"}
     ]

The same works for _HT/ if that include would return just

[
  "a",
  "b",
  {"c":"d"}
]

or any other array. Nice :)

Also took some time to restructure and cleanup the transformation documentation. Maybe you find it interesting to read if it makes (more) sense.

---

Regarding your case, the following should now work correctly:

- name: interpreted-and-serialized-flow-style-yaml
  policy: |-
    _HT!toPrettyJson|
      {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Effect": "{{ printf "%s" "Allow" }}",
            "Resource": "*",
            "Action": "ssm:GetParameter*"
          }
        ]
      }

As an bonus I found that it even works with block style input - but only when serializing the result during the process:

- name: interpreted-and-serialized-block-style-yaml
  policy: |-
    _HT!toPrettyJson|
    Version: "2012-10-17"
    Statement: 
    - Effect: "{{ printf "%s" "Allow" }}"
      Resource: "*"
      Action: "ssm:GetParameter*"

But please note that block style syntax is generally not supported! It only (seems) to work when paired with a serialization command and still has the known pitfalls due to indentation and white space chomping. So the general and safe advice is to use flow style syntax.

Will inform here when release(s) have been built soon.

[eshepelyuk]

Great news, appreciated for update.
I am assuming this is gonna be released soon ?

[gre9ory]

@eshepelyuk

HULL 1.31.2 is released.

Releases for 1.30 and 1.29 will follow in the next days.

Let me know if it works for you as expected. Thanks.

[eshepelyuk]

Sure, as soon as 1.30.x released.

[gre9ory]

Here is 1.30.7

[eshepelyuk]

Confirmed, works as described !
Thank you