caddyfile: Pass blocks to import for snippets

[elee1766]

Snippet import blocks

This PR allows the usage of blocks as input arguments when invoking snippets, which can be resolved by placeholders within the snippet.

one can use key-value pairs as arguments, and refer to them by name in the snippet. the intention of this feature is to add two capabilities to caddyfile

1. named arguments for snippets arguments, for order independent parameters
2. inversion of control for snippet imports.

examples

For instance, this config

(snippet) {
	header {
		{blocks.foo}
	}
	header {
		{blocks.bar}
	}
}
example.com {
	import snippet {
		foo {
			foo a
		}
		bar {
			bar b
		}
	}
}

will render the same as

example.com {
	header {
		foo a
	}
	header {
		bar b
	}
}

as one can see, the tokens within the block labeled foo will populate the {blocks.foo} placeholder in the snippet.

Along with this, one can use the placeholder {block} to substitute it with the entire block passed to the snippet
For example, the example below

(snippet) {
	header {
		{block}
	}
}
example.com {
	import snippet {
		foo bar
	}
}

will render the same as

example.com {
	header {
		foo bar
	}
}

as one can see, the entire block passed into the snippet will populate the {block} placeholder in the snippet.

---

---

---

Original PR (inaccurate, but kept for posterity)

as the amount of caddyfiles and snippets in our configuration grows, we have found it might be useful to be able to declare something like an {outlet} which spits out a configured block in the import statement.

The benefit is that it allows us to create snippets which can serve as entrypoints that must contain a superset. this is makes the configs less prone to error, as the rule becomes to use the snippet, instead of to not forget the import.

for instance, imagine a config like this:

(default_config) {
  foo bar
  bar foo
  one two 
}

:3838 {
  route / {
    module {
      import (default_config)
      extended_one one
      extended_two 2
    }
    respond "hello"
  }
}

after:

(module_func) {
    module {
      foo bar
      bar foo
      one two 
      {outlet}
    }
}
:3838 {
  route / {
    import module_func {
      extended_one one
      extended_two 2
    }
    respond "hello"
  }
}

my snippet, module_func, is now able to act as a default configuration for module. downstream users can be told ONLY to use module_func. This will avoid the mistake of ever using module without import default_config.

maybe this is a feature you guys might also find interesting?

as import statements currently don't consume blocks at all, i dont think it would break anything. it would allow for some more contrived caddyfiles (is that a bad thing)

i've made this a PR as we were testing this out with our config, and those few lines in the code made it possible - but maybe im missing something and this is really dangerous and there's a good reason the feature doesn't exist.

we refactored parts of our config to use it and it does seem safer - doing X is much easier than not forgetting Y.

[francislavoie]

Interesting idea. I could see the value in that, since right now only simple arguments are supported.

I think this is still quite limited, and there's room for making it even more powerful. Also, I don't like the "outlet" name, doesn't feel like it fits well with Caddyfile conventions.

I only spend like two minutes thinking about it, but I could see something like this being possible; since we already have {args[*]}, we could introduce {block} and {block.*}. The former is basically exactly what you have. But {block.*} would allow for "named blocks" to be used, so you could have more than one "outlet". I'm taking inspiration from the "slots" pattern in React's JSX.

(snippet) {
	module_one {
		foo
		{block.one}
	}
	module_two {
		{block.two}
	}
}

:8080 {
	import snippet {
		one {
			foo bar
		}
		two {
			bar baz
		}
	}
}

Producing 👇:

:8080 {
	module_one {
		foo
		foo bar
	}
	module_two {
		bar baz
	}
}

The implementation may be tricky, means having to reach into the block if we encounter blocks.* to grab the segment marked by that name.

[elee1766]

Interesting idea. I could see the value in that, since right now only simple arguments are supported.

the limitation is as you point out - we are currently able to pass []string into snippets, so i was looking for a way to provide []Token. Being able to pass tokens is what opens up a lot of windows, since now snippets can become almost like an inline ast transformer.

I think this is still quite limited, and there's room for making it even more powerful. Also, I don't like the "outlet" name, doesn't feel like it fits well with Caddyfile conventions.

I only spend like two minutes thinking about it, but I could see something like this being possible; since we already have {args[*]}, we could introduce {block} and {block.*}. The former is basically exactly what you have. But {block.*} would allow for "named blocks" to be used, so you could have more than one "outlet". I'm taking inspiration from the "slots" pattern in React's JSX.

"blocks" makes more sense to me. (funny, outlet was taken from a react project as well https://reactrouter.com/en/main/components/outlet )

i think multiple named blocks makes sense. for instance, if args supplies a []string, then blocks would provide {string: []Token}, with an omitted name being available at {blocks.}.

The implementation may be tricky, means having to reach into the block if we encounter blocks.* to grab the segment marked by that name.

another option would be arrays, ala

(snippet) {
	{blocks[0]}
	{blocks[1]}
}
import snippet {
	{
		bar
	}
	{
		foo
	}
}

however, i'm not convinced that this makes implementation any easier.

[francislavoie]

"anonymous" blocks is not a convention we support, except for global options specifically as a special case. So I don't think numerically indexed blocks like that is the right approach.

Of course "naming things" is always hard (often tricky with named matchers, and even snippet names, what should they be called?) but it's definitely more flexible/scalable than numeric indexes, and less confusing to read because the name can carry some intent.

One thing to think about is we did deprecate {args.*} in favour of {args[*]} recently because [] lets us do fun things like [1:] and whatnot. So because of that, I feel like {blocks.*} might feel incongruous? I don't think {blocks[foo]} makes sense though, the [] syntax only really works with numbers as indexes.

So yeah my thoughts right now are {block} (singular) and {blocks.*} (plural) as the syntax.

[elee1766]

ah, i didn't know unnamed blocks were a no-no.

some more things i thought of:

1. should nested keys should exist? for instance -

import snippet {
    foo {
       bar {
          baz 
       }
    }
}

{blocks.foo.bar} -> [baz]
{blocks.foo} -> [bar {\n baz\n }\n]
{blocks.*} -> [foo {\n bar {\n baz\n }\n }\n]

imo, if . will have special meaning - it might be prudent to ensure that caddy will error if keys contain ., so that this feature can be implemented in the future, if not immediately.

2. why just blocks? i think we could also allow tokenizing the remaining args of the line.

import snippet {
    foo hello world
}

{blocks.foo} -> [hello world]
{blocks.*} -> [foo hello world]

3. it feels weird to have {block} and {blocks.*} when {blocks.foo} can be thought of as short fo {blocks.foo.*} so perhaps the block containing all tokens should be {blocks.*} and not {block}

import snippet {
    foo hello world
}
{blocks.*} -> [foo hello world]
{blocks} -> [foo hello world]

4. since the parsed blocks are just arrays of tokens, could [] syntax be used to index the tokens of the matched block? this also doesn't need to be implemented immediately, it seems pretty niche, but it might be prudent to forbid "[" and "]" in keys regardless

import snippet {
    foo hello world
}

{blocks.foo[1]} -> [world]

[francislavoie]

should nested keys should exist

I don't think so, that's out of scope. It's not import's responsibility to deal with nesting. We want to keep it simple.

{blocks.*}

I don't think we'd want {blocks.*} to literally be a thing. You'd either use {block} or {blocks.foo}. Trying to splat all blocks doesn't make sense IMO, just use {block} for that.

imo, if . will have special meaning - it might be prudent to ensure that caddy will error if keys contain ., so that this feature can be implemented in the future, if not immediately.

Not really a concern, . is conventional in placeholder keys. Just use \. to escape a dot. But either way, no nesting is allowed for this IMO so it doesn't matter.

why just blocks? i think we could also allow tokenizing the remaining args of the line.

It's not just blocks, you will still be able to use the args on the same line, backwards compatible. It doesn't make sense to have two sets of args.

[elee1766]

(snippet_block) {
    basic_auth {
      {block}
    }
}
(snippet) {
    basic_auth {
      {blocks.users}
      {blocks.admin}
    }
}

# pw: hiccup
:3838 {
  route / {
    import snippet {
      users {
        user $2a$14$Zkx19XLiW6VYouLHR5NmfOFU0z2GTNmpkT/5qqR7hx4IjWJPDhjvG
      }
      admin admin $2a$14$Zkx19XLiW6VYouLHR5NmfOFU0z2GTNmpkT/5qqR7hx4IjWJPDhjvG
    }
    respond "hello"
  }
  route /block {
    import snippet_block {
      block $2a$14$Zkx19XLiW6VYouLHR5NmfOFU0z2GTNmpkT/5qqR7hx4IjWJPDhjvG
    }
    respond "hello"
  }
}

this simple example works as intended! so it's a start

i'm parsing by just putting the tokens (for handling {block}) into a newly made dispenser. that's the only real hack and the rest seems to play nicely?

there's no logic for nested. i extract the key by doing strings.TrimPrefix(strings.TrimSuffix(token.Text, "}"), "{blocks.")

[elee1766]

https://github.com/caddyserver/caddy/tree/master/caddytest/integration/caddyfile_adapt

should tests for this be created here? if i create new .caddyfiletest in the directory, will it just work tm

[francislavoie]

Yep, files in there are automatically run.

[elee1766]

created some simple tests, block, blocks, nested snippet with name conflict. let me know if you have any other thoughts, i can try to make some more

[elee1766]

@francislavoie

i was wondering if i needed to do anything to move this along. if not, thats fine, just not sure if i should be doing anything other than merging master and making sure it continues to work and build

[mholt]

Apologies from my end; you're doing great work, and this is a good contribution; I am just behind. I have less than 2 pages left on my notifications backlog though so I am getting around to it!

Francis I'm sure will also get around to it eventually but between any of us maintainers, we will address this :)

[mholt]

This is still on my list just so you know! My list is much smaller now, and this is near the top. Now we're getting ready for the 2.8 release, and this might have to come after 2.8, but it'll be one of the first new features we circle back to in that case. 👍

[elee1766]

@mholt do you want me to keep updating the branch and making sure things are good, or should i wait you to tell me its back on your docket

[mholt]

@elee1766 Thanks for asking! I won't be able to get back to this until after 2.8 (which is nearly ready, RC is going out in the next few days and then we'll tag 2.8 when we're comfortable doing so). So while another rebase may be required before we merge this, I'll let you know when we're ready for that so you don't keep rebasing over and over.

[elee1766]

@mholt sounds good thank u. excited for 2.8! we have been super happy with caddy in production with the new fs directive

[mholt]

This looks good to me, though I haven't thoroughly vetted it. I appreciate the several test files. @francislavoie What do you think? Shall we merge?

[francislavoie]

Yeah I think this is good as well. Pretty straight forwards I think.

[francislavoie]

We'll need to update the documentation as well on the website repo (anyone can do that). I think it would be good to clean up the top comment on the PR cause surely people will land here when wanting to learn how the feature works and may be confused by the now-misleading original implementation description

[elee1766]

We'll need to update the documentation as well on the website repo (anyone can do that). I think it would be good to clean up the top comment on the PR cause surely people will land here when wanting to learn how the feature works and may be confused by the now-misleading original implementation description

i can do both.

where is the correct place to update and PR for the website?

[mholt]

At https://github.com/caddyserver/website

When you're ready @elee1766 , go ahead and convert this to "Ready to review" (we can do it for you but I want to make sure you're ready first).

[elee1766]

@mholt i marked it as ready, but the real answer is i don't know, and i don't really know if anyone knows. im pretty sure the block placeholders not resolving when there is no match should not break any existing configurations.

thinking about this also ran me into a bit of a tangent.

whenever i work on caddy, I always wonder if there is more I should be doing in terms of testing/integration/regression. except then i look back at the repo and how things have been, and decide that it's acceptable in terms of the project historically. i wasn't super scared with the filesystem directive because it was already labeled experimental, this one i am a little more concerned with, but we have been running this fork on production for months using rather complicated snippet blocks, so if there are issues, i would hope to not see them in standard use.

which did lead me to a question, that is - is the only reason that there aren't proper regression suites, e2e testing, unit tests, that nobody is putting in the time? I remember reading the blog article about the want for a feature freeze.

both our api gateway and static frontends are now served by caddy, that is to say, all interaction between our users and our services is done through caddy instances. we have somewhat complex routing rules, use live config loading and reloading, proxy protocol, custom plugins, http2 and 3, custom tracing, telemetry, etc. as a result, we already have to do some of our own testing testing whenever we update caddy ourselves, and we are only making our testing more and more robust+automated as we depend on caddy more for critical infrastructure. coming from the crypto industry, we are rather used to needing to have internal infrastructure for testing irreplaceable open source software, which regularly, like caddy, can have many many patch releases.

what im trying to say is - if the answer really is that other people have no time and we aren't at risk of doing double work with anyone - we would probably be happy to develop more robust testing for caddy publicly instead of just internally. the advantages of having more eyes on it, and the testing being more robust, would more than pay for our engineering hours; it's very easy to justify for us.

[elee1766]

my long winded rant aside - caddyserver/website#400

[francislavoie]

Yeah largely time has been the issue re testing, but also lack of energy to dedicate to it when we're also maintaining and adding new features. The only full time person on Caddy is Matt right now (and he's much busier with life than before) and the rest of us are volunteers with full time jobs.

Testing infrastructure isn't one of my strong suits personally so I don't have so many creative ideas in that direction, I don't have the motivation to work on testing myself. I know @mohammed90 has a lot of good ideas for how we could develop portable test suites but we're lacking time to bring it to fruition I think.

So yeah we'd love all the help we can get. I think Matt could probably invite you to our Slack so we can talk about it further if you want to lend a hand! 😊

[elee1766]

Yeah largely time has been the issue re testing, but also lack of energy to dedicate to it when we're also maintaining and adding new features. The only full time person on Caddy is Matt right now (and he's much busier with life than before) and the rest of us are volunteers with full time jobs.

Testing infrastructure isn't one of my strong suits personally so I don't have so many creative ideas in that direction, I don't have the motivation to work on testing myself. I know @mohammed90 has a lot of good ideas for how we could develop portable test suites but we're lacking time to bring it to fruition I think.

So yeah we'd love all the help we can get. I think Matt could probably invite you to our Slack so we can talk about it further if you want to lend a hand! 😊

yeah, i would be happy to help! if slack is where you guys talk, [email protected] should have a slack account attached to it.

i have some experience in testing; for the public i wrote a small framework for e2e+regress testing the eth2 http. A sort of portable test suite is exactly what i was imagining. importantly, i want to make sure that we are able to test not only caddy, but also use the same framework to test caddy using the same framework with a fork (im calling plugins forks, because for all intents and purposes, they are)

[mohammed90]

@elee1766, for testing, check my proposed approach in #6255.

[mholt]

@elee1766 Ah, yeah, I see what you mean. Thanks for writing that up and expressing your feelings and thoughts about the matter.

When I initiated the feature freeze early last year, I planned on devoting several months (about 6?) to improving the testing situation for the project. But shortly after the freeze, my wife's pregnancy got complicated and she had to be hospitalized for an extended time, which threw a lot of things out of whack. The baby came two months earlier than expected (and the hospital caught fire) so it turned things upside-down and I got so far behind on the regular backlog that it didn't make sense to delay the next Caddy release 6 more months, so I ended the feature freeze early to clear out the backlog.

Since then, open source issues/PRs and sponsor work has kept me busy (a good thing!) and ultimately I haven't had time to lead a testing overhaul effort.

I could definitely do this with a sufficient sponsorship from a company that needs this, or perhaps someone else could lead it. I started writing this reply last night and now I see there have been some new comments and it looks like you're willing to help. I'll send you a Slack invite if you're not already in there and we can chat easier that way with the team.

Would be thrilled to get something put together :) Thank you for participating!