Add examples / best practises on testing restate applications

[sdegroot]

One of the things I am missing in the documentation is how you recommend to test restate applications. For me proper and easy testing is extremely important but apart from this PR restatedev/examples#75 I couldn't find much on the topic.

Discussed this for a bit in Discord. Personally, I believe it would be best to have a fully featured TestContext. However, we're not there just yet.

Regardless, I'd really like to see some examples and best practises on how to test restate applications. Ideally, all examples in the repositories are provided with proper tests. Apart from that, it would be great if the documentation has a chapter on this topic where it is explained what options one has to test restate applications (and where you're heading with this).

I'd write something myself but I only ran my first restate application this week :)

[slinkydeveloper]

Hey @sdegroot , so far the suggestion is to use testcontainers and that PR you linked is the example we have so far for TS. In that example your code runs on the local node VM, so you'll be able to run it with the debugger enabled.
For Java we ship a little module that bundles testcontainers and our container setup, for TS we might want to do the same in the near future too.

I personally tried at some point to build something like a TestContext/MockContext, but always ended up re-building something like mockito in order to have a flexible mock that could serve as many testing cases as possible... Perhaps using something like https://github.com/NagRock/ts-mockito is worth a try?

[sdegroot]

Hi @slinkydeveloper, one question about the example from the pull request: where can I find it in the main branch? I can't find it anywhere but in the PR. Did it get deleted?

For the long run, I would really like to see testing to be a first class citizen in restate. Personally, I prefer to run my tests without the need to run test containers. For certain cases you will have to, but ideally for most you won't. And as for the mocking approach, it's a valid one but from a DX experience it's less than ideal. Maybe you could elaborate on why you ended up rebuilding something like mockito?

I'd really like to have restate provide a testing suite which makes it extremely natural to write tests. Things I'd expect to be in the testing suite would be dealing with the clock, dealing with promises, state and awakeables.

That being said, I'll have a look at ts-mockito. Thanks! :)

[slinkydeveloper]

Did it get deleted?

Probably yes when we reorganized the examples repo a month ago. We will re-introduce those soon.

And as for the mocking approach, it's a valid one but from a DX experience it's less than ideal.

I absolutely agree that both test containers and mockito are less than ideal, and what you describe would be an amazing DX. Unfortunately this requires either run some parts of the restate runtime logic within the SDK testing infra, or somehow duplicate that logic on each SDK test infra in every language. We even made an attempt at this a while ago in java, where we duplicated parts of the business logic of the runtime within the SDK test infrastructure, but it proved too hard to maintain, didn't fully replicate some of the runtime behaviors, and still missed some useful test assertions like "verify what the result of this side effect is" and so on. Unfortunately we don't have bandwidth for such a test infra at the moment, but we'll definitely keep this feedback in mind once we get to that.

Another direction we also have in mind is to have some features in the runtime itself that are useful for testing, and are enabled only when you do local development/testing, for example stuff like "fire manually a timer", "mock a response", etc. You would still need to run the runtime as a side process somehow (be it testcontainers, or some runner that runs it as child process together with jest/junit/etc), but this would definitely give you some useful testing primitives.

Maybe you could elaborate on why you ended up rebuilding something like mockito?

Because at some point I found myself adding for every ctx method run counts and when called closures, basically mockito :)

[sdegroot]

Unfortunately this requires either run some parts of the restate runtime logic within the SDK testing infra, or somehow duplicate that logic on each SDK test infra in every language

That would indeed be quite the effort. I have seen competing/similar products to Restate struggle with this as well.

nother direction we also have in mind is to have some features in the runtime itself that are useful for testing, and are enabled only when you do local development/testing

This would prevent you having to duplicate the logic and it'll be better than what's currently available. Creating new SDK's for different platforms would remain relatively simple.

I wonder though if there is anything you could do to decouple the restate client API from the runtime more to make it easier duplicating logic across SDK's. Just thinking out loud here, more meant as a rhetorical statement

--

Back to the topic though: guess the action points would be to add the examples with testing back into the repo and maybe write some words on it in the docs. Do you want to keep this issue open until then?

[slinkydeveloper]

Closing this for now, as we solved it for Typescript here: https://docs.restate.dev/develop/ts/testing