[Discussion] Tests

[Anuril]

Hi Everyone,

The zammad_py package uses the vcr test suite to record and play back HTTP interactions with an instance of Zammad. While this approach has worked well for testing existing functionality, it presents a challenge when it comes to testing new endpoints and features. Since not everyone has access to a "clean-slate" instance of Zammad, it can be difficult to extend the test suite and ensure that the package is working correctly.

I wanted to start a discussion to see if we could come up with some solutions to this problem. One possible approach I thought about would be to use a mock server like WireMock to simulate the behavior of Zammad. With WireMock, it would be possible to create a mock Zammad API that can be used for testing new endpoints and features, without requiring access to a live instance of Zammad.

Another approach would be to create a test Zammad instance that contributors could request access to for testing purposes. This instance would be reset every week to ensure that it remains a clean slate for testing. With this approach, contributors could request access to the test instance and use it to run the vcr test suite and test new features and endpoints. While this approach would require some setup and maintenance, it would provide a live instance of Zammad for testing purposes, without requiring users to have their own installation or rely on a mock server.

A third, (IMHO less desirable) option would be to use classic mock objects with unittest.mock or pytest.mock. While this would be a more labor-intensive option and may not provide as comprehensive testing as the other two options, it could be a good choice for contributors who are familiar with these libraries.

What do you think? Have you used WireMock or other mock technologies in other projects? Which approach do you think would be a better solution for the zammad_py package?

I'm interested in hearing your thoughts and feedback on these ideas.

Best Regards,
Chris

[joeirimpan]

Hi @Anuril ,
I am not familiar with WireMock. VCR was the easiest way to setup integration tests with a local docker test instance of Zammad. Like you said, setting up a dedicated instance for this lib is a total overkill. For the small wrapper lib like zammad_py, a basic testsuite with unit + integrations will suffice.

[Anuril]

Thanks for sharing your thoughts on the matter.

VCR was the easiest way to setup integration tests with a local docker test instance of Zammad.
setting up a dedicated instance for this lib is a total overkill.

I agree and I can see why the current setup with VCR and docker was a probably the easiest way to set it up.

However, with how VCR is currently set up, a unified setup for all developers would be required, as the test config & cassettes are shipped with the code. This requires either an instance running on localhost on port 80, or manually searching & replaceing the url strings in the test cassettes before pushing the code.

I think this is a limiting factor, that's why I am trying to find an alternative testing approach that might be more accessible, easily replicated and less prone to errors.

I guess - for starters, we could use a .env file or something like that to save the url / credentials that are used, so the tests can be run, but that would still not solve the problem of the cassettes later containing the wrong url for the tests done by the CI. Maybe a default value for the url could be used, and we could add the searching & replacing to the end of the test suite to make it repeatable.