Support real-time execution reporters

[nvborisenko]

Plugin can handle SuiteExecutionResult message to get a test report. But plugin gets this report at the end of execution. What if we have loooong tests and want to see real-time report.

I propose to extend existing messages and add something like:

• SuiteExecutionStarted
• SpecExecutionStarted
• SpecExecutionEnded (with exact spec execution results)
• ScenarioExecutionStarted (with scenario info)
• ScenarioExecutionEnded (with scenario execution result)
• etc

[sriv]

We built something along those lines as a prototype: https://github.com/getgauge/flash

Is this similar to what you are looking for?

[nvborisenko]

It's similar, but not enough. I though that the purpose of these messages is for runners as documented on wiki. Like messages to run specific before/after hooks. I would like to get full information about entity when this entity starts/finishes. Currently I don't see a description of scenario/spec, or start/end time, I see only name and tags. Please correct me if I miss something

[sriv]

yes, I think what you seek is a little more than what is offered right now. We can work to improve that.

But I'd like to understand what kind of information would make sense? Is it enough to give what SuiteExecutionResult gives, except that it would be available at entity level (ex. Scenario result, after the scenario execution is complete) ?

Note : SuiteExecutionResult has some primitive fields, as well as some computed fields. The computed fields will only be made available after the execution.

[nvborisenko]

I am not familiar with gauge framework well, but I see the following:
When scenario starts:

• Name
• Description
• Tags
• Start time (this is exact time when scenario starts)
• ID (some unique ID of scenario to identify it, not necessary, we can use some comparer)
• SpecID (for understanding that this scenario belongs to specific spec)

When scenario ends:

• ID (identify which exact scenario ended. We know metainfo about this scenario because we handled ScenarioStarts event)
• Status
• Errors/Stackstraces
• End time

Generally we need everything about execution what framework can provide.

[ajoecker]

I support this... currently it seems that there is a only a limit grade of details given, which hinters developers to work with that information.
I would suggest to get the more open way and just transfer information as much as possible, so that people can retrieve any information they want.
@nvborisenko list is quite accurate for this

[nvborisenko]

Hello! Is there any update here?

[nvborisenko]

Awaiting gRPC support. It promises to be great feature for plugins.

[nvborisenko]

Trying to explore it: faced an issue getgauge/gauge-csharp-core#1, please help.