Failed test does not automatically cause the dependent test to fail

[alekdavis]

I can't find the conversation, but I think I read that if test B references test A, and test A fails, test B should be automatically skipped (so no need to use the $cancel variable). It was not clarified what a failure means, but I assume that the test fails if (1) at least one of the assertions fails or (2) if the request call fails altogether. If my understanding is correct, then it does not seem to be the case.

EXAMPLE:

### Test X
# @name X
GET https://httpbin.org/json

?? status == 201

### Test Y
# @name Y
# #ref X
GET https://httpbin.org/anything

?? status == 200

RESULTS:

---------------------

=== Test X ===

GET https://httpbin.org/json
=> 200 (379 ms, 607 B)
✖ status == 201 (AssertionError [ERR_ASSERTION]: status (200) == 201 - c:\Users\xxx\.vscode\extensions\anweber.vscode-httpyac-6.15.1\dist\extension.js:177:36504)

---------------------

=== Test Y ===

GET https://httpbin.org/anything
=> 200 (745 ms, 562 B)
✓ status == 200

P.S. While on the topic, is there a way to not print the - c:\Users\xxx\.vscode\extensions\anweber.vscode-httpyac-6.15.1\dist\extension.js:177:36504) part on assertion failures?

[AnWeber]

@alekdavis Yes, I thought that ref were already skipped. But I didn't implement it correctly. Behavior now is that on error or when ref is skipped, the actual region is marked as skipped.

P.S. While on the topic, is there a way to not print the - c:\Users\xxx.vscode\extensions\anweber.vscode-httpyac-6.15.1\dist\extension.js:177:36504) part on assertion failures?

I have now removed the file output if you use the ? syntax. When using the script notation, it should point to the respective code line, which makes sense in my eyes.

[alekdavis]

@AnWeber I'm sorry, I do not follow.

Behavior now is that on error or when ref is skipped, the actual region is marked as skipped.

Are you saying that tests are marked as skipped when they should be marked as failed? Maybe it's the case (I haven't noticed), but my question is not about marks. I am asking if, in a scenario when testY references testX and an assertion of testX fails or if the request of testX fails due to an exception, e.g. network issue or whatever, will testY be skipped automatically (assuming it does not have the $cancel variable set to true in the pre-request script)? I think you mentioned that $cancel is not required in case a referenced test fails, but I don't see this this to be the case, unless I misunderstood the criteria of the test failure.

I have now removed the file output if you use the ? syntax. When using the script notation, it should point to the respective code line, which makes sense in my eyes.

What is "script notation"? Do you mean ?? js vs ??. IMHO, syntax should not matter. As an end user, I can't think of a case, when I would need to know the details of the extension source code for a normal assertion failure. If there were some syntax error or another unexpected exception, maybe, even though it gives me no useful info (since I am not the extension developer), but it could be helpful for submitting an issue. For a standard assertion failure, what's the point?

Assertion failures are part of the normal testing process. I just need to know if the assertion was successful or, if it failed, what assertion condition was not met. That's it. Well, the line of code where I invoke the assertion (in my http/rest file) could be helpful (can make it easier to find a failed test), but I have no interest in knowing which line of the extension source code caught the assertion failure. It's just noise. If there is a use case I'm missing, maybe have it show when the log level is higher DEBUG or higher?

Sorry about asking so many questions. I'm preparing a presentation for a cross-team demo and need to clarify a couple of things, so I do not present wrong info.

[AnWeber]

If testX fails, testY is not run, so it is skipped. This seems to be the best/ expected result after only runnig testY

what is script notation

You can also write the tests in javascript. Using ?? should only remove some boilerplate and simplify the code.

GET https://postman-echo.com/get

{{
  const { equal } = require('assert');
  test('status code 200', () => {
    equal(response.statusCode, 200);
  });
}}

If you write your tests with javascript, it is helpful if you know which line in your tests fails.

Sorry about asking so many questions. I'm preparing a presentation for a cross-team demo and need to clarify a couple of things, so I do not present wrong info.

Always happy to help. Unfortunately, it's just getting a bit lost as I have a “fun” job at the moment. So forgive me if I'm a bit behind in replying.

[alekdavis]

@AnWeber I think you are doing far and beyond what could be expected, so thank you so much for being patient and replying to the questions. I really appreciate it. And please excuse me for posting often. I'm trying to get httpYac adopted across teams and need to understand a few things, so I can answer questions internally (I have a couple more I need to clarify, but I'll give a break for a few days ;-)).

Getting back to the original issues. I'm really sorry, but I still am confused. And even more than before. :-)

TEST DEPENDENCY

If testX fails, testY is not run, so it is skipped. This seems to be the best/ expected result after only runnig testY

Are you saying that the behavior is different depending on whether testY is executed alone or as part of a batch? Regardless, I just tried and it does not seem to work like you say it should. Unless I am misunderstanding you. Just to clarify: you are saying that if testX fails, testY will be skipped, right? But it is NOT skipped. Here I run testY alone:

Is the screenshot shows, it is NOT skipped. TestY is also NOT skipped when I execute the whole HTTP file (with two tests):

Am I totally misunderstanding what you are trying to explain because what I see is not what you seem to be claiming to happen. Are you saying that on your system testY is skipped? Because on mine, it is not, as you can see from the screenshots.

LINE NUMBER

If you write your tests with javascript, it is helpful if you know which line in your tests fails.

I agree 100%. Except that (1) as you can see from the screenshot, the error message does not refer to my code, it refers to the source code of the extension, which is meaningless to me and (2) I am not even using JavaScript tests, I get the line number error from the ?? status assertion.

[AnWeber]

I changed behaviour with v6.16. now it should get skipped

[alekdavis]

I just installed v6.16, but the behavior (for the skipping, or should I say NOT skipping) is still the same.

I did notice that the error message no longer includes the extension file line number. Looks cleaner now. THANK YOU!

[AnWeber]

you have a typo at using ref

# @ref X

[alekdavis]

you have a typo at using ref

Ah, right, got a wrong screenshot. But it still does not work with @ref. It does work with @forceRef, though, but I assume that both should work the same.

HttpYacRef.mp4

[AnWeber]

Now I understand the problem. If I just run Y, it works correctly the first time. The second time it would no longer execute because of @ref X, since the X has already generated a response and this is now being used. That's why forceRef also works, because X is always executed again here.
The solution would be to delete the response variable again, or only set it if all TestResults were successful in the end. Updated with v6.16.2

[alekdavis]

OK, looks like v6.16.2 fixed it for `@ref``. Thank you!