Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve Then steps in API acceptance tests #39512

Closed
phil-davis opened this issue Nov 22, 2021 · 5 comments
Closed

Improve Then steps in API acceptance tests #39512

phil-davis opened this issue Nov 22, 2021 · 5 comments
Labels
QA:team

Comments

@phil-davis
Copy link
Contributor

phil-davis commented Nov 22, 2021 •
edited
Loading

We run the core API acceptance tests in various different environments including:

  • regular owncloud/core
  • oC10 apps (for example, user_ldap, files_primary_s3, admin_audit, activity, password_policy...) where significant different backend components, hooks, events happen for those apps, and there can potentially be any sort of regression of behavior.
  • owncloud/ocis which has a completely different server-under-test
  • cs3org/reva which has a completely different server-under-test

When steps are expected to run just the When action, and not fail. The later Then steps in the scenario should check the result of the When step. We do this because:

  • sometimes a When step is expected to sometimes give a "bad" HTTP status. For example, if dome with a valid user the API request returns 2xx, if done with a non-existent user it gives a 4xx. So the When step code cannot and should not guess if a 4xx is "a bad thing".
  • in API tests we want to be clear about what are the expected results of the When action. So writing out the expected HTTP status, and something about the expected request body etc in separate Then steps is useful for people who read the scenario. (rather than hiding this inside the test code as "assumed knowledge")

Issue owncloud/ocis#1420 has a discussion about this. In that case the scenario did not have Then steps to check the HTTP status. The test for public link access failed, and it was not clear that actually the When step failed completely.

ToDo: add Then steps to existing API test scenarios to better check the results of When steps.
@phil-davis
Copy link
Contributor Author

Whoever starts this, please do an initial PR that just adjusts a single API feature file. Then we can all review and discuss to make sure we choose a "best" way forward.

This was referenced Nov 22, 2021
Closed
Closed
@phil-davis
Copy link
Contributor Author

Note: there are some simple examples done for disable-user tests at commit 6e5eb23

@phil-davis phil-davis mentioned this issue Dec 21, 2021
Merged
11 tasks
@kiranparajuli589
Copy link
Contributor

@grgprarup is working on this

@grgprarup
Copy link
Contributor

grgprarup commented Feb 15, 2022 •
edited by SagarGi
Loading

API acceptance tests suites to cover:

A single PR for a single suite would be greatly appreciated.

@grgprarup grgprarup mentioned this issue Feb 15, 2022
Merged
11 tasks
@SagarGi SagarGi mentioned this issue Feb 17, 2022
Merged
11 tasks
@sushmita56 sushmita56 mentioned this issue Feb 17, 2022
Merged
10 tasks
@sushmita56 sushmita56 linked a pull request Feb 17, 2022 that will close this issue
[tests-only][full-ci]added then steps for apiMain suite #39802
Merged
10 tasks
@sushmita56 sushmita56 removed a link to a pull request Feb 17, 2022
[tests-only][full-ci]added then steps for apiMain suite #39802
Merged
10 tasks
@SagarGi SagarGi mentioned this issue Feb 18, 2022
Merged
11 tasks
@grgprarup grgprarup mentioned this issue Feb 21, 2022
Merged
11 tasks
@SagarGi SagarGi mentioned this issue Feb 21, 2022
Merged
11 tasks
@grgprarup grgprarup mentioned this issue Feb 22, 2022
Merged
11 tasks
@SagarGi SagarGi mentioned this issue Feb 24, 2022
Merged
11 tasks
@sushmita56 sushmita56 mentioned this issue Feb 24, 2022
Merged
10 tasks
@grgprarup grgprarup mentioned this issue Feb 28, 2022
Merged
11 tasks
@SagarGi SagarGi mentioned this issue Feb 28, 2022
Merged
11 tasks
This was referenced Feb 28, 2022
Merged
Merged
This was referenced Mar 2, 2022
Merged
Merged
@sushmita56 sushmita56 mentioned this issue Mar 4, 2022
Merged
10 tasks
@SagarGi SagarGi mentioned this issue Mar 7, 2022
Merged
11 tasks
This was referenced Apr 4, 2022
Merged
Merged
Merged
@grgprarup grgprarup mentioned this issue Apr 5, 2022
Merged
11 tasks
@SagarGi SagarGi mentioned this issue Apr 6, 2022
Merged
11 tasks
This was referenced Apr 8, 2022
Merged
Merged
@SagarGi SagarGi mentioned this issue Apr 11, 2022
Merged
11 tasks
@amrita-shrestha amrita-shrestha mentioned this issue Apr 11, 2022
Merged
11 tasks
@SagarGi SagarGi mentioned this issue Apr 12, 2022
Merged
11 tasks
@amrita-shrestha amrita-shrestha mentioned this issue Apr 13, 2022
Merged
11 tasks
@SagarGi SagarGi mentioned this issue Apr 13, 2022
Merged
11 tasks
This was referenced Apr 19, 2022
Merged
Merged
This was referenced Apr 20, 2022
Merged
Merged
@kiranparajuli589 kiranparajuli589 mentioned this issue Apr 22, 2022
Merged
@SagarGi SagarGi mentioned this issue Apr 25, 2022
Merged
11 tasks
@sushmita56 sushmita56 mentioned this issue Apr 26, 2022
Merged
10 tasks
@amrita-shrestha amrita-shrestha mentioned this issue Apr 27, 2022
Merged
11 tasks
@kiranparajuli589 kiranparajuli589 mentioned this issue Apr 27, 2022
Merged
11 tasks
@SagarGi SagarGi mentioned this issue Apr 27, 2022
Merged
11 tasks
@grgprarup grgprarup removed their assignment Apr 28, 2022
@SagarGi SagarGi mentioned this issue Apr 28, 2022
Merged
11 tasks
@SagarGi SagarGi removed their assignment May 3, 2022
@kiranparajuli589
Copy link
Contributor

finally done! great work @SagarGi, @amrita-shrestha, @grgprarup and @sushmita56 👑 👍

@amrita-shrestha amrita-shrestha mentioned this issue Aug 22, 2023
Closed
6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
QA:team
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@phil-davis @grgprarup @kiranparajuli589 @SagarGi @amrita-shrestha @sushmita56