Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve read operation performance of azuredevops_users data source #939

Merged
merged 5 commits into from
Jan 2, 2024

Conversation

rdalbuquerque
Copy link
Contributor

All Submissions:

  • Have you added an explanation of what your changes do and why you'd like us to include them?
  • I have updated the documentation accordingly.
  • I have added tests to cover my changes.
  • All new and existing tests passed.
  • My code follows the code style of this project.
  • I ran lint checks locally prior to submission.
  • Have you checked to ensure there aren't other open PRs for the same update/change?

What about the current behavior has changed?

While using azuredevops_users data source to fetch all users, or even svc subject type users, the reading operation can take too long depending on the number of users on the organization.

So to improve performance, it was added a features block with concurrent_workers argument so the user can set concurrent processing for the user data source, greatly decreasing the time it takes to finish read operation.

Does this introduce a change to go.mod, go.sum or vendor/?

  • Yes
  • No

Does this introduce a breaking change?

  • Yes
  • No

Any relevant logs, error output, etc?

Without concurrent workers:

image

With concurrent_workers set to 10:

image

Other information

azuredevops_users data source with features block:

data "azuredevops_users" "users" {
  features {
    concurrent_workers = 10
  }
}

@rdalbuquerque rdalbuquerque changed the title Fix user data source Improve read operation performance of azuredevops_users data source Dec 13, 2023
@xuzhang3 xuzhang3 merged commit 0a8010e into microsoft:main Jan 2, 2024
3 checks passed
@drdamour
Copy link
Contributor

pretty concerned this can make things even worse..the underlying problem with azuredevops_users is it pulls every user in cases you only want 1

@rdalbuquerque
Copy link
Contributor Author

@drdamour on my use cases it either drastically improved performance (like the print on this PR) or stayed the same, and default is a single go routine.

Feel free to contact me with any use case that this might be negatively impacting and we can discuss possible solutions.

@xuzhang3
Copy link
Collaborator

@rdalbuquerque @drdamour This can be controlled by adding a custom attribute to control the number of users that can be pulled. Similar like the azuredevops_variable_group, a custom property search_depth was used to control how many secrets can be searched.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants