Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add NVTX support for performance analysis #144

Merged
merged 5 commits into from
Nov 30, 2021

Conversation

gigony
Copy link
Contributor

@gigony gigony commented Nov 9, 2021

Need to rebase once #141 is merged.

  • Instrumented important methods in cuCIM with NVTX so can see performance bottlenecks easily with NSight systems.
  • Tracing can be enabled through config file or environment variable or through API and less than 1% performance overheads in normal execution.

Enabling Tracing

Through .cucim.json file

{
        "profiler" : { "trace": true }
}

Through Environment variable

CUCIM_TRACE=1 python

Through API

from cucim import CuImage

CuImage.profiler(trace=True)
#or
CuImage.profiler().trace(True)

CuImage.profiler().config
# {'trace': True}
CuImage.profiler().trace()
# True
CuImage.is_trace_enabled # this is simpler method.
# True

Profiling with NVIDIA Nsight Systems

nsys profile -f true -t cuda,nvtx,osrt -s cpu -x true --trace-fork-before-exec true -o my_profile `which python` benchmark.py
# can add `--stats true`

Then, execute nsight-sys to open the profile results (my_profile.qdrep).

image

closes #137

@gigony gigony added the feature request New feature or request label Nov 9, 2021
@gigony gigony added this to the v21.12.00 milestone Nov 9, 2021
@gigony gigony self-assigned this Nov 9, 2021
@gigony gigony requested review from a team as code owners November 9, 2021 23:48
@gigony gigony requested a review from a team as a code owner November 9, 2021 23:48
@gigony gigony requested a review from jakirkham November 9, 2021 23:48
@gigony gigony force-pushed the support_nvtx branch 2 times, most recently from 60bc2ff to 8dddd15 Compare November 10, 2021 02:14
@gigony gigony added the non-breaking Introduces a non-breaking change label Nov 10, 2021
@gigony gigony force-pushed the support_nvtx branch 9 times, most recently from 0faa489 to b83c51b Compare November 12, 2021 19:02
@jakirkham
Copy link
Member

PR ( #141 ) is in. So this is ready for rebasing

@gigony gigony changed the title WIP: Add NVTX support for performance analysis Add NVTX support for performance analysis Nov 24, 2021
@gigony
Copy link
Contributor Author

gigony commented Nov 24, 2021

PR ( #141 ) is in. So this is ready for rebasing

@jakirkham
Please help merge this when you have time.

@gigony gigony mentioned this pull request Nov 29, 2021
deps-nvtx3
GIT_REPOSITORY https://github.com/NVIDIA/NVTX.git
GIT_TAG 3c98c8425b0376fd8653aac7cfc6a864f3897752
# GIT_SHALLOW TRUE # Uncomment this when the official release of nvtx3-cpp is available.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we raise an issue to track this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @jakirkham!

Created #168 and updated the comment in nvtx3.cmake.

cpp/cmake/deps/nvtx3.cmake Outdated Show resolved Hide resolved
@jakirkham
Copy link
Member

Had a couple questions above. Otherwise LGTM

@ajschmidt8
Copy link
Member

CI is passing. Merging.

@ajschmidt8 ajschmidt8 merged commit fb85f01 into rapidsai:branch-21.12 Nov 30, 2021
@jakirkham
Copy link
Member

Thanks AJ! 😄

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request New feature or request non-breaking Introduces a non-breaking change
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[FEA] Support profiling with NVTX
3 participants