Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Roctracer flush activity fix + perfetto.cfg #317

Merged
merged 22 commits into from
Jan 10, 2024

Conversation

jrmadsen
Copy link
Collaborator

  • Fixes missing GPU activity (either HIP or HSA) when omnitrace finalizes
  • This is a byproduct of roctracer_flush_activity() being invoked in omnitrace_finalize after omnitrace set the state to State::Finalized and the activity callbacks immediately returned because get_state() != State::Active
    • A call to roctracer_flush_activity() is required to ensure that any async activity in roctracer in a partially filled buffer is delivered to omnitrace

jrmadsen and others added 4 commits January 9, 2024 09:19
- invoke roctracer_flush_activity() before disabling domains
- real issue was the global state when roctracer_flush_activity() was called
- provide definition of comp::roctracer::flush when OMNITRACE_USE_ROCTRACER is not defined
@jrmadsen jrmadsen force-pushed the roctracer-flush-activity-fix branch from 85d480c to 25274f7 Compare January 9, 2024 15:19
- rename provided perfetto config file (omnitrace.cfg) to perfetto.cfg to avoid confusion
- gpu.hpp: defines for OMNITRACE_USE_{HIP,ROCTRACER,ROCPROFILER,ROCM_SMI}
- gpu.cpp
  - include core/hip_runtime.hpp
  - fix serialization of hipDeviceProp_t
- add hip_runtime.hpp
  -  ensure proper inclusion of hip_runtime.h
- add rccl.hpp
  - ensure proper inclusion of rccl.h
- rcclp.cpp
  - update includes for rccl
- roctracer.hpp
  - update includes for hip_runtime
- components/comm_data.hpp
  - update includes for rccl
- components/rcclp.hpp
  - update includes for rccl
- update includes for hip_runtime
- fix find_package for rccl when CI enabled
- set cmake policy CMP0135 to NEW for cmake >= 3.24
  - Enable DOWNLOAD_EXTRACT_TIMESTAMP with ExternalProject_Add + URL download method
- include rccl.h only if OMNITRACE_USE_RCCL > 0
- reintroduce some ppdefs
@jrmadsen jrmadsen added timemory Issue affects/involves timemory features/capabilities submodule Updates a git submodule perfetto Issue affects/involves perfetto features/capabilities libomnitrace Involves omnitrace library omnitrace-avail Involves the omnitrace-avail executable (info tool) libomnitrace-core Internal library containing core capabilities labels Jan 10, 2024
@jrmadsen jrmadsen changed the title Roctracer flush activity fix Roctracer flush activity fix + perfetto.cfg Jan 10, 2024
- fix ifdef on OMNITRACE_HIP_VERSION
- fix static assert for OMNITRACE_HIP_VERSION_MINOR when HIP version 4.x or older (unreliable minor versions)
- fix ifdef on OMNITRACE_HIP_VERSION
- disable OMNITRACE_PERFETTO_COMBINE_TRACES by default
- if unable to open perfetto temp file, return the ReadTraceBlocking()
- flush tmpfile before closing
@jrmadsen jrmadsen merged commit 7bc50f5 into ROCm:main Jan 10, 2024
49 checks passed
@jrmadsen jrmadsen deleted the roctracer-flush-activity-fix branch January 10, 2024 11:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug fix Fixes a bug libomnitrace Involves omnitrace library libomnitrace-core Internal library containing core capabilities omnitrace-avail Involves the omnitrace-avail executable (info tool) perfetto Issue affects/involves perfetto features/capabilities roctracer GPU kernel tracing submodule Updates a git submodule timemory Issue affects/involves timemory features/capabilities
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant