Improve console output from python tool for failed/gpu/photon event logs

[parthosa]

Fixes #1126.

#1187 added feature in scala tools to include the status report of all apps even those which had failed or skipped due to photon event logs. This PR improves python tools to alway display num of processed apps (even if passed GPU event logs or Streaming logs or any other failure)

Output

Case 1 : Event logs with no successfull apps: photon/gpu/csp event log with authentication issue

spark_rapids qualification --platform <platform>  --eventlogs </path/to/gpu-or-photon-logs>  --tools_jar <tools_jar> --verbose

Previously

____________________________________________________________________________________________________
                                        QUALIFICATION Report
____________________________________________________________________________________________________
The Qualification tool did not generate any output. Nothing to display.

After this change

    - Intermediate output generated by tools: /Users/psarthi/Work/tools-run/qual_20240726231326_caB38e8A/intermediate_output
    - Application status report: /Users/psarthi/Work/tools-run/qual_20240726231326_caB38e8A/rapids_4_spark_qualification_output/rapids_4_spark_qualification_output_status.csv

Qualification tool found no successful applications to process.

Report Summary:
----------------------  -
Total applications      3
Processed applications  0
Top candidates          0
----------------------  -

Case 2: Event logs with no top candidates

Console Output

    - Summarized savings and speedups CSV report: /Users/psarthi/Work/tools-run/qual_20240726211957_B9A2A6A9/qualification_summary.csv
    - Intermediate output generated by tools: /Users/psarthi/Work/tools-run/qual_20240726211957_B9A2A6A9/intermediate_output
    - *Cluster config recommendations: /Users/psarthi/Work/tools-run/qual_20240726211957_B9A2A6A9/rapids_4_spark_qualification_output/tuning
    - Metadata file with cluster recommendation and tuning details: /Users/psarthi/Work/tools-run/qual_20240726211957_B9A2A6A9/qualification_summary_metadata.json
    - Application status report: /Users/psarthi/Work/tools-run/qual_20240726211957_B9A2A6A9/rapids_4_spark_qualification_output/rapids_4_spark_qualification_output_status.csv

Qualification tool found no qualified applications after applying the filters.
See the CSV file for full report or disable the filters.

Report Summary:
----------------------  -
Total applications      3
Processed applications  3
Top candidates          0
----------------------  -

Case 3: Event logs path with some top candidate apps, some gpu apps/photon apps

Console Output

    - Summarized savings and speedups CSV report: /Users/psarthi/Work/tools-run/qual_20240726225824_511D59E9/qualification_summary.csv
    - Intermediate output generated by tools: /Users/psarthi/Work/tools-run/qual_20240726225824_511D59E9/intermediate_output
    - *Cluster config recommendations: /Users/psarthi/Work/tools-run/qual_20240726225824_511D59E9/rapids_4_spark_qualification_output/tuning
    - Metadata file with cluster recommendation and tuning details: /Users/psarthi/Work/tools-run/qual_20240726225824_511D59E9/qualification_summary_metadata.json
    - Application status report: /Users/psarthi/Work/tools-run/qual_20240726225824_511D59E9/rapids_4_spark_qualification_output/rapids_4_spark_qualification_output_status.csv
+----+-------------------------+-------------------------+-----------------+---------------------------+------------------------------+-----------------------------+
|    | App Name                | App ID                  | Estimated GPU   | Qualified Node            | Full Cluster                 | GPU Config                  |
|    |                         |                         | Speedup         | Recommendation            | Config                       | Recommendation              |
|    |                         |                         | Category**      |                           | Recommendations*             | Breakdown*                  |
|----+-------------------------+-------------------------+-----------------+---------------------------+------------------------------+-----------------------------|
|  1 | test_spark_app_111111   | app-20240311195738-0000 | Small           | r5d.2xlarge to g5.2xlarge | app-20240311195738-0000.conf | app-20240311195738-0000.log |
|  0 | test_spark_app_222222   | app-20240311074805-0000 | Small           | r5d.2xlarge to g5.2xlarge | app-20240311074805-0000.conf | app-20240311074805-0000.log |
+----+-------------------------+-------------------------+-----------------+---------------------------+------------------------------+-----------------------------+

Report Summary:
----------------------  -
Total applications      7
Processed applications  4
Top candidates          2
----------------------  -

Changes

• In class qualification.py::QualificationSummary:
  • Removed duplicate all_apps and df_results DFs
  • Renaming of class variables with comments to state what they refer to
  • Renaming of getters
  • Comments that list output files should be shown even if there are no candidates (Eg Case 2 above)
• In class qualification.py::Qualification:
  • Introduced a helper method _read_qualification_output_file() that reads files generated from the scala output folder
  • Now that we are reading status report, add event log in the metadata json file for each app.

[amahussein]

LGTME!
Thanks @parthosa

[cindyyuanjiang]

Thanks @parthosa! LGTM.

[tgravescs]

did this change the output of the top candidate table and location of where the * and ** definition is?

like:

• *Cluster config recommendations: /Users/psarthi/Work/tools-run/qual_20240726225824_511D59E9/rapids_4_spark_qualification_output/tuning

is now above the table?