Skip to content

Commit

Permalink
[SPARK-35087][UI] Some columns in table Aggregated Metrics by Executo…
Browse files Browse the repository at this point in the history
…r of stage-detail page shows incorrectly.

### What changes were proposed in this pull request?

 columns like 'Shuffle Read Size / Records', 'Output Size/ Records' etc  in table ` Aggregated Metrics by Executor` of stage-detail page should be sorted as numerical-order instead of lexicographical-order.

### Why are the changes needed?
buf fix,the sorting style should be consistent between different columns.

The correspondence between the table and the index is shown below(it is defined in stagespage-template.html):
| index | column name                            |
| ----- | -------------------------------------- |
| 0     | Executor ID                            |
| 1     | Logs                                   |
| 2     | Address                                |
| 3     | Task Time                              |
| 4     | Total Tasks                            |
| 5     | Failed Tasks                           |
| 6     | Killed Tasks                           |
| 7     | Succeeded Tasks                        |
| 8     | Excluded                               |
| 9     | Input Size / Records                   |
| 10    | Output Size / Records                  |
| 11    | Shuffle Read Size / Records            |
| 12    | Shuffle Write Size / Records           |
| 13    | Spill (Memory)                         |
| 14    | Spill (Disk)                           |
| 15    | Peak JVM Memory OnHeap / OffHeap       |
| 16    | Peak Execution Memory OnHeap / OffHeap |
| 17    | Peak Storage Memory OnHeap / OffHeap   |
| 18    | Peak Pool Memory Direct / Mapped       |

I constructed some data to simulate the sorting results of the index columns from 9 to 18.
As shown below,it can be seen that the sorting results of columns 9-12 are wrong:

![simulate-result](https://user-images.githubusercontent.com/52202080/115120775-c9fa1580-9fe1-11eb-8514-71f29db3a5eb.png)

The reason is that the real data corresponding to columns 9-12 (note that it is not the data displayed on the page) are **all strings similar to`94685/131`(bytes/records),while the real data corresponding to columns 13-18 are all numbers,**
so the sorting corresponding to columns 13-18 loos well, but the results of columns 9-12 are incorrect because the strings are sorted according to lexicographical order.

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
Only JS was modified, and the manual test result works well.

**before modified:**
![looks-illegal](https://user-images.githubusercontent.com/52202080/115120812-06c60c80-9fe2-11eb-9ada-fa520fe43c4e.png)

**after modified:**
![sort-result-corrent](https://user-images.githubusercontent.com/52202080/114865187-7c847980-9e24-11eb-9fbc-39ee224726d6.png)

Closes apache#32190 from kyoty/aggregated-metrics-by-executor-sorted-incorrectly.

Authored-by: kyoty <[email protected]>
Signed-off-by: Kousuke Saruta <[email protected]>
  • Loading branch information
echohlne authored and sarutak committed Apr 26, 2021
1 parent 6f782ef commit 2d6467d
Showing 1 changed file with 38 additions and 4 deletions.
42 changes: 38 additions & 4 deletions core/src/main/resources/org/apache/spark/ui/static/stagepage.js
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,23 @@ $.extend( $.fn.dataTable.ext.type.order, {
a = ConvertDurationString( a );
b = ConvertDurationString( b );
return ((a < b) ? 1 : ((a > b) ? -1 : 0));
},

"size-pre": function (data) {
var floatValue = parseFloat(data)
return isNaN(floatValue) ? 0 : floatValue;
},

"size-asc": function (a, b) {
a = parseFloat(a);
b = parseFloat(b);
return ((a < b) ? -1 : ((a > b) ? 1 : 0));
},

"size-desc": function (a, b) {
a = parseFloat(a);
b = parseFloat(b);
return ((a < b) ? 1 : ((a > b) ? -1 : 0));
}
} );

Expand Down Expand Up @@ -562,10 +579,27 @@ $(document).ready(function () {
}
],
"columnDefs": [
{ "visible": false, "targets": 15 },
{ "visible": false, "targets": 16 },
{ "visible": false, "targets": 17 },
{ "visible": false, "targets": 18 }
// SPARK-35087 [type:size] means String with structures like : 'size / records',
// they should be sorted as numerical-order instead of lexicographical-order by default.
// The targets: $id represents column id which comes from stagespage-template.html
// #summary-executor-table.If the relative position of the columns in the table
// #summary-executor-table has changed,please be careful to adjust the column index here
// Input Size / Records
{"type": "size", "targets": 9},
// Output Size / Records
{"type": "size", "targets": 10},
// Shuffle Read Size / Records
{"type": "size", "targets": 11},
// Shuffle Write Size / Records
{"type": "size", "targets": 12},
// Peak JVM Memory OnHeap / OffHeap
{"visible": false, "targets": 15},
// Peak Execution Memory OnHeap / OffHeap
{"visible": false, "targets": 16},
// Peak Storage Memory OnHeap / OffHeap
{"visible": false, "targets": 17},
// Peak Pool Memory Direct / Mapped
{"visible": false, "targets": 18}
],
"deferRender": true,
"order": [[0, "asc"]],
Expand Down

0 comments on commit 2d6467d

Please sign in to comment.