Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

C++ refactoring: reducers #1099

Merged
merged 26 commits into from
Sep 30, 2021
Merged

C++ refactoring: reducers #1099

merged 26 commits into from
Sep 30, 2021

Conversation

ianna
Copy link
Collaborator

@ianna ianna commented Sep 17, 2021

@ianna ianna marked this pull request as draft September 17, 2021 15:11
@ianna
Copy link
Collaborator Author

ianna commented Sep 23, 2021

@jpivarski - please, have a look. I've added a few FIXME notes to discuss. Thanks!

@ianna ianna marked this pull request as ready for review September 23, 2021 12:44
@ianna
Copy link
Collaborator Author

ianna commented Sep 23, 2021

UnionArray reducers need needs #1082

Copy link
Member

@jpivarski jpivarski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I recognize this is a lot of work, and what's here looks great! I made a few "cleanup" changes to the branch, including a replacement of reducer strings with reducer singleton objects, since you create the objects anyway to give them different apply methods. (The singletons are implemented as class objects with @classmethods, so that we can use inheritance, but there's no need to create instances of these classes just to use a stateless function.)

Some of the implementations were missing, and that's surely due to underwhelming test coverage in the original v1 (my fault). I've added the implementations where it was easy and NotImplementedError with the commented-out C++ code where it wasn't. UnionArray can't be done without simplify_uniontype, so that will have to wait for after this PR, but the others could be done (with added tests, to improve the test coverage).

Instead of commenting out failing tests, I added pytest marks to them, so that they show up in the report. That makes it harder to overlook the fact that they're not done.

The missing implementations are indicated with FIXMEs after their NotImplementedErrors. As stated above, UnionArray should wait until after this PR.

Copy link
Collaborator Author

@ianna ianna left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jpivarski - I think, most of the comments are addressed. More tests are needed, I'm working on these. I need some guidance on introducing a dimension_optiontype and an initial for min and max reducers.

src/awkward/_v2/contents/regulararray.py Outdated Show resolved Hide resolved
@ianna
Copy link
Collaborator Author

ianna commented Sep 28, 2021

@jpivarski - as discussed: coverage for src/awkward/_v2/_reducers.py : 96%
Show keyboard shortcuts
136 statements 131 run 5 missing 0 excluded

Note, the missing, not covered code in this case is for Windows. Here is the rest:

Coverage report: 81%

Module statements missing excluded coverage
Total 19965 3844 0 81%
src/awkward/_connect/_autograd.py 39 30 0 23%
src/awkward/_typeparser/generated_parser.py 1597 829 0 48%
src/awkward/_connect/_jax/init.py 18 9 0 50%
src/awkward/_connect/_numpy.py 277 131 0 53%
src/awkward/_v2/identifier.py 78 36 0 54%
src/awkward/_v2/contents/bitmaskedarray.py 147 57 0 61%
src/awkward/_v2/contents/virtualarray.py 290 109 0 62%
src/awkward/_v2/forms/virtualform.py 80 29 0 64%
src/awkward/_v2/forms/bytemaskedform.py 66 22 0 67%
src/awkward/_v2/forms/unionform.py 116 38 0 67%
src/awkward/_v2/forms/bitmaskedform.py 73 23 0 68%
src/awkward/operations/describe.py 73 23 0 68%
src/awkward/_v2/contents/unmaskedarray.py 104 32 0 69%
src/awkward/_v2/forms/regularform.py 68 21 0 69%
src/awkward/nplike.py 332 104 0 69%
src/awkward/_v2/contents/listoffsetarray.py 332 99 0 70%
src/awkward/_v2/forms/emptyform.py 46 14 0 70%
src/awkward/_v2/forms/indexedoptionform.py 60 18 0 70%
src/awkward/_v2/forms/recordform.py 164 50 0 70%
src/awkward/_v2/forms/unmaskedform.py 53 16 0 70%
src/awkward/_v2/forms/indexedform.py 60 17 0 72%
src/awkward/_v2/contents/emptyarray.py 88 24 0 73%
src/awkward/_v2/contents/indexedarray.py 160 41 0 74%
src/awkward/highlevel.py 622 161 0 74%
src/awkward/_v2/contents/indexedoptionarray.py 231 57 0 75%
src/awkward/_v2/contents/regulararray.py 290 70 0 76%
src/awkward/_v2/forms/listform.py 74 18 0 76%
src/awkward/_typeparser/parser.py 225 52 0 77%
src/awkward/_v2/contents/recordarray.py 220 50 0 77%
src/awkward/_v2/record.py 84 19 0 77%
src/awkward/operations/convert.py 2094 474 0 77%
src/awkward/_v2/contents/bytemaskedarray.py 208 46 0 78%
src/awkward/_v2/contents/content.py 408 84 0 79%
src/awkward/_v2/types/uniontype.py 38 8 0 79%
src/awkward/operations/reducers.py 287 61 0 79%
src/awkward/_v2/contents/unionarray.py 191 36 0 81%
src/awkward/behaviors/string.py 149 28 0 81%
src/awkward/_v2/contents/listarray.py 250 44 0 82%
src/awkward/_v2/tmp_for_testing.py 232 41 0 82%
src/awkward/partition.py 441 79 0 82%
src/awkward/forth.py 6 1 0 83%
src/awkward/_v2/forms/listoffsetform.py 67 11 0 84%
src/awkward/_v2/index.py 106 17 0 84%
src/awkward/_cpu_kernels.py 13 2 0 85%
src/awkward/_v2/forms/form.py 131 20 0 85%
src/awkward/_v2/types/recordtype.py 65 10 0 85%
src/awkward/operations/structure.py 1161 172 0 85%
src/awkward/_util.py 1030 142 0 86%
src/awkward/_v2/_slicing.py 198 28 0 86%
src/awkward/_v2/types/arraytype.py 22 3 0 86%
src/awkward/_v2/types/regulartype.py 38 5 0 87%
src/awkward/_connect/_numba/layout.py 1458 179 0 88%
src/awkward/_v2/forms/numpyform.py 96 12 0 88%
src/awkward/_v2/types/listtype.py 32 4 0 88%
src/awkward/_v2/types/optiontype.py 32 4 0 88%
src/awkward/_v2/contents/numpyarray.py 221 25 0 89%
src/awkward/behaviors/categorical.py 145 16 0 89%
src/awkward/_connect/_numba/init.py 73 7 0 90%
src/awkward/_connect/_numba/builder.py 351 36 0 90%
src/awkward/_connect/_numexpr.py 75 7 0 91%
src/awkward/_v2/types/unknowntype.py 22 2 0 91%
src/awkward/_v2/types/numpytype.py 56 4 0 93%
src/awkward/types.py 14 1 0 93%
src/awkward/init.py 35 2 0 94%
src/awkward/forms.py 18 1 0 94%
src/awkward/_v2/types/type.py 40 2 0 95%
src/awkward/behaviors/mixins.py 44 2 0 95%
src/awkward/_v2/_reducers.py 136 5 0 96%
src/awkward/_connect/_numba/arrayview.py 894 21 0 98%
src/awkward/_libawkward.py 83 2 0 98%
src/awkward/layout.py 44 1 0 98%
src/awkward/_connect/init.py 1 0 0 100%
src/awkward/_kernel_signatures.py 2840 0 0 100%
src/awkward/_typeparser/init.py 1 0 0 100%
src/awkward/_v2/init.py 8 0 0 100%
src/awkward/_v2/contents/init.py 17 0 0 100%
src/awkward/_v2/forms/init.py 15 0 0 100%
src/awkward/_v2/types/init.py 10 0 0 100%
src/awkward/behaviors/init.py 1 0 0 100%
src/awkward/operations/init.py 1 0 0 100%
Name                                                                                         Stmts   Miss  Cover   Missing
--------------------------------------------------------------------------------------------------------------------------
src/awkward/__init__.py                                                                         35      2    94%   15, 72
src/awkward/_connect/__init__.py                                                                 1      0   100%
src/awkward/_connect/_autograd.py                                                               39     30    23%   15-49, 56-85
src/awkward/_connect/_jax/__init__.py                                                           18      9    50%   15-36, 40
src/awkward/_connect/_numba/__init__.py                                                         73      7    90%   17-18, 31, 90, 92, 98, 107
src/awkward/_connect/_numba/arrayview.py                                                       894     21    98%   20-21, 75, 192, 224, 265, 273, 337, 512, 519, 589, 680-681, 879, 1000, 105
7-1058, 1513, 1568, 1744-1749
src/awkward/_connect/_numba/builder.py                                                         351     36    90%   152, 159-162, 176, 190, 204, 214, 224, 238, 252, 262, 271-278, 292, 302, 3
48-367, 381, 434, 438, 524-534, 672, 685-693
src/awkward/_connect/_numba/layout.py                                                         1458    179    88%   44, 123, 137, 238, 243, 256-258, 266-268, 275, 286, 288, 292, 300, 312, 31
6, 337, 475, 523, 539, 547, 551, 553, 555, 557, 559, 561, 563, 567, 604, 711, 751, 755, 759, 763, 782-783, 868-869, 895, 899, 983, 1059, 1061, 1065, 1080, 1144, 1148, 1152, 1156, 1232, 1236
, 1251, 1336, 1432, 1510, 1514, 1518, 1522, 1620, 1716, 1720, 1724, 1728, 1793, 1852, 1856, 1860, 1864, 1970, 1989, 2020-2028, 2044-2052, 2066, 2252, 2256, 2360, 2362, 2366-2371, 2390, 2393
-2394, 2400-2401, 2407-2408, 2426, 2444, 2451, 2459-2465, 2469-2470, 2474, 2478-2483, 2497, 2520, 2536, 2546, 2588, 2591-2608, 2644, 2653, 2784, 2788, 2792, 2796, 2800-2840, 2847-2882, 2889
-2926
src/awkward/_connect/_numexpr.py                                                                75      7    91%   19-20, 33, 60-61, 125-126
src/awkward/_connect/_numpy.py                                                                 277    131    53%   17, 40, 68, 76, 109, 231-233, 246-249, 270-342, 393-478
src/awkward/_cpu_kernels.py                                                                     13      2    85%   12, 16
src/awkward/_kernel_signatures.py                                                             2840      0   100%
src/awkward/_libawkward.py                                                                      83      2    98%   10, 14
src/awkward/_typeparser/__init__.py                                                              1      0   100%
src/awkward/_typeparser/generated_parser.py                                                   1597    829    48%   51-54, 63-73, 77-114, 119-139, 145-166, 172-176, 221-222, 225-234, 250-251
, 264-265, 268, 271, 299-314, 319-320, 327-342, 350, 362-364, 367, 370, 373-383, 387, 390-393, 396, 399, 403-411, 415, 419, 438-457, 460, 474-489, 492-502, 505-514, 517-518, 522, 526, 530, 
534, 542-548, 553, 556-558, 561, 568, 571-574, 583-608, 614-615, 623, 627, 630, 638-640, 644-646, 654-659, 663-669, 674-678, 685-690, 693, 697, 700, 707-712, 716-726, 731, 735-750, 754, 756
, 758, 761, 766-787, 793-795, 798-815, 818-835, 838-840, 846, 856, 859-860, 863, 869, 880-881, 885, 898-902, 905, 921-926, 932, 935, 943, 953-954, 957, 962, 965, 968, 975, 979-982, 1010, 10
15, 1027-1030, 1033, 1043-1045, 1058, 1069, 1072, 1075, 1079, 1101-1102, 1110, 1113-1118, 1123-1125, 1128-1129, 1145-1147, 1149, 1167, 1170-1172, 1208-1217, 1237-1242, 1259-1261, 1268-1271,
 1283-1285, 1290-1291, 1308, 1352-1358, 1378-1386, 1401, 1406, 1409-1447, 1452-1454, 1457-1470, 1477-1493, 1524-1527, 1543, 1553-1555, 1558-1582, 1586-1589, 1596-1597, 1600-1629, 1633-1636,
 1640-1646, 1650-1656, 1696, 1699, 1701, 1709, 1719-1725, 1735, 1769, 1778, 1791-1793, 1814, 1847-1862, 1869, 1871, 1884-1893, 1915-1927, 1934, 1936, 1939-1976, 1984, 1994, 1999-2000, 2003,
 2006-2007, 2020-2022, 2042, 2047, 2054, 2058-2062, 2065, 2069, 2155, 2207, 2212, 2219, 2223, 2228-2229, 2232-2233, 2236, 2256-2407, 2410, 2415, 2433-2435, 2439-2440, 2445-2446, 2452, 2460,
 2486-2490, 2495-2500, 2503, 2508-2513, 2517, 2524-2551, 2555-2556, 2560, 2564-2565, 2569
src/awkward/_typeparser/parser.py                                                              225     52    77%   16, 31, 34, 37, 62-66, 68-72, 76-77, 86, 99-100, 106, 114-115, 120-121, 12
4, 130-131, 138-139, 148-149, 157-158, 168-169, 176-177, 186-187, 196-197, 205-206, 216-217, 225-226, 243-244, 273-274, 287
src/awkward/_util.py                                                                          1030    142    86%   15-17, 30, 59, 69, 102, 174-175, 178-179, 182-186, 216-217, 220, 253, 286-
288, 291-293, 306-308, 311-313, 420, 447, 482-487, 518, 520, 527-545, 553-557, 560, 563, 601, 624, 694, 726, 784, 943-949, 1014, 1018, 1026, 1030, 1038, 1060, 1070, 1119-1129, 1187, 1229-12
90, 1315, 1327, 1336, 1362, 1386, 1402, 1460, 1469, 1490, 1526, 1583, 1593, 1632, 1651, 1660, 1737, 1739, 1760, 1766, 1786, 1788, 1792, 1801, 1803, 1805, 1874-1887
src/awkward/_v2/__init__.py                                                                      8      0   100%
src/awkward/_v2/_reducers.py                                                                   136      5    96%   22, 32, 164-180
src/awkward/_v2/_slicing.py                                                                    198     28    86%   7-8, 64-65, 68-69, 74, 100, 103, 106, 126, 138-139, 150, 163, 218-219, 276-281, 318-319, 327, 380-381, 389, 423, 455
src/awkward/_v2/contents/__init__.py                                                            17      0   100%
src/awkward/_v2/contents/bitmaskedarray.py                                                     147     57    61%   29, 35, 41, 47, 53, 59, 65, 100, 120, 123-136, 163-178, 186-203, 206, 237, 252, 257-282, 317, 330, 332, 343
src/awkward/_v2/contents/bytemaskedarray.py                                                    208     46    78%   19, 25, 31, 37, 66, 85, 88-99, 102-117, 123, 125, 130, 176-180, 225, 301-308, 366, 438-453, 472, 486, 489, 521, 529, 540
src/awkward/_v2/contents/content.py                                                            408     84    79%   7-8, 23, 29, 64, 79, 88, 145, 230, 238, 245, 254, 266, 277, 296, 299, 344, 373, 391, 429-451, 457, 463, 469, 475, 489-493, 510, 517, 523, 535, 620, 625-632, 667, 671, 718, 724, 729-760, 763-794, 797, 802, 807-820, 827, 832
src/awkward/_v2/contents/emptyarray.py                                                          88     24    73%   19, 22-30, 59, 84, 97, 102-118, 121
src/awkward/_v2/contents/indexedarray.py                                                       160     41    74%   25, 30, 54, 72, 75-84, 87, 92, 132-136, 148, 217, 223, 260, 275-279, 336, 340, 376, 383-410
src/awkward/_v2/contents/indexedoptionarray.py                                                 231     57    75%   24, 29, 53, 71, 74-83, 86-89, 97-100, 103, 146-150, 196, 263-270, 273-298, 309-315, 321, 421, 449-459, 533-547, 565, 582, 613, 620-627, 641
src/awkward/_v2/contents/listarray.py                                                          250     44    82%   22, 27, 32, 67, 86, 89-99, 106, 155, 189, 285, 366-376, 562, 593-630, 638, 713, 723-730, 738
src/awkward/_v2/contents/listoffsetarray.py                                                    332     99    70%   21, 26, 32, 82, 107, 112-113, 116, 130, 160-164, 189, 196, 203, 238-461, 497, 506, 541, 543, 683, 987, 998, 1008-1015, 1023
src/awkward/_v2/contents/numpyarray.py                                                         221     25    89%   22, 32, 70, 97-98, 169-170, 195, 212-213, 234, 264-270, 277, 286, 303, 377, 466, 469, 472
src/awkward/_v2/contents/recordarray.py                                                        220     50    77%   9-10, 24, 30, 33, 41, 48, 54, 62, 64, 70, 76, 94, 126-131, 161-164, 192, 208, 231-235, 247-253, 278-281, 317, 320, 323, 352, 357, 365, 384-395, 436, 442
src/awkward/_v2/contents/regulararray.py                                                       290     70    76%   16, 22, 28, 56, 74, 77-87, 94, 145-146, 150-153, 156, 197, 204, 211, 253, 410, 413-416, 446, 453, 490, 528, 557, 563, 572, 574, 626-631, 658-702, 707-715
src/awkward/_v2/contents/unionarray.py                                                         191     36    81%   7-8, 22, 32, 38, 44, 48, 55, 87, 127, 172-176, 256, 299-303, 313, 322-329, 340, 355-361, 375, 379, 396-403, 409
src/awkward/_v2/contents/unmaskedarray.py                                                      104     32    69%   15, 34, 51, 54-62, 65, 74-75, 83-86, 89, 147-154, 182-188, 204, 223-227, 249
src/awkward/_v2/contents/virtualarray.py                                                       290    109    62%   11-12, 25, 31, 48, 56, 62, 68-85, 98, 104, 110, 125, 129, 133, 136, 139-149, 159, 169, 173, 176, 179-185, 196-206, 227, 233, 239, 270, 282, 297, 302, 305-335, 356, 366, 370, 391, 409, 414, 432, 445, 448-452, 455-459, 474, 486
src/awkward/_v2/forms/__init__.py                                                               15      0   100%
src/awkward/_v2/forms/bitmaskedform.py                                                          73     23    68%   22, 28, 34, 40, 54, 58, 62, 66, 101, 104-119, 122, 132, 143, 154, 165, 169, 181, 185
src/awkward/_v2/forms/bytemaskedform.py                                                         66     22    67%   20, 26, 32, 45, 49, 53, 75-85, 88-102, 105, 115, 125, 135, 146, 162, 166
src/awkward/_v2/forms/emptyform.py                                                              46     14    70%   28-37, 40, 47, 50, 53, 61, 65, 77, 81
src/awkward/_v2/forms/form.py                                                                  131     20    85%   166, 179-183, 185-189, 194, 201-202, 216, 222, 228, 249-252, 289, 292
src/awkward/_v2/forms/indexedform.py                                                            60     17    72%   19, 25, 37, 41, 67, 70-83, 86, 95, 104, 113, 123, 139
src/awkward/_v2/forms/indexedoptionform.py                                                      60     18    70%   19, 25, 37, 41, 67, 70-83, 86, 95, 104, 113, 123, 139, 143
src/awkward/_v2/forms/listform.py                                                               74     18    76%   20, 26, 32, 45, 49, 53, 85, 91, 102, 115, 125, 135, 146, 151, 158, 166, 173, 177
src/awkward/_v2/forms/listoffsetform.py                                                         67     11    84%   15, 27, 31, 60, 66, 76, 97, 117, 122, 129, 148
src/awkward/_v2/forms/numpyform.py                                                              96     12    88%   7-8, 62, 72, 131, 137, 148, 160, 163, 166, 176, 197
src/awkward/_v2/forms/recordform.py                                                            164     50    70%   7-8, 25, 32, 38, 51, 61, 70, 74, 84-85, 104-109, 119, 151, 156, 160, 164-166, 172, 186-191, 194, 207-211, 214-233, 242-251, 261, 270, 281, 291, 296
src/awkward/_v2/forms/regularform.py                                                            68     21    69%   14, 20, 32, 36, 62, 65-78, 81, 90, 99, 108, 118, 123, 130, 138, 145, 149
src/awkward/_v2/forms/unionform.py                                                             116     38    67%   7-8, 25, 31, 37, 44, 57, 61, 65, 68, 104, 108-110, 113-128, 131, 141, 151, 161, 172-176, 185, 192, 203, 211, 213, 225-228
src/awkward/_v2/forms/unmaskedform.py                                                           53     16    70%   17, 28, 52, 55-66, 69, 77, 85, 93, 102, 118, 122
src/awkward/_v2/forms/virtualform.py                                                            80     29    64%   18, 24, 40, 64, 68, 73, 79, 84, 93, 104, 115, 125-130, 134-139, 143-148, 152-157, 161-166, 170-175
src/awkward/_v2/identifier.py                                                                   78     36    54%   15-17, 25, 30, 32, 40, 44, 48, 52, 56, 60, 64, 70, 73, 76, 79, 82, 85-104, 107, 110
src/awkward/_v2/index.py                                                                       106     17    84%   27, 41, 48, 93, 105-119, 122, 142, 145, 148
src/awkward/_v2/record.py                                                                       84     19    77%   7-8, 19, 23, 30, 50, 54, 81, 84, 90, 93, 96, 102, 105, 108, 111, 116-120
src/awkward/_v2/tmp_for_testing.py                                                             232     41    82%   16, 23, 26, 29, 174-180, 187, 340-386, 393, 468-469, 482-483, 485-486, 529-530, 544, 553, 564, 580, 586, 595
src/awkward/_v2/types/__init__.py                                                               10      0   100%
src/awkward/_v2/types/arraytype.py                                                              22      3    86%   12, 28, 32
src/awkward/_v2/types/listtype.py                                                               32      4    88%   12, 18, 24, 35
src/awkward/_v2/types/numpytype.py                                                              56      4    93%   46, 54, 60, 71
src/awkward/_v2/types/optiontype.py                                                             32      4    88%   14, 20, 26, 37
src/awkward/_v2/types/recordtype.py                                                             65     10    85%   7-8, 19, 25, 28, 34, 40, 46, 58, 62
src/awkward/_v2/types/regulartype.py                                                            38      5    87%   12, 24, 30, 42, 46
src/awkward/_v2/types/type.py                                                                   40      2    95%   15, 25
src/awkward/_v2/types/uniontype.py                                                              38      8    79%   7-8, 17, 23, 26, 32, 38, 49
src/awkward/_v2/types/unknowntype.py                                                            22      2    91%   12, 18
src/awkward/behaviors/__init__.py                                                                1      0   100%
src/awkward/behaviors/categorical.py                                                           145     16    89%   19-21, 24, 27, 36-37, 40, 43, 48, 50, 52, 165, 169, 341, 350
src/awkward/behaviors/mixins.py                                                                 44      2    95%   86, 110
src/awkward/behaviors/string.py                                                                149     28    81%   18, 27-28, 31-34, 37, 40-43, 46-49, 60, 69-70, 73-76, 79, 82-85, 88-91, 165, 272
src/awkward/forms.py                                                                            18      1    94%   40
src/awkward/forth.py                                                                             6      1    83%   12
src/awkward/highlevel.py                                                                       622    161    74%   11-13, 28, 211, 223, 247, 259, 272, 334, 363, 369-383, 395, 398, 401, 404-426, 520, 523, 998, 1063, 1072, 1116, 1121-1122, 1192, 1199, 1206, 1213, 1220, 1227, 1234, 1419, 1465, 1472-1473, 1494-1497, 1553, 1559, 1562, 1567-1574, 1577, 1590, 1596, 1598, 1648, 1677, 1683, 1697, 1717, 1729, 1732, 1735-1736, 1746, 1777, 1779, 1801-1812, 1848-1849, 1865, 1885, 1895, 1905, 1915, 1925, 1935, 1945, 1955, 1965, 1975, 2001, 2009-2025, 2066, 2073-2074, 2090, 2096, 2102-2105, 2281, 2296, 2302, 2314-2321, 2329-2330, 2358, 2366-2377, 2387, 2397, 2406, 2426-2429, 2685, 2701, 2711-2727, 2766-2767, 2770, 2773, 2796
src/awkward/layout.py                                                                           44      1    98%   106
src/awkward/nplike.py                                                                          332    104    69%   21, 28-31, 43-46, 144, 203, 221, 246, 266, 270, 310, 355, 364, 376, 412, 415, 418-430, 434, 441, 448, 451-466, 469-484, 487, 490-491, 495-498, 502-510, 514-530, 535-539, 542-546, 549-553, 556-560, 563-567, 570-574, 577-581, 584-588, 591-595, 601
src/awkward/operations/__init__.py                                                               1      0   100%
src/awkward/operations/convert.py                                                             2094    474    77%   18-20, 33, 36-39, 95-108, 131-141, 152-153, 160-163, 195, 198, 201, 207-208, 211, 214, 241, 246, 261-264, 271-272, 284-288, 303-305, 320, 336, 342, 370, 374-384, 415-432, 449-552, 583-600, 617-726, 775, 778, 784-787, 892, 930, 1003, 1009, 1103, 1127, 1192, 1195, 1198, 1201, 1207, 1210, 1216, 1225, 1310-1313, 1315-1318, 1320, 1322, 1364, 1394-1434, 1452-1462, 1473-1482, 1487, 1501, 1508-1509, 1519-1539, 1543-1559, 1563-1568, 1574-1575, 1580, 1589, 1608, 1619, 1625, 1635, 1640, 1674, 1680, 1685, 1694, 1702, 1718, 1724, 1751, 1766, 1774, 1790-1795, 1802-1803, 1808, 1814, 1824-1849, 1891, 1894, 1904, 1913, 1956, 1971-1972, 1988, 2165-2187, 2190-2203, 2360-2366, 2376, 2409, 2430-2436, 2500, 2508, 2722, 2733, 2780-2790, 2808-2818, 2858, 2890, 2914, 2918-2919, 2939, 3080, 3117, 3127-3128, 3209-3214, 3233, 3241, 3249, 3264, 3297, 3323-3332, 3402-3406, 3413, 3417, 3443, 3461, 3464, 3484, 3528, 3546, 3564, 3585, 3661, 3713, 3742, 3820, 3824-3829, 3893, 3930-3932, 3946, 4141, 4143, 4149, 4187, 4372-4391, 4410, 4439-4440, 4491, 4520, 4522, 4547, 4549, 4572, 4589, 4591, 4621, 4623, 4659, 4661, 4668, 4677, 4680, 4705, 4707, 4743, 4758, 4792, 4794, 4811, 4842, 4844, 4851, 4860, 4863, 4931, 5047, 5073, 5081, 5129, 5253-5254, 5275, 5281, 5289, 5329, 5357, 5366, 5418
src/awkward/operations/describe.py                                                              73     23    68%   52, 64, 68-72, 125, 128, 131, 134, 155, 161, 170-182, 191, 279
src/awkward/operations/reducers.py                                                             287     61    79%   88, 141, 338, 387, 438, 489, 543, 547-548, 604, 608-609, 674, 732-742, 802-812, 884-890, 966-972, 1040-1046, 1170-1176, 1245-1257, 1333-1351, 1420, 1422, 1424, 1426, 1429-1448, 1512
src/awkward/operations/structure.py                                                           1161    172    85%   10-11, 157, 339, 371, 385, 400, 417, 433, 573, 606-608, 619, 679, 725, 886, 919, 950, 952, 1004, 1182, 1364, 1389, 1430, 1456, 1475-1482, 1504-1507, 1680, 1697, 1700, 1724, 1744, 1845, 1853, 1856, 1859, 1867, 1896, 1982-1986, 1990, 1994, 2004, 2020, 2060, 2082, 2097, 2150, 2170, 2232, 2269, 2293, 2310, 2331, 2685, 2698-2707, 2719, 2772, 2824, 2985, 2994, 2999, 3230, 3242, 3245-3249, 3258, 3263, 3274, 3276, 3287, 3297, 3329-3356, 3412, 3414, 3433, 3539, 3566-3570, 3733, 3790, 3792, 3795, 3824-3830, 3905, 3910, 3929, 3936, 4113, 4118, 4210, 4220-4221, 4229, 4240, 4268, 4276, 4278, 4280, 4282-4293, 4295, 4306-4317, 4324, 4330, 4351, 4420, 4448-4477, 4554, 4651
src/awkward/partition.py                                                                       441     79    82%   9-10, 19-22, 33-36, 42-55, 75-82, 110, 131-136, 140, 158-159, 163, 175, 179, 185, 188, 195, 214, 244, 270, 295, 314, 388, 390, 413, 420, 423, 462, 465-474, 510, 518, 546, 561-563, 599-608, 617, 630, 683-684, 696, 702
src/awkward/types.py  


@jpivarski
Copy link
Member

No complaints about the 96%! (I don't think 100% is a goal because of things like the Windows cases. Twisting them to count as covered, for instance with one-line X if P else Y is an abuse in that it might make the code less readable.)

I'm curious about the coverage for the whole src/awkward/_v2 subdirectory, though it's not a part of this PR.

@scikit-hep scikit-hep deleted a comment from jpivarski Sep 29, 2021
@ianna
Copy link
Collaborator Author

ianna commented Sep 29, 2021

@jpivarski - apologies, I did not mean to delete your comment I just added a tick mark...

@ianna
Copy link
Collaborator Author

ianna commented Sep 29, 2021

The checklist of things remaining:

toRegularArray, which enables some implementations. This is not a huge project and would fit into the scope of this PR.
UnionArray: not a part of this PR, as it requires merge.
axis=None: not a part of this PR, as it requires ak._util.completely_flatten to recognize v2 classes.
On the last of these, instead of modifying ak._util to know about v2, I'm thinking of duplicating the high-level layer into ak._v2 with appropriate translations to make it v2-only. That would give us a chance to review algorithms that unnecessarily do if-elif-else over node types, rather than calling a method on the nodes. ak._util.completely_flatten, for example, could be a node method. That would make it easier to cut off v1 and go v2-only when the time comes. The disadvantage is that in the interim, any bug-fixes to v1 code would have to be also fixed in v2. We've seen one instance of that so far, and I think it's not too bad. The advantage of being able to cleanly cut over to v2 when it's ready may outweigh it. Before doing that, I'll have to have a clear schedule, so that we know how much time we're having to apply bug-fixes in both branches.

@ianna
Copy link
Collaborator Author

ianna commented Sep 29, 2021

@jpivarski - pre-commit complains about B904: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling. See https://docs.python.org/3/tutorial/errors.html#exception-chaining for details.
I guess, it would be fixed by rebasing the branch? Thanks.

@jpivarski
Copy link
Member

@jpivarski - pre-commit complains about B904: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling.

This seems to be a new rule, and it pops up in many places. We probably need to just ignore it globally (setup.cfg) for now, until we find out how to deal with it. The fix it's proposing uses some special syntax (the from keyword), which might not exist in Python 2. Although we will be dropping Python 2 support and the v2 parts are excluding Python 2 from the outset, we can't cross the boundary of having non-Python 2 syntax, because Python's initial .py → .pyc scan will fail.

@ianna
Copy link
Collaborator Author

ianna commented Sep 30, 2021

@jpivarski - I think, I've addressed all issues. Please, have a look. Thanks

@jpivarski
Copy link
Member

I agree. Looks great, thanks!

@jpivarski jpivarski merged commit ae47f48 into main Sep 30, 2021
@jpivarski jpivarski deleted the ianna/refactoring-reducers branch September 30, 2021 12:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants