Fixes a race in DeviceRunLengthEncode::NonTrivialRuns

[elstehle]

Description

This PR addresses #351.

WarpScanAllocations() and Scatter() alias/repurpose the same shared memory allocation. Previously it could happen that some threads may have reached the Scatter() stage, overwriting shared memory that was yet-to-be-read by other threads still within WarpScanAllocations() stage. A CTA_SYNC was added to WarpScanAllocations to make sure other threads wouldn't race ahead to the Scatter() stage.

Lines that were reading potentially corrupted data:

        warp_aggregate = temp_storage.aliasable.scan_storage.warp_aggregates.Alias()[warp_id];
        ...
        tile_aggregate = temp_storage.aliasable.scan_storage.warp_aggregates.Alias()[0];

This fix resolves the race condition being reported by compute-sanitizer and has had more than 90k successful test runs (previously avg. failure rate was one in ~1300 runs).

Checklist

• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[miscco]

Great find 🥇

[gevtushenko]

Thank you for finding the race!

[gevtushenko]

remark: the memory seems to be used by the prefix operation sooner than in the scatter.

[elstehle]

I think the prefix and warp_aggregates use different memory within the ScanStorage compartment, whereas the ScatterAliasable and the ScanStorage alias one another?

        // Aliasable storage layout
        union Aliasable
        {
            struct ScanStorage
            {
                typename BlockDiscontinuityT::TempStorage       discontinuity;              // Smem needed for discontinuity detection
                typename WarpScanPairs::TempStorage             warp_scan[WARPS];           // Smem needed for warp-synchronous scans
                Uninitialized<LengthOffsetPair[WARPS]>          warp_aggregates;            // Smem needed for sharing warp-wide aggregates
                typename TilePrefixCallbackOpT::TempStorage     prefix;                     // Smem needed for cooperative prefix callback
            } scan_storage;

            // Smem needed for input loading
            typename BlockLoadT::TempStorage                    load;

            // Aliasable layout needed for two-phase scatter
            union ScatterAliasable
            {
                unsigned long long                              align;
                WarpExchangePairsStorage                        exchange_pairs[ACTIVE_EXCHANGE_WARPS];
                typename WarpExchangeOffsets::TempStorage       exchange_offsets[ACTIVE_EXCHANGE_WARPS];
                typename WarpExchangeLengths::TempStorage       exchange_lengths[ACTIVE_EXCHANGE_WARPS];
            } scatter_aliasable;

        } aliasable;

Btw., the culprit really is in the code branch of the first tile:

if (tile_idx == 0)
        {
            // First tile
            ...
            BlockLoadT().Load();
            CTA_SYNC();
            ...
            InitializeSelections(); // using scan_storage
            WarpScanAllocations(); // using scan_storage
            prefix_op(); // using scan_storage
            Scatter(); // using scatter_aliasable