Fix nonlocal unsoundness vs. atomic store (#1)

MOVNTI, MOVNTDQ, and friends weaken TSO when next to other stores. As
most stores are not nontemporal, LLVM uses simple stores when lowering
LLVMIR like `atomic store ... release` on x86. These facts could allow
something like the following code to be emitted:

```asm
vmovntdq [addr],     ymmreg
vmovntdq [addr+N],   ymmreg
vmovntdq [addr+N*2], ymmreg
vmovntdq [addr+N*3], ymmreg
mov byte ptr [flag], 1 ; producer-consumer flag
```

But these stores are NOT ordered with respect to each other! Nontemporal
stores induce the CPU to use write-combining buffers. These writes will
be resolved in bursts instead of at once, and the write may be further
deferred until a serialization point. Even a non-temporal write to any
other location will not force the deferred writes to be resolved first.
Thus, assuming cache-line-sized buffers of 64 bytes, the CPU may resolve
these writes in e.g. this actual order:

```asm
vmovntdq [addr+N*2], ymmreg
vmovntdq [addr+N*3], ymmreg
mov byte ptr [flag], 1
vmovntdq [addr+N],   ymmreg
vmovntdq [addr],     ymmreg
```

This could e.g. result in other threads accessing this address after the
flag is set, thus accessing memory via safe code that was assumed to be
correctly synchronized. This could result in observing tearing or other
inconsistent program states, especially as the number of writes, thus
the number of write buffers that may begin retiring simultaneously,
thus the chance of them resolving in an unfortunate order, increases.
If using `&mut [u8]` to write uninitialized memory is permitted ( per
rust-lang/unsafe-code-guidelines#346 ), it could even result in an access
to `&[u8]` actually being reading uninitialized memory in safe code!

To guarantee program soundness, code using nontemporal stores must
currently use SFENCE in its safety boundary, unless and until LLVM
decides this combination of facts should be considered a miscompilation
and motivation to choose lowerings that do not require explicit SFENCE.

src/x86_64.rs

@@ -36,7 +36,12 @@ pub fn memset(slice: &mut [u8], byte: u8) {
 
     let ptr = get_impl().load(Ordering::Relaxed);
     let ptr = unsafe { mem::transmute::<usize, FnSig>(ptr) };
-    ptr(slice, byte)
+    ptr(slice, byte);
+
+    // Required before returning to code that may set atomic flags that invite concurrent reads,
+    // as LLVM will not lower `atomic store ... release`, thus `AtomicBool::store(true, Release)`
+    // on x86-64 to emit SFENCE, even though it is required in the presence of nontemporal stores.
+    unsafe { _mm_sfence() };
 }
 
 #[repr(packed)]