Poor UFO writing performance on Windows

[madig]

See

• https://gregoryszorc.com/blog/2021/04/06/surprisingly-slow/ -> Closing File Handles on Windows
• https://github.com/rust-lang/rustup/pull/1850/files
• https://www.mercurial-scm.org/pipermail/mercurial-devel/2016-January/078404.html

As far as I can tell, as long as Windows Defender (and presumably other A/V scanners) are running, there's no way to make the Windows I/O APIs consistently fast. You can disable A/V scanning (at your own peril). But the trick that Mercurial employs (which has later been emulated by rustup among other tools) is to use a thread pool for calling CloseHandle(). Even if you perform all file open and write I/O on a single thread and use a background thread pool only for calling CloseHandle(), you can see a >3x speedup in time to write files. This optimization should ideally be employed by any software that creates or mutates as little as a few hundred files on Windows. This includes version control tools, installers, and archive extraction tools. Fun fact: rustup can extract tar files on Windows faster than open source and commercial fast extraction/copy tools because it employs this trick and more. I believe rustup on Windows is actually faster at extracting tar archives than it is on Linux!

Maybe we can make norad close file handles in a different thread on windows? On my Linux machine, writing a 1000 glyph UFO takes 0.2s or something, on my Windows work machine 5s.

[cmyr]

Has this been improved by #150?

[cmyr]

@madig ping again, were the numbers you mentioned in #160 for windows? we should now be closing those FDs on multiple threads, so I was hoping this would improve somewhat.

[madig]

They were. Not sure if I'm still overlooking something... Maybe I need to test ufofmt with and without the new norad.

[madig]

How easy would it be to drop file handles on Windows in a different thread?

[cmyr]

hmm, is your idea to send a handle to a cleanup thread that we don't care about, and then everything will be freed when the application exits? Did you read about this somewhere?

[madig]

I suppose? See the first link in the root comment.

[cmyr]

😅 ah, oops. Okay yes I think this is definitely something we can look into.

[madig]

Here's a video on Windows file system performance considerations, at the point where he talks about closing file handles in a different thread: https://youtu.be/qbKGw8MQ0i8?t=1213

[RickyDaMa]

Completed by #335 🎉