Update of cadcproxy.pem should be atomic

[ijiraq]

The cadcproxy.pem files on the platform are updated once an hour. While they are being updated they are truncated and not usable for access control. This causes error if one tries to use them for access authentication during update, which is not unusually if you are running a few 1000 containers through a processing graph.

If the update process was atomic (so mv temp.file cadcproxy.pem ?) rather than open for write the existing file and then write content when it arrives) these sorts of 'empty file' errors (see below) would be less common.

curl: (58) could not load PEM client certificate, OpenSSL error error:0909006C:PEM routines:get_name:no start line, (no key found, wrong pass phrase, or wrong file format?)

[ijiraq]

The issue might be more complicated than just 'atomic'... The follow errors occur intermittently when I submitted a series of 40 headless jobs. (each error occurred once and the script does a retry on error and the retry then worked). I wonder if this is somehow a file-system issue. The middle error is the one that suggests the cadcproxy.pem file is empty but the others are more vague.. and they all occurred within a few minutes but with successful submissions in-between so this can't just be about the cert being updated? Can it?

curl: (58) unable to set private key file: '/arc/home/jkavelaars/.ssl/cadcproxy.pem' type PEM
curl: (58) could not load PEM client certificate, OpenSSL error error:0909006C:PEM routines:get_name:no start line, (no key found, wrong pass phrase, or wrong file format?)
curl: (58) unable to set private key file: '/arc/home/jkavelaars/.ssl/cadcproxy.pem' type PEM
curl: (58) unable to set private key file: '/arc/home/jkavelaars/.ssl/cadcproxy.pem' type PEM

[brianmajor]

The system will put a new cadcproxy.pem file in your home directory at the start of the launch of any type of session. So I think your initial assessment of that action needing to be atomic is correct. If you controller session is launching jobs concurrently (or even just rapidly) I can see this issue happening quite easily. We'll have this bug fixed soon I hope.