Warning: "Errorcode: %d cannot get x_attr(%s,%c) from object %s, process %d"

[jrse]

**Concurrent read and delete for same object **

The sdbox / mdbox storage plugin, does detect (and sync) concurrent access to a mail object with a filelock. In Ceph is no similar mechanism for an object. So dependent on the actual test (number of users, concurrent users, type of mail operation, ...) the warning with errorcode -2 will occur and the mail object will be marked as expunged.

There is no easy/obvious solution to avoid this, but the timespan a index is locked or has unsaved modifications can be minimized.
e.g. closing the index, before executing / waiting for object deletion.

Discussions are welcome!

[jrse]

A workable solution may be using librados watch/notify mechanism.
It could be implemented in the following way:
rbox_mail.cpp creates at the first opportunity a watch on the mail object, it want's to read.
rbox_sync.cpp calls the notify (synchronous) method before deleting a mail object from the object store.

The sdbox implementation creates a file handle when reading metadata, save_date, get_stream and removes it in dbox_mail_close (sdbox_mail.c). sdbox_sync.c will wait with the deletion using unlink which blocks until all file handler are removed before deleting a file.

[jrse]

librados create watcher:

rados_watch2(io, object,&cookie, watch_notify2_cb, watch_notify2_errcb,NULL);

wather callbacks:
void watch_notify2_cb(void *arg, uint64_t notify_id, uint64_t cookie, uint64_t notifier_gid, void *data,size_t data_len);
void watch_notify2_errcb(void *arg, uint64_t cookie, int err);

notify:
rados_notify2(rados_ioctx_t io, const char * o, const char * buf, int buf_len, uint64_t timeout_ms, char ** reply_buffer, size_t * reply_buffer_len)

see: http://docs.ceph.com/docs/jewel/rados/api/librados/

[jrse]

• re implement the mail index with appropriate sync mechanism.

-> more tests under heavy load required to determine scope of this issue.