rbd: wait for safe callback for write requests

rbd: wait for safe callback for write requests

When a request is sent to the osd, the sender can indicate what
"level" of completion desired before the request is acknowledged.
There will always be at least an acknowledgement sent to indicate
the osd had received the message.  But for a write request the
sender may request that the acknowledgement indicate when the write
operation is durable on the osd.  The osd marks a response with the
ONDISK flag to signal this in its acknowledgement.

When a request is acknowledged a callback function is run and an
event is completed that a caller can wait for.  When the ONDISK flag
is set in an acknowledgement, an additional callback is used to
allow the caller to record when a request has been sent to an osd
(making it "unsafe"), and when an acknowledgement indicating such a
request has been made durable on the osd (so it is no longer
unsafe).  A "safe completion" is signaled to unblock any waiters.

With that as background...

Currently the rbd client waits only for the acknowledgement response
for all requests, which isn't safe for writes.  Fix that by defining
and using a different callback function that marks write requests
done only when the ONDISK notification arrives.

This resolves:
    http://tracker.ceph.com/issues/5146

Signed-off-by: Alex Elder <elder@inktank.com>
Reviewed-by: Josh Durgin <josh.durgin@inktank.com>