dm thin: fix pool_preresume resize with heavy IO races

dm thin: fix pool_preresume resize with heavy IO races

Before, when a pool is being resized, on resume the pool's mode was
being immediately set to PM_WRITE and the process_* methods would be set
to the normal writeable operations.  This was occuring before
pool_resume() was able to actually resize either the metadata or data
device and resulted in the resize failing.

Now, the pool is forced to stay in read-only mode if it was already in
read-only mode.  This prevents bouncing the pool's mode and associated
process_* methods and consistently keeps read-only processing in place
until the resize(s) complete.  To achieve this the pool can now be in a
PM_READ_ONLY state but the metadata's is !read_only -- so as to allow
the commit() the follows the resize to take place.  Differentiate
between commit_metadata_superblock() and commit() since different
negative checks apply (but factor out a common __commit that each
calls).

With this patch the resize_io test passes (on fast storage):
 dmtest run --suite thin-provisioning -n /resize_io/

Otherwise, problems like the following 2 examples were seen (again when
testing on really fast PCIe SSD storage):

1)
kernel: device-mapper: thin: 253:2: growing the data device from 2048 to 4096 blocks
kernel: device-mapper: thin: 253:2: reached low water mark for data device: sending event.
kernel: device-mapper: thin: 253:2: no free data space available.
kernel: device-mapper: thin: 253:2: switching pool to read-only mode
kernel: device-mapper: thin: 253:2: switching pool to write mode (from PM_WRITE)
kernel: device-mapper: thin: 253:2: growing the data device from 4096 to 6144 blocks
kernel: device-mapper: thin: 253:2: reached low water mark for data device: sending event.
kernel: device-mapper: thin: 253:2: no free data space available.
kernel: device-mapper: thin: 253:2: switching pool to read-only mode
kernel: device-mapper: thin: 253:2: switching pool to write mode (from PM_WRITE)
kernel: device-mapper: thin: 253:2: metadata operation 'dm_pool_commit_metadata' failed: error = -1
kernel: device-mapper: thin: 253:2: switching pool to read-only mode
kernel: device-mapper: thin: 253:2: growing the data device from 6144 to 8192 blocks
kernel: ------------[ cut here ]------------
kernel: WARNING: CPU: 1 PID: 22083 at drivers/md/dm-thin.c:934 alloc_data_block+0x171/0x1a0 [dm_thin_pool]()

dm-thin.c:934 was a WARN_ON() I added if alloc_data_block() was
called when pool mode != PM_WRITE.

The above clearly illustrated that alloc_data_block was still getting
called like crazy even though the pool went read-only waiting for a
resize.  This, and the following example, occurred due to the thin
device calling wake_worker() which kicked do_worker() during
pool_preresume() -- which caused calls to prcess_* methods to occur
during the resize.

2)
And this failure offers a clear indicator we weren't properly resizing,
but we were operating like we did:

kernel: device-mapper: thin: 253:2: switching pool to write mode
kernel: device-mapper: thin: 253:2: growing the data device from 6144 to 8192 blocks
kernel: device-mapper: thin: 253:2: no free data space available.
kernel: device-mapper: thin: 253:2: switching pool to read-only mode
kernel: device-mapper: thin: 253:2: switching pool to write mode
kernel: device-mapper: thin: 253:2: growing the data device from 6144 to 10240 blocks
kernel: attempt to access beyond end of device
kernel: dm-2: rw=17, want=1310848, limit=1310720
kernel: attempt to access beyond end of device
kernel: dm-2: rw=17, want=1310976, limit=1310720

Signed-off-by: Mike Snitzer <snitzer@redhat.com>