From: Xue jiufei <xuejiufei@huawei.com>
Date: Thu, 22 May 2014 00:42:31 +0000 (+1000)
Subject: ocfs2/dlm: fix possible convert=sion deadlock
X-Git-Tag: next-20140530~2^2~430
X-Git-Url: https://git.karo-electronics.de/?a=commitdiff_plain;h=6bb3beba75a0af2296ddf8e43a83d8fa8d056481;p=karo-tx-linux.git

ocfs2/dlm: fix possible convert=sion deadlock

We found there is a conversion deadlock when the owner of lockres happened
to crash before send DLM_PROXY_AST_MSG for a downconverting lock.  The
situation is as follows:

Node1                            Node2                  Node3
                           the owner of lockresA
lock_1 granted at EX mode
and call ocfs2_cluster_unlock
to decrease ex_holders.
                                                 converting lock_3 from
                                                 NL to EX
                           send DLM_PROXY_AST_MSG
                           to Node1, asking Node 1
                           to downconvert.
receiving DLM_PROXY_AST_MSG,
thread ocfs2dc send
DLM_CONVERT_LOCK_MSG
to Node2 to downconvert
lock_1(EX->NL).
                           lock_1 can be granted and
                           put it into pending_asts
                           list, return DLM_NORMAL.
                           then something happened
                           and Node2 crashed.
received DLM_NORMAL, waiting
for DLM_PROXY_AST_MSG.
                                               selected as the recovery
                                               master, receving migrate
                                               lock from Node1, queue
                                               lock_1 to the tail of
                                               converting list.

After dlm recovery, converting list in the master of lockresA(Node3) will
be: converting list head <-> lock_3(NL->EX) <->lock_1(EX<->NL).  Requested
mode of lock_3 is not compatible with the granted mode of lock_1, so it
can not be granted.  and lock_1 can not downconvert because covnerting
queue is strictly FIFO.  So a deadlock is created.  We think function
dlm_process_recovery_data() should queue_ast for lock_1 or alter the order
of lock_1 and lock_3, so dlm_thread can process lock_1 first.  And if
there are multiple downconverting locks, they must convert form PR to NL,
so no need to sort them.

Signed-off-by: joyce.xue <xuejiufei@huawei.com>
Cc: Mark Fasheh <mfasheh@suse.com>
Cc: Joel Becker <jlbec@evilplan.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---

diff --git a/fs/ocfs2/dlm/dlmrecovery.c b/fs/ocfs2/dlm/dlmrecovery.c
index fe29f7978f81..5de019437ea5 100644
--- a/fs/ocfs2/dlm/dlmrecovery.c
+++ b/fs/ocfs2/dlm/dlmrecovery.c
@@ -1986,7 +1986,15 @@ skip_lvb:
 		}
 		if (!bad) {
 			dlm_lock_get(newlock);
-			list_add_tail(&newlock->list, queue);
+			if (mres->flags & DLM_MRES_RECOVERY &&
+					ml->list == DLM_CONVERTING_LIST &&
+					newlock->ml.type >
+					newlock->ml.convert_type) {
+				/* newlock is doing downconvert, add it to the
+				 * head of converting list */
+				list_add(&newlock->list, queue);
+			} else
+				list_add_tail(&newlock->list, queue);
 			mlog(0, "%s:%.*s: added lock for node %u, "
 			     "setting refmap bit\n", dlm->name,
 			     res->lockname.len, res->lockname.name, ml->node);