doc: Update and wordsmith rculist_nulls.rst

Do some wordsmithing and breaking up of RCU readers.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

Documentation/RCU/rculist_nulls.rst

@@ -14,19 +14,19 @@ Using 'nulls'
 =============
 
 Using special makers (called 'nulls') is a convenient way
-to solve following problem :
+to solve following problem.
 
-A typical RCU linked list managing objects which are
+Without 'nulls', a typical RCU linked list managing objects which are
-allocated with SLAB_TYPESAFE_BY_RCU kmem_cache can
+allocated with SLAB_TYPESAFE_BY_RCU kmem_cache can use the following
-use following algos :
+algorithms:
 
-1) Lookup algo
+1) Lookup algorithm
---------------
+-------------------
 
 ::
 
-  rcu_read_lock()
   begin:
+  rcu_read_lock()
   obj = lockless_lookup(key);
   if (obj) {
     if (!try_get_ref(obj)) // might fail for free objects
@@ -38,6 +38,7 @@ use following algos :
     */
     if (obj->key != key) { // not the object we expected
       put_ref(obj);
+      rcu_read_unlock();
       goto begin;
     }
   }
@@ -82,14 +83,14 @@ Quoting Corey Minyard::
   solved by pre-fetching the "next" field (with proper barriers) before
   checking the key."
 
-2) Insert algo
+2) Insertion algorithm
---------------
+----------------------
 
 We need to make sure a reader cannot read the new 'obj->obj_next' value
-and previous value of 'obj->key'. Or else, an item could be deleted
+and previous value of 'obj->key'. Otherwise, an item could be deleted
 from a chain, and inserted into another chain. If new chain was empty
-before the move, 'next' pointer is NULL, and lockless reader can
+before the move, 'next' pointer is NULL, and lockless reader can not
-not detect it missed following items in original chain.
+detect the fact that it missed following items in original chain.
 
 ::
 
@@ -100,18 +101,14 @@ not detect it missed following items in original chain.
   obj = kmem_cache_alloc(...);
   lock_chain(); // typically a spin_lock()
   obj->key = key;
-  /*
+  atomic_set_release(&obj->refcnt, 1); // key before refcnt
-  * we need to make sure obj->key is updated before obj->next
-  * or obj->refcnt
-  */
-  smp_wmb();
-  atomic_set(&obj->refcnt, 1);
   hlist_add_head_rcu(&obj->obj_node, list);
   unlock_chain(); // typically a spin_unlock()
 
 
-3) Remove algo
+3) Removal algorithm
---------------
+--------------------
+
 Nothing special here, we can use a standard RCU hlist deletion.
 But thanks to SLAB_TYPESAFE_BY_RCU, beware a deleted object can be reused
 very very fast (before the end of RCU grace period)
@@ -133,7 +130,7 @@ Avoiding extra smp_rmb()
 ========================
 
 With hlist_nulls we can avoid extra smp_rmb() in lockless_lookup()
-and extra smp_wmb() in insert function.
+and extra _release() in insert function.
 
 For example, if we choose to store the slot number as the 'nulls'
 end-of-list marker for each slot of the hash table, we can detect
@@ -142,42 +139,48 @@ to another chain) checking the final 'nulls' value if
 the lookup met the end of chain. If final 'nulls' value
 is not the slot number, then we must restart the lookup at
 the beginning. If the object was moved to the same chain,
-then the reader doesn't care : It might eventually
+then the reader doesn't care: It might occasionally
 scan the list again without harm.
 
 
-1) lookup algo
+1) lookup algorithm
---------------
+-------------------
 
 ::
 
   head = &table[slot];
-  rcu_read_lock();
   begin:
+  rcu_read_lock();
   hlist_nulls_for_each_entry_rcu(obj, node, head, member) {
     if (obj->key == key) {
-      if (!try_get_ref(obj)) // might fail for free objects
+      if (!try_get_ref(obj)) { // might fail for free objects
+	rcu_read_unlock();
         goto begin;
+      }
       if (obj->key != key) { // not the object we expected
         put_ref(obj);
+	rcu_read_unlock();
         goto begin;
       }
       goto out;
     }
-  /*
+  }
-  * if the nulls value we got at the end of this lookup is
+
-  * not the expected one, we must restart lookup.
+  // If the nulls value we got at the end of this lookup is
-  * We probably met an item that was moved to another chain.
+  // not the expected one, we must restart lookup.
-  */
+  // We probably met an item that was moved to another chain.
-  if (get_nulls_value(node) != slot)
+  if (get_nulls_value(node) != slot) {
+    put_ref(obj);
+    rcu_read_unlock();
     goto begin;
+  }
   obj = NULL;
 
   out:
   rcu_read_unlock();
 
-2) Insert function
+2) Insert algorithm
-------------------
+-------------------
 
 ::
 
@@ -188,11 +191,7 @@ scan the list again without harm.
   obj = kmem_cache_alloc(cachep);
   lock_chain(); // typically a spin_lock()
   obj->key = key;
-  /*
+  atomic_set_release(&obj->refcnt, 1); // key before refcnt
-  * changes to obj->key must be visible before refcnt one
-  */
-  smp_wmb();
-  atomic_set(&obj->refcnt, 1);
   /*
    * insert obj in RCU way (readers might be traversing chain)
    */