[PATCH] struct dentry: place d_hash close to d_parent and d_name to speedup lookups

dentry cache uses sophisticated RCU technology (and prefetching if available) but touches 2 cache lines per dentry during hlist lookup. This patch moves d_hash in the same cache line than d_parent and d_name fields so that : 1) One cache line is needed instead of two. 2) the hlist_for_each_rcu() prefetching has a chance to bring all the needed data in advance, not only the part that includes d_hash.next. I also changed one old comment that was wrong for 64bits. A further optimisation would be to separate dentry in two parts, one that is mostly read, and one writen (d_count/d_lock) to avoid false sharing on SMP/NUMA but this would need different field placement depending on 32bits or 64bits platform. Signed-off-by: Eric Dumazet <dada1@cosmosbay.com> Signed-off-by: Andrew Morton <akpm@osdl.org> Signed-off-by: Linus Torvalds <torvalds@osdl.org>

[PATCH] struct dentry: place d_hash close to d_parent and d_name to speedup lookups
dentry cache uses sophisticated RCU technology (and prefetching if available) but touches 2 cache lines per dentry during hlist lookup. This patch moves d_hash in the same cache line than d_parent and d_name fields so that : 1) One cache line is needed instead of two. 2) the hlist_for_each_rcu() prefetching has a chance to bring all the needed data in advance, not only the part that includes d_hash.next. I also changed one old comment that was wrong for 64bits. A further optimisation would be to separate dentry in two parts, one that is mostly read, and one writen (d_count/d_lock) to avoid false sharing on SMP/NUMA but this would need different field placement depending on 32bits or 64bits platform. Signed-off-by: Eric Dumazet <dada1@cosmosbay.com> Signed-off-by: Andrew Morton <akpm@osdl.org> Signed-off-by: Linus Torvalds <torvalds@osdl.org>
Eric Dumazet · Linus Torvalds
1 parent 0811bab24f
Showing 1 changed file with 2 additions and 2 deletions Side-by-side Diff
include/linux/dcache.h
@@ -88,8 +88,9 @@
 					 * negative */
 	/*
 	 * The next three fields are touched by __d_lookup.  Place them here
-	 * so they all fit in a 16-byte range, with 16-byte alignment.
+	 * so they all fit in a cache line.
 	 */
+	struct hlist_node d_hash;	/* lookup hash list */
 	struct dentry *d_parent;	/* parent directory */
 	struct qstr d_name;
  
@@ -103,7 +104,6 @@
 	void *d_fsdata;			/* fs-specific data */
  	struct rcu_head d_rcu;
 	struct dcookie_struct *d_cookie; /* cookie, if any */
-	struct hlist_node d_hash;	/* lookup hash list */	
 	int d_mounted;
 	unsigned char d_iname[DNAME_INLINE_LEN_MIN];	/* small names */
 };
...	...	@@ -88,8 +88,9 @@
88	88	* negative */
89	89	/*
90	90	* The next three fields are touched by __d_lookup. Place them here
91		- * so they all fit in a 16-byte range, with 16-byte alignment.
	91	+ * so they all fit in a cache line.
92	92	*/
	93	+ struct hlist_node d_hash; /* lookup hash list */
93	94	struct dentry d_parent; / parent directory */
94	95	struct qstr d_name;
95	96
...	...	@@ -103,7 +104,6 @@
103	104	void d_fsdata; / fs-specific data */
104	105	struct rcu_head d_rcu;
105	106	struct dcookie_struct d_cookie; / cookie, if any */
106		- struct hlist_node d_hash; /* lookup hash list */
107	107	int d_mounted;
108	108	unsigned char d_iname[DNAME_INLINE_LEN_MIN]; /* small names */
109	109	};