03 Jul, 2008
3 commits
-
Signed-off-by: Jens Axboe
-
Now that blktrace has the ability to carry arbitrary messages in
its stream, use that for some CFQ logging.Signed-off-by: Jens Axboe
-
If we have multiple tasks freeing cfq_io_contexts when cfq-iosched
is being unloaded, we could complete() ioc_gone twice. Fix that by
protecting ioc_gone complete() and clearing with a spinlock for
just that purpose. Doesn't matter from a performance perspective,
since it'll only enter that path when ioc_gone != NULL (when cfq-iosched
is being rmmod'ed).Signed-off-by: Jens Axboe
28 May, 2008
2 commits
-
cfq_cic_lookup() needs to properly protect ioc->ioc_data before
dereferencing it and also exclude updaters of ioc->ioc_data as well.Also add a number of comments documenting why the existing RCU usage
is OK.Thanks a lot to "Paul E. McKenney" for
review and comments!Signed-off-by: Jens Axboe
-
saves 8 bytes of padding & increases objects/slab from 30 to 32 on my
AMD64 configSigned-off-by: Richard Kennedy
Signed-off-by: Jens Axboe
07 May, 2008
2 commits
-
We currently set all processes to the best-effort scheduling class,
regardless of what CPU scheduling class they belong to. Improve that
so that we correctly track idle and rt scheduling classes as well.Signed-off-by: Jens Axboe
-
put_io_context() drops the RCU read lock before calling into cfq_dtor(),
however we need to hold off freeing there before grabbing and
dereferencing the first object on the list.So extend the rcu_read_lock() scope to cover the calling of cfq_dtor(),
and optimize cfq_free_io_context() to use a new variant for
call_for_each_cic() that assumes the RCU read lock is already held.Hit in the wild by Alexey Dobriyan
Signed-off-by: Jens Axboe
10 Apr, 2008
1 commit
-
When switching scheduler from cfq, cfq_exit_queue() does not clear
ioc->ioc_data, leaving a dangling pointer that can deceive the following
lookups when the iosched is switched back to cfq. The pattern that can
trigger that is the following:- elevator switch from cfq to something else;
- module unloading, with elv_unregister() that calls cfq_free_io_context()
on ioc freeing the cic (via the .trim op);
- module gets reloaded and the elevator switches back to cfq;
- reallocation of a cic at the same address as before (with a valid key).To fix it just assign NULL to ioc_data in __cfq_exit_single_io_context(),
that is called from the regular exit path and from the elevator switching
code. The only path that frees a cic and is not covered is the error handling
one, but cic's freed in this way are never cached in ioc_data.Signed-off-by: Fabio Checconi
Signed-off-by: Jens Axboe
02 Apr, 2008
1 commit
-
SLAB_DESTROY_BY_RCU is not a direct substitute for normal call_rcu()
freeing, since it'll page freeing but NOT object freeing. So change
cfq to do the freeing on its own.Signed-off-by: Fabio Checconi
Acked-by: Paul E. McKenney
Signed-off-by: Jens Axboe
19 Feb, 2008
1 commit
-
It's cumbersome to browse a radix tree from start to finish, especially
since we modify keys when a process exits. So add a hlist for the single
purpose of browsing over all known cfq_io_contexts, used for exit,
io prio change, etc.This fixes http://bugzilla.kernel.org/show_bug.cgi?id=9948
Signed-off-by: Jens Axboe
01 Feb, 2008
1 commit
-
Signed-off-by: Jens Axboe
28 Jan, 2008
5 commits
-
Use of inlines were a bit over the top, trim them down a bit.
Signed-off-by: Jens Axboe
-
Currently you must be root to set idle io prio class on a process. This
is due to the fact that the idle class is implemented as a true idle
class, meaning that it will not make progress if someone else is
requesting disk access. Unfortunately this means that it opens DOS
opportunities by locking down file system resources, hence it is root
only at the moment.This patch relaxes the idle class a little, by removing the truly idle
part (which entals a grace period with associated timer). The
modifications make the idle class as close to zero impact as can be done
while still guarenteeing progress. This means we can relax the root only
criteria as well.Signed-off-by: Jens Axboe
-
The io context sharing introduced a per-ioc spinlock, that would protect
the cfq io context lookup. That is a regression from the original, since
we never needed any locking there because the ioc/cic were process private.The cic lookup is changed from an rbtree construct to a radix tree, which
we can then use RCU to make the reader side lockless. That is the performance
critical path, modifying the radix tree is only done on process creation
(when that process first does IO, actually) and on process exit (if that
process has done IO).As it so happens, radix trees are also much faster for this type of
lookup where the key is a pointer. It's a very sparse tree.Signed-off-by: Jens Axboe
-
changes in the cfq for io_context sharing
Signed-off-by: Jens Axboe
-
This is where it belongs and then it doesn't take up space for a
process that doesn't do IO.Signed-off-by: Jens Axboe
18 Dec, 2007
1 commit
-
elv_register() always returns 0, and there isn't anything it does where
it should return an error (the only error condition is so grave that
it's handled with a BUG_ON).Signed-off-by: Adrian Bunk
Signed-off-by: Jens Axboe
07 Nov, 2007
3 commits
-
In theory, if the queue was idle long enough, cfq_idle_class_timer may have
a false (and very long) timeout because jiffies can wrap into the past wrt
->last_end_request.Signed-off-by: Oleg Nesterov
Signed-off-by: Jens Axboe -
After the fresh boot:
ionice -c3 -p $$
echo cfq >> /sys/block/XXX/queue/scheduler
dd if=/dev/XXX of=/dev/null bs=512 count=1Now dd hangs in D state and the queue is completely stalled for approximately
INITIAL_JIFFIES + CFQ_IDLE_GRACE jiffies. This is because cfq_init_queue()
forgets to initialize cfq_data->last_end_request.(I guess this patch is not complete, overflow is still possible)
Signed-off-by: Oleg Nesterov
Signed-off-by: Jens Axboe -
Spotted by Nick , hopefully can explain the second trace in
http://bugzilla.kernel.org/show_bug.cgi?id=9180.If ->async_idle_cfqq != NULL cfq_put_async_queues() puts it IOPRIO_BE_NR times
in a loop. Fix this.Signed-off-by: Oleg Nesterov
Signed-off-by: Jens Axboe
29 Oct, 2007
2 commits
-
cfq_get_queue()->cfq_find_alloc_queue() can fail, check the returned value.
Signed-off-by: Oleg Nesterov
Note that this isn't a bug at the moment, since the regular IO path
does not call this path without __GFP_WAIT set. However, it could be a
future bug, so I've applied it.Signed-off-by: Jens Axboe
-
Spotted by Nick , perhaps explains the first trace in
http://bugzilla.kernel.org/show_bug.cgi?id=9180.cfq_exit_queue() should cancel cfqd->unplug_work before freeing cfqd.
blk_sync_queue() seems unneeded, removed.Q: why cfq_exit_queue() calls cfq_shutdown_timer_wq() twice?
Signed-off-by: Oleg Nesterov
Signed-off-by: Jens Axboe
24 Jul, 2007
1 commit
-
Some of the code has been gradually transitioned to using the proper
struct request_queue, but there's lots left. So do a full sweet of
the kernel and get rid of this typedef and replace its uses with
the proper type.Signed-off-by: Jens Axboe
20 Jul, 2007
2 commits
-
There are some leftover bits from the task cooperator patch, that was
yanked out again. While it will get reintroduced, no point in having
this write-only stuff in the tree. So yank it.Signed-off-by: Jens Axboe
-
If we have two processes with different ioprio_class, but the same
ioprio_data, their async requests will fall into the same queue. I guess
such behavior is not expected, because it's not right to put real-time
requests and best-effort requests in the same queue.The attached patch fixes the problem by introducing additional *cfqq
fields on cfqd, pointing to per-(class,priority) async queues.Signed-off-by: Jens Axboe
18 Jul, 2007
1 commit
-
kmalloc_node() and kmem_cache_alloc_node() were not available in a zeroing
variant in the past. But with __GFP_ZERO it is possible now to do zeroing
while allocating.Use __GFP_ZERO to remove the explicit clearing of memory via memset whereever
we can.Signed-off-by: Christoph Lameter
Signed-off-by: Andrew Morton
Signed-off-by: Linus Torvalds
10 Jul, 2007
1 commit
-
With the cfq_queue hash removal, we inadvertently got rid of the
async queue sharing. This was not intentional, in fact CFQ purposely
shares the async queue per priority level to get good merging for
async writes.So put some logic in cfq_get_queue() to track the shared queues.
Signed-off-by: Jens Axboe
08 May, 2007
1 commit
-
This patch provides a new macro
KMEM_CACHE(, )
to simplify slab creation. KMEM_CACHE creates a slab with the name of the
struct, with the size of the struct and with the alignment of the struct.
Additional slab flags may be specified if necessary.Example
struct test_slab {
int a,b,c;
struct list_head;
} __cacheline_aligned_in_smp;test_slab_cache = KMEM_CACHE(test_slab, SLAB_PANIC)
will create a new slab named "test_slab" of the size sizeof(struct
test_slab) and aligned to the alignment of test slab. If it fails then we
panic.Signed-off-by: Christoph Lameter
Signed-off-by: Andrew Morton
Signed-off-by: Linus Torvalds
30 Apr, 2007
12 commits
-
We often lookup the same queue many times in succession, so cache
the last looked up queue to avoid browsing the rbtree.Signed-off-by: Jens Axboe
-
cfq hash is no more necessary. We always can get cfqq from io context.
cfq_get_io_context_noalloc() function is introduced, because we don't
want to allocate cic on merging and checking may_queue. In order to
identify sync queue we've used hash key = CFQ_KEY_ASYNC. Since hash is
eliminated we need to use other criterion: sync flag for queue is added.
In all places where we dig in rb_tree we're in current context, so no
additional locking is required.Advantages of this patch: no additional memory for hash, no seeking in
hash, code is cleaner. But it is necessary now to seek cic in per-ioc
rbtree, but it is faster:
- most processes work only with few devices
- most systems have only few block devices
- it is a rb-treeSigned-off-by: Vasily Tarasov
Changes by me:
- Merge into CFQ devel branch
- Get rid of cfq_get_io_context_noalloc()
- Fix various bugs with dereferencing cic->cfqq[] with offset other
than 0 or 1.
- Fix bug in cfqq setup, is_sync condition was reversed.
- Fix bug where only bio_sync() is used, we need to check for a READ tooSigned-off-by: Jens Axboe
-
For tagged devices, allow overlap of requests if the idle window
isn't enabled on the current active queue.Signed-off-by: Jens Axboe
-
Signed-off-by: Jens Axboe
-
We don't enable it by default, don't let it get enabled during
runtime.Signed-off-by: Jens Axboe
-
We can track it fairly accurately locally, let the slice handling
take care of the rest.Signed-off-by: Jens Axboe
-
We don't use it anymore in the slice expiry handling.
Signed-off-by: Jens Axboe
-
It's only used for preemption now that the IDLE and RT queues also
use the rbtree. If we pass an 'add_front' variable to
cfq_service_tree_add(), we can set ->rb_key to 0 to force insertion
at the front of the tree.Signed-off-by: Jens Axboe
-
Use the max_slice-cur_slice as the multipler for the insertion offset.
Signed-off-by: Jens Axboe
-
Signed-off-by: Jens Axboe
-
Same treatment as the RT conversion, just put the sorted idle
branch at the end of the tree.Signed-off-by: Jens Axboe
-
Currently CFQ does a linked insert into the current list for RT
queues. We can just factor the class into the rb insertion,
and then we don't have to treat RT queues in a special way. It's
faster, too.Signed-off-by: Jens Axboe