31 Oct, 2006
2 commits
-
When the ioprio code recently got juggled a bit, a bug was introduced.
changed_ioprio() is no longer called with interrupts disabled, so using
plain spin_lock() on the queue_lock is a bug.Signed-off-by: Jens Axboe
Signed-off-by: Linus Torvalds -
If cfq_set_request() is called for a new process AND a non-fs io
request (so that __GFP_WAIT may not be set), cfq_cic_link() may
use spin_lock_irq() and spin_unlock_irq() with interrupts already
disabled.Fix is to always use irq safe locking in cfq_cic_link()
Acked-By: Arjan van de Ven
Acked-by: Ingo Molnar
Signed-off-by: Jens Axboe
Signed-off-by: Linus Torvalds
01 Oct, 2006
18 commits
-
All on stack DECLARE_COMPLETIONs should be replaced by:
DECLARE_COMPLETION_ONSTACKSigned-off-by: Peter Zijlstra
Acked-by: Ingo Molnar
Acked-by: Benjamin Herrenschmidt
Signed-off-by: Andrew Morton
Signed-off-by: Linus Torvalds -
As people often look for the copyright in files to see who to mail,
update the link to a neutral one.Signed-off-by: Jens Axboe
-
Give meta data reads preference over regular reads, as the process
often needs to get that out of the way to do the io it was actually
interested in.Signed-off-by: Jens Axboe
-
Don't touch the current queues, just make sure that the wanted queue
is selected next. Simplifies the logic.Signed-off-by: Jens Axboe
-
CFQ implements this on its own now, but it's really block layer
knowledge. Tells a device queue to start dispatching requests to
the driver, taking care to unplug if needed. Also fixes the issue
where as/cfq will invoke a stopped queue, which we really don't
want.Signed-off-by: Jens Axboe
-
No point in having a place holder list just for empty queues, so remove
it. It's not used for anything other than to keep ->cfq_list busy.Signed-off-by: Jens Axboe
-
Currently it scales with number of processes in that priority group,
which is potentially not very nice as it's called quite often.
Basically we always need to do tail inserts, except for the case of a
new process. So just mark/detect a queue as such.Signed-off-by: Jens Axboe
-
Some were kmalloc_node(), some were still kmalloc(). Change them all to
kmalloc_node().Signed-off-by: Jens Axboe
-
Kill a few inlines that bring in too much code to more than one location
Shrinks kernel text by about 300 bytes on 32-bit x86.Signed-off-by: Jens Axboe
-
It's ok if the read path is a lot more costly, as long as inc/dec is
really cheap. The inc/dec will happen for each created/freed io context,
while the reading only happens when a disk queue exits.Signed-off-by: Jens Axboe
-
cfq_exit_lock is protecting two things now:
- The per-ioc rbtree of cfq_io_contexts
- The per-cfqd linked list of cfq_io_contexts
The per-cfqd linked list can be protected by the queue lock, as it is (by
definition) per cfqd as the queue lock is.The per-ioc rbtree is mainly used and updated by the process itself only.
The only outside use is the io priority changing. If we move the
priority changing to not browsing the rbtree, we can remove any locking
from the rbtree updates and lookup completely. Let the sys_ioprio syscall
just mark processes as having the iopriority changed and lazily update
the private cfq io contexts the next time io is queued, and we can
remove this locking as well.Signed-off-by: Jens Axboe
-
A collection of little fixes and cleanups:
- We don't use the 'queued' sysfs exported attribute, since the
may_queue() logic was rewritten. So kill it.- Remove dead defines.
- cfq_set_active_queue() can be rewritten cleaner with else if conditions.
- Several places had cfq_exit_cfqq() like logic, abstract that out and
use that.- Annotate the cfqq kmem_cache_alloc() so the allocator knows that this
is a repeat allocation if it fails with __GFP_WAIT set. Allows the
allocator to start freeing some memory, if needed. CFQ already loops for
this condition, so might as well pass the hint down.- Remove cfqd->rq_starved logic. It's not needed anymore after we dropped
the crq allocation in cfq_set_request().- Remove uneeded parameter passing.
Signed-off-by: Jens Axboe
-
It's not needed for anything, so kill the bio passing.
Signed-off-by: Jens Axboe
-
Get rid of the cfq_rq request type. With the added elevator_private2, we
have enough room in struct request to get rid of any crq allocation/free
for each request.Signed-off-by: Jens Axboe
-
There's just one flag currently (SYNC), and that one can be grabbed from
the request.Signed-off-by: Jens Axboe
-
Signed-off-by: Jens Axboe
-
This removes the rbtree handling from CFQ.
Signed-off-by: Jens Axboe
-
Right now, every IO scheduler implements its own backmerging (except for
noop, which does no merging). That results in duplicated code for
essentially the same operation, which is never a good thing. This patch
moves the backmerging out of the io schedulers and into the elevator
core. We save 1.6kb of text and as a bonus get backmerging for noop as
well. Win-win!Signed-off-by: Jens Axboe
21 Aug, 2006
1 commit
-
Obviously, cfq_cic_link() shouldn't free a just allocated cfq_io_context?
The dead key is from __cic, so drop that.Signed-off-by: Oleg Nesterov
Signed-off-by: Jens Axboe
25 Jul, 2006
1 commit
-
The CIC_SEEKY() test really wants to use the minimum of either:
- 2 msecs (not jiffies)
- or, the pending slice time
So code it like that.
Signed-off-by: Jens Axboe
01 Jul, 2006
1 commit
-
Signed-off-by: Jörn Engel
Signed-off-by: Adrian Bunk
23 Jun, 2006
6 commits
-
They all duplicate macros to check for empty root and/or node, and
clearing a node. So put those in rbtree.h.Signed-off-by: Jens Axboe
-
- Remember to set ->last_sector so that the cfq_choose_req() logic
works correctly.- Remove redundant call to cfq_choose_req()
Signed-off-by: Jens Axboe
-
This is a collection of patches that greatly improve CFQ performance
in some circumstances.- Change the idling logic to only kick in after a request is done and we
are deciding what to do. Before the idling included the request service
time, so it was hard to adjust. Now it's true think/idle time.- Take advantage of TCQ/NCQ/queueing for seeky sync workloads, but keep
it in control for sync and sequential (or close to) workloads.- Expire queues immediately and move on to other busy queues, if we are
not going to idle after the current one finishes.- Don't rearm idle timer if there are no busy queues. Just leave the
system idle.Signed-off-by: Jens Axboe
-
Patch originally from Vasily Tarasov
If you set io-priority of process 1 using sys_ioprio_set system call by
another process 2 (like ionice do), then cfq_init_prio_data() function
sets priority of process 2 (current) on queue of process 1 and clears
the flag, that designates change of ioprio. So the process 1 will work
like with priority of process 2.I propose not to call cfq_init_prio_data() on io-priority change, but
only mark queue as queue with changed prority. Every time when new
request comes cfq-scheduler checks for this flag and atomaticaly changes
priority of queue to new value.Signed-off-by: Jens Axboe
-
A process flag to indicate whether we are doing sync io is incredibly
ugly. It also causes performance problems when one does a lot of async
io and then proceeds to sync it. Part of the io will go out as async,
and the other part as sync. This causes a disconnect between the
previously submitted io and the synced io. For io schedulers such as CFQ,
this will cause us lost merges and suboptimal behaviour in scheduling.Remove PF_SYNCWRITE completely from the fsync/msync paths, and let
the O_DIRECT path just directly indicate that the writes are sync
by using WRITE_SYNC instead.Signed-off-by: Jens Axboe
-
We cannot update them if the user changes nr_requests, so don't
set it in the first place. The gains are pretty questionable as
well. The batching loss has been shown to decrease throughput.Signed-off-by: Jens Axboe
21 Jun, 2006
2 commits
-
The color is now in the low bits of the parent pointer, and initializing
it to 0 happens as part of the whole memset above, so just remove the
unnecessary RB_CLEAR_COLOR.Signed-off-by: Linus Torvalds
-
* git://git.infradead.org/~dwmw2/rbtree-2.6:
[RBTREE] Switch rb_colour() et al to en_US spelling of 'color' for consistency
Update UML kernel/physmem.c to use rb_parent() accessor macro
[RBTREE] Update hrtimers to use rb_parent() accessor macro.
[RBTREE] Add explicit alignment to sizeof(long) for struct rb_node.
[RBTREE] Merge colour and parent fields of struct rb_node.
[RBTREE] Remove dead code in rb_erase()
[RBTREE] Update JFFS2 to use rb_parent() accessor macro.
[RBTREE] Update eventpoll.c to use rb_parent() accessor macro.
[RBTREE] Update key.c to use rb_parent() accessor macro.
[RBTREE] Update ext3 to use rb_parent() accessor macro.
[RBTREE] Change rbtree off-tree marking in I/O schedulers.
[RBTREE] Add accessor macros for colour and parent fields of rb_node
15 Jun, 2006
1 commit
-
We don't clear the seek stat values in cfq_alloc_io_context(), and if
->seek_mean is unlucky enough to be set to -36 by chance, the first
invocation of cfq_update_io_seektime() will oops with a divide by zero
in do_div().Just memset the entire cic instead of filling invididual values
independently.Signed-off-by: Jens Axboe
Signed-off-by: Linus Torvalds
09 Jun, 2006
1 commit
-
There's a race between shutting down one io scheduler and firing up the
next, in which a new io could enter and cause the io scheduler to be
invoked with bad or NULL data.To fix this, we need to maintain the queue lock for a bit longer.
Unfortunately we cannot do that, since the elevator init requires to be
run without the lock held. This isn't easily fixable, without also
changing the mempool API. So split the initialization into two parts,
and alloc-init operation and an attach operation. Then we can
preallocate the io scheduler and related structures, and run the attach
inside the lock after we detach the old one.This patch has survived 30 minutes of 1 second io scheduler switching
with a very busy io load.Signed-off-by: Jens Axboe
Signed-off-by: Linus Torvalds
02 Jun, 2006
1 commit
-
Now that we select busy_rr for possible service, insert entries at the
back of that list instead of at the front.Signed-off-by: Jens Axboe
01 Jun, 2006
4 commits
-
There's a small window from when the timer is entered and we grab
the queue lock, where cfq_set_active_queue() could be rearming the
timer for us. Seen in the wild on a 12-way ppc box. Fix this by
just using mod_timer(), which will do the right thing for us.Signed-off-by: Jens Axboe
-
If the hardware is doing real queueing, decide that it's worthless to
idle the hardware. It does reasonable simultaneous io in that case
anyways, and the idling hurts some work loads.Signed-off-by: Jens Axboe
-
If we are anticipating a sync request from this process and we are
waiting for that and see an async request come in, expire that slice
and move on.Signed-off-by: Jens Axboe
-
For just one busy queue (like async write out), we often overlooked
that we could queue more io and decided we were idle instead. This causes
us quite a bit of performance loss.Signed-off-by: Jens Axboe
31 May, 2006
1 commit
-
- Drop cic from the list when seen as dead.
- Fixup the locking, just use a simple spinlock.Signed-off-by: Jens Axboe
Signed-off-by: Linus Torvalds
21 Apr, 2006
1 commit
-
They were abusing the rb_color field to mark nodes which weren't currently
on the tree. Fix that to use the same method as eventpoll did -- setting
the parent pointer to point back to itself. And use the appropriate
accessor macros for setting and reading the parent.Signed-off-by: David Woodhouse
