30 May, 2014
4 commits
-
Currently blk-mq registers all the hardware queues in sysfs,
regardless of whether it uses them (e.g. they have CPU mappings)
or not. The unused hardware queues lack the cpux/ directories,
and the other sysfs entries (like active, pending, etc) are all
zeroes.Change this so that sysfs correctly reflects the current mappings
of the hardware queues.Signed-off-by: Jens Axboe
-
flush request is special, which borrows the tag from the parent
request. Hence blk_mq_tag_to_rq needs special handling to return
the flush request from the tag.Signed-off-by: Shaohua Li
Signed-off-by: Jens Axboe -
filter gets assigned the address of blk_default_cmd_filter on
entry to this function, so the !filter condition can never be true.Signed-off-by: Dave Jones
Signed-off-by: Jens Axboe -
We currently clear a lot more than we need to, so make that a bit
more clever. Make some of the init dependent on features, like
only setting start_time if we are going to use it.Signed-off-by: Jens Axboe
29 May, 2014
4 commits
-
If devices are not SG starved, we waste a lot of time potentially
collapsing SG segments. Enough that 1.5% of the CPU time goes
to this, at only 400K IOPS. Add a queue flag, QUEUE_FLAG_NO_SG_MERGE,
which just returns the number of vectors in a bio instead of looping
over all segments and checking for collapsible ones.Add a BLK_MQ_F_SG_MERGE flag so that drivers can opt-in on the sg
merging, if they so desire.Signed-off-by: Jens Axboe
-
I don't think we've ever caught any bugs with this, and there's the
list poisoning for the plug lists to catch uninitialized cases.
So remove the magic member and save 8 bytes in the struct.Signed-off-by: Jens Axboe
-
There is no need for drivers to control hardware context allocation
now that we do the context to node mapping in common code.Signed-off-by: Christoph Hellwig
Signed-off-by: Jens Axboe -
None of the blk-mq files have an explanatory comment at the top
for what that particular file does. Add that and add appropriate
copyright notices as well.Signed-off-by: Jens Axboe
28 May, 2014
9 commits
-
We now only have one caller left and can open code it there in a cleaner
way.Signed-off-by: Christoph Hellwig
Signed-off-by: Jens Axboe -
We already do a non-blocking allocation in blk_mq_map_request, no need
to repeat it. Just call __blk_mq_alloc_request to wait directly.Signed-off-by: Christoph Hellwig
Signed-off-by: Jens Axboe -
The current logic for blocking tag allocation is rather confusing, as we
first allocated and then free again a tag in blk_mq_wait_for_tags, just
to attempt a non-blocking allocation and then repeat if someone else
managed to grab the tag before us.Instead change blk_mq_alloc_request_pinned to simply do a blocking tag
allocation itself and use the request we get back from it.Signed-off-by: Christoph Hellwig
Signed-off-by: Jens Axboe -
Both callers if __blk_mq_alloc_request want to initialize the request, so
lift it into the common path.Signed-off-by: Christoph Hellwig
Signed-off-by: Jens Axboe -
Instead of having two almost identical copies of the same code just let
the callers pass in the reserved flag directly.Signed-off-by: Christoph Hellwig
Signed-off-by: Jens Axboe -
Both the cache flush state machine and the SCSI midlayer want to submit
requests from irq context, and the current per-request requeue_work
unfortunately causes corruption due to sharing with the csd field for
flushes. Replace them with a per-request_queue list of requests to
be requeued.Based on an earlier test by Ming Lei.
Signed-off-by: Christoph Hellwig
Reported-by: Ming Lei
Tested-by: Ming Lei
Signed-off-by: Jens Axboe -
It works for both IPI and local completions as of commit
95f096849932.Signed-off-by: Jens Axboe
-
Right now we export two ways of completing a request:
1) blk_mq_complete_request(). This uses an IPI (if needed) and
completes through q->softirq_done_fn(). It also works with
timeouts.2) blk_mq_end_io(). This completes inline, and ignores any timeout
state of the request.Let blk_mq_complete_request() handle non-softirq_done_fn completions
as well, by just completing inline. If a driver has enough completion
ports to place completions correctly, it need not define a
mq_ops->complete() and we can avoid an indirect function call by
doing the completion inline.Signed-off-by: Jens Axboe
-
Drivers currently have to figure this out on their own, and they
are missing information to do it properly. The ones that did
attempt to do it, do it wrong.So just pass in the suggested node directly to the alloc
function.Signed-off-by: Jens Axboe
27 May, 2014
5 commits
-
The percpu counter is only used for blk-mq, so move
its allocation and free inside blk-mq, and don't
allocate it for legacy queue device.Signed-off-by: Ming Lei
Signed-off-by: Jens Axboe -
blk_mq_exit_hw_queues() and blk_mq_free_hw_queues()
are introduced to avoid code duplication.Signed-off-by: Ming Lei
Signed-off-by: Jens Axboe -
hctx->ctx_map should have been freed inside blk_mq_free_queue().
Signed-off-by: Ming Lei
Signed-off-by: Jens Axboe -
__blkdev_issue_zeroout is only used in blk-lib.c
Cc: Jens Axboe
Cc: Andrew Morton
Signed-off-by: Fabian Frederick
Signed-off-by: Jens Axboe -
Without this we can leak the active_queues reference if a command is
freed while it is considered active.Signed-off-by: Christoph Hellwig
Signed-off-by: Jens Axboe
24 May, 2014
2 commits
-
Currently blk-mq uses the queue timeout for all requests. But
for some commands, drivers may want to set a specific timeout
for special requests. Allow this to be passed in through
request->timeout, and use it if set.Signed-off-by: Jens Axboe
-
Export the blk-mq in-flight tag iterator for driver consumption.
This is particularly useful in exception paths or SRSI where
in-flight IOs need to be cancelled and/or reissued. The NVMe driver
conversion will use this.Signed-off-by: Sam Bradshaw
Signed-off-by: Matias Bjørling
Signed-off-by: Jens Axboe
23 May, 2014
1 commit
-
We want slightly different behavior from them:
- On single queue devices, we currently use the per-process plug
for deferred IO and for merging.- On multi queue devices, we don't use the per-process plug, but
we want to go straight to hardware for SYNC IO.Split blk_mq_make_request() into a blk_sq_make_request() for single
queue devices, and retain blk_mq_make_request() for multi queue
devices. Then we don't need multiple checks for q->nr_hw_queues
in the request mapping.Signed-off-by: Jens Axboe
22 May, 2014
2 commits
-
Depending on the topology of the machine and the number of queues
exposed by a device, we can end up in a situation where some of
the hardware queues are unused (as in, they don't map to any
software queues). For this case, free up the memory used by the
request map, as we will not use it. This can be a substantial
amount of memory, depending on the number of queues vs CPUs and
the queue depth of the device.Signed-off-by: Jens Axboe
-
Prepare this for the next patch which adds more smarts in the
plugging logic, so that we can save some memory.Signed-off-by: Jens Axboe
21 May, 2014
3 commits
-
In blk_mq_make_request(), do the blk_queue_nomerges() check
outside the call to blk_attempt_plug_merge() to eliminate
function call overhead when nomerges=2 (disabled)Signed-off-by: Robert Elliott
Signed-off-by: Jens Axboe -
blk_queue_make_requests() overwrites our set value for q->nr_requests,
turning it into the default of 128. Set this appropriately after
initializing queue values in blk_queue_make_request().Signed-off-by: Jens Axboe
-
For request_fn based devices, the block layer exports a 'nr_requests'
file through sysfs to allow adjusting of queue depth on the fly.
Currently this returns -EINVAL for blk-mq, since it's not wired up.
Wire this up for blk-mq, so that it now also always dynamic
adjustments of the allowed queue depth for any given block device
managed by blk-mq.Signed-off-by: Jens Axboe
20 May, 2014
6 commits
-
Commit f9c78b2be2ca moved bio.c from fs/ to block/, but didn't
update the docbook location. Fix that up.Signed-off-by: Jens Axboe
-
Continue moving some of the block files that are scattered around.
bounce.c contains only code for bouncing the contents of a bio.
It's block proper code, not mm code.Suggested-by: Ming Lei
Signed-off-by: Jens Axboe -
Signed-off-by: Jens Axboe
Conflicts:
block/blk-mq-tag.c -
Each hardware queue has a bitmap of software queues with pending
requests. When new IO is queued on a software queue, the bit is
set, and when IO is pruned on a hardware queue run, the bit is
cleared. This causes a lot of traffic. Switch this from the regular
BITS_PER_LONG bitmap to a sparser layout, similarly to what was
done for blk-mq tagging.20% performance increase was observed for single threaded IO, and
about 15% performanc increase on multiple threads driving the
same device.Signed-off-by: Jens Axboe
-
We will use it for the pending list in blk-mq core as well.
Signed-off-by: Jens Axboe
-
Like commit f9c78b2b, move this block related file outside
of fs/ and into the core block directory, block/.Signed-off-by: Jens Axboe
19 May, 2014
1 commit
-
They really belong in block/, especially now since it's not in
drivers/block/ anymore. Additionally, the get_maintainer script
gets it wrong when in fs/.Suggested-by: Christoph Hellwig
Acked-by: Al Viro
Signed-off-by: Jens Axboe
14 May, 2014
1 commit
-
This adds support for active queue tracking, meaning that the
blk-mq tagging maintains a count of active users of a tag set.
This allows us to maintain a notion of fairness between users,
so that we can distribute the tag depth evenly without starving
some users while allowing others to try unfair deep queues.If sharing of a tag set is detected, each hardware queue will
track the depth of its own queue. And if this exceeds the total
depth divided by the number of active queues, the user is actively
throttled down.The active queue count is done lazily to avoid bouncing that data
between submitter and completer. Each hardware queue gets marked
active when it allocates its first tag, and gets marked inactive
when 1) the last tag is cleared, and 2) the queue timeout grace
period has passed.Signed-off-by: Jens Axboe
11 May, 2014
2 commits
-
Both nr_cache and nr_tags arn't needed for bitmap tag anymore.
Signed-off-by: Ming Lei
Signed-off-by: Jens Axboe