io_uring: use percpu counters to track inflight requests

Even though we place the req_issued and req_complete in separate
cachelines, there's considerable overhead in doing the atomics
particularly on the completion side.

Get rid of having the two counters, and just use a percpu_counter for
this. That's what it was made for, after all. This considerably
reduces the overhead in __io_free_req().

Signed-off-by: Jens Axboe <axboe@kernel.dk>
This commit is contained in:
Jens Axboe
2020-10-15 16:24:45 -06:00
parent 500a373d73
commit d8a6df10aa
2 changed files with 30 additions and 27 deletions

View File

@@ -23,13 +23,10 @@ struct io_uring_task {
struct xarray xa;
struct wait_queue_head wait;
struct file *last;
atomic_long_t req_issue;
struct percpu_counter inflight;
struct io_identity __identity;
struct io_identity *identity;
/* completion side */
bool in_idle ____cacheline_aligned_in_smp;
atomic_long_t req_complete;
bool in_idle;
};
#if defined(CONFIG_IO_URING)