Re: Block device throttling [Re: Distributed storage.]

Previous message: [thread] [date] [author]
Next message: [thread] [date] [author]
From: Daniel Phillips
Date: Monday, August 13, 2007 - 6:04 am

On Monday 13 August 2007 05:18, Evgeniy Polyakov wrote:

The problem is, the sleep does not occur before the virtual device 
mapping function is called.  Let's consider two devices, a physical 
device named pdev and a virtual device sitting on top of it called 
vdev.   vdev's throttle limit is just one element, but we will see that 
in spite of this, two bios can be handled by the vdev's mapping method 
before any IO completes, which violates the throttling rules. According 
to your patch it works like this:

                     Thread 1                                Thread  2

	<no wait because vdev->bio_queued is zero>

	vdev->q->bio_queued++

	<enter devmapper map method>

	blk_set_bdev(bio, pdev)
	     vdev->bio_queued--
	   
					<no wait because vdev->bio_queued is zero>

					vdev->q->bio_queued++

					<enter devmapper map method>

					whoops!  Our virtual device mapping
					function has now allocated resources
					for two in-flight bios in spite of having its
					throttle limit set to 1.

Perhaps you never worried about the resources that the device mapper 
mapping function allocates to handle each bio and so did not consider 
this hole significant.  These resources can be significant, as is the 
case with ddsnap.  It is essential to close that window through with 
the virtual device's queue limit may be violated.  Not doing so will 
allow deadlock.

Regards,

Daniel
-
Previous message: [thread] [date] [author]
Next message: [thread] [date] [author]

Messages in current thread:
Distributed storage., Evgeniy Polyakov, (Tue Jul 31, 10:13 am)
Re: Distributed storage., Daniel Phillips, (Thu Aug 2, 2:08 pm)
Re: Distributed storage., Mike Snitzer, (Thu Aug 2, 9:09 pm)
Re: Distributed storage., Manu Abraham, (Thu Aug 2, 10:04 pm)
Re: Distributed storage., Evgeniy Polyakov, (Fri Aug 3, 3:26 am)
Re: Distributed storage., Evgeniy Polyakov, (Fri Aug 3, 3:42 am)
Re: Distributed storage., Evgeniy Polyakov, (Fri Aug 3, 3:44 am)
Re: Distributed storage., Evgeniy Polyakov, (Fri Aug 3, 3:57 am)
Re: Distributed storage., Peter Zijlstra, (Fri Aug 3, 5:27 am)
Re: Distributed storage., Evgeniy Polyakov, (Fri Aug 3, 6:49 am)
Re: Distributed storage., Peter Zijlstra, (Fri Aug 3, 7:53 am)
Re: Distributed storage., Daniel Phillips, (Fri Aug 3, 12:41 pm)
Re: Distributed storage., Daniel Phillips, (Fri Aug 3, 12:48 pm)
Re: Distributed storage., Daniel Phillips, (Fri Aug 3, 5:41 pm)
Re: Distributed storage., Daniel Phillips, (Fri Aug 3, 5:49 pm)
Re: Distributed storage., Daniel Phillips, (Fri Aug 3, 6:19 pm)
Re: Distributed storage., Dave Dillow, (Fri Aug 3, 7:51 pm)
Re: Distributed storage., Manu Abraham, (Fri Aug 3, 8:44 pm)
Re: Distributed storage., Evgeniy Polyakov, (Sat Aug 4, 9:37 am)
Re: Distributed storage., Evgeniy Polyakov, (Sat Aug 4, 9:44 am)
Re: Distributed storage., Evgeniy Polyakov, (Sat Aug 4, 10:03 am)
Re: Distributed storage., Daniel Phillips, (Sun Aug 5, 1:04 am)
Re: Distributed storage., Daniel Phillips, (Sun Aug 5, 1:06 am)
Re: Distributed storage., Evgeniy Polyakov, (Sun Aug 5, 8:01 am)
Re: Distributed storage., Evgeniy Polyakov, (Sun Aug 5, 8:08 am)
Re: Distributed storage., Daniel Phillips, (Sun Aug 5, 2:23 pm)
Re: Distributed storage., Daniel Phillips, (Sun Aug 5, 2:35 pm)
Re: Distributed storage., Evgeniy Polyakov, (Mon Aug 6, 1:25 am)
Re: Distributed storage., Evgeniy Polyakov, (Mon Aug 6, 1:28 am)
Re: Distributed storage., Jens Axboe, (Tue Aug 7, 5:05 am)
Re: Distributed storage., Daniel Phillips, (Tue Aug 7, 11:24 am)
Re: Distributed storage., Jens Axboe, (Tue Aug 7, 1:55 pm)
Block device throttling [Re: Distributed storage.], Evgeniy Polyakov, (Wed Aug 8, 2:54 am)
[1/1] Block device throttling [Re: Distributed storage.], Evgeniy Polyakov, (Wed Aug 8, 3:17 am)
Re: [1/1] Block device throttling [Re: Distributed storage.], Evgeniy Polyakov, (Wed Aug 8, 6:28 am)
Re: [1/1] Block device throttling [Re: Distributed storage.], Daniel Phillips, (Sun Aug 12, 4:16 pm)
Re: Distributed storage., Daniel Phillips, (Sun Aug 12, 4:36 pm)
Re: Block device throttling [Re: Distributed storage.], Daniel Phillips, (Sun Aug 12, 10:22 pm)
Re: Block device throttling [Re: Distributed storage.], Daniel Phillips, (Sun Aug 12, 10:36 pm)
Re: Block device throttling [Re: Distributed storage.], Daniel Phillips, (Sun Aug 12, 11:44 pm)
Re: Distributed storage., Jens Axboe, (Mon Aug 13, 12:28 am)
Re: Distributed storage., Jens Axboe, (Mon Aug 13, 12:45 am)
Re: Block device throttling [Re: Distributed storage.], Evgeniy Polyakov, (Mon Aug 13, 1:14 am)
Re: [1/1] Block device throttling [Re: Distributed storage.], Evgeniy Polyakov, (Mon Aug 13, 1:18 am)
Re: Block device throttling [Re: Distributed storage.], Evgeniy Polyakov, (Mon Aug 13, 1:23 am)
Re: Distributed storage., Daniel Phillips, (Mon Aug 13, 1:59 am)
Re: Distributed storage., Daniel Phillips, (Mon Aug 13, 2:08 am)
Re: Distributed storage., Jens Axboe, (Mon Aug 13, 2:12 am)
Re: Distributed storage., Jens Axboe, (Mon Aug 13, 2:13 am)
Re: Distributed storage., Evgeniy Polyakov, (Mon Aug 13, 2:18 am)
Re: Distributed storage., Daniel Phillips, (Mon Aug 13, 2:55 am)
Re: Distributed storage., Jens Axboe, (Mon Aug 13, 3:06 am)
Re: Distributed storage., Daniel Phillips, (Mon Aug 13, 3:12 am)
Re: Distributed storage., Daniel Phillips, (Mon Aug 13, 3:15 am)
Re: Distributed storage., Jens Axboe, (Mon Aug 13, 3:22 am)
Re: Distributed storage., Daniel Phillips, (Mon Aug 13, 3:32 am)
Re: Distributed storage., Evgeniy Polyakov, (Mon Aug 13, 4:03 am)
Re: Block device throttling [Re: Distributed storage.], Daniel Phillips, (Mon Aug 13, 4:04 am)
Re: Block device throttling [Re: Distributed storage.], Daniel Phillips, (Mon Aug 13, 4:18 am)
Re: Distributed storage., Daniel Phillips, (Mon Aug 13, 4:45 am)
Re: Block device throttling [Re: Distributed storage.], Evgeniy Polyakov, (Mon Aug 13, 5:04 am)
Re: Block device throttling [Re: Distributed storage.], Evgeniy Polyakov, (Mon Aug 13, 5:18 am)
Re: Block device throttling [Re: Distributed storage.], Daniel Phillips, (Mon Aug 13, 5:18 am)
Re: Block device throttling [Re: Distributed storage.], Evgeniy Polyakov, (Mon Aug 13, 5:24 am)
Re: Block device throttling [Re: Distributed storage.], Daniel Phillips, (Mon Aug 13, 6:04 am)
Re: Distributed storage., Daniel Phillips, (Mon Aug 13, 4:27 pm)
Re: Block device throttling [Re: Distributed storage.], Evgeniy Polyakov, (Tue Aug 14, 1:46 am)
Re: Block device throttling [Re: Distributed storage.], Daniel Phillips, (Tue Aug 14, 4:13 am)
Re: Block device throttling [Re: Distributed storage.], Evgeniy Polyakov, (Tue Aug 14, 4:30 am)
Re: Block device throttling [Re: Distributed storage.], Daniel Phillips, (Tue Aug 14, 4:35 am)
Re: Block device throttling [Re: Distributed storage.], Evgeniy Polyakov, (Tue Aug 14, 4:50 am)
Re: Block device throttling [Re: Distributed storage.], Daniel Phillips, (Tue Aug 14, 5:32 am)
Re: Block device throttling [Re: Distributed storage.], Evgeniy Polyakov, (Tue Aug 14, 5:46 am)
Re: Block device throttling [Re: Distributed storage.], Daniel Phillips, (Tue Aug 14, 5:54 am)
Re: [1/1] Block device throttling [Re: Distributed storage.], Daniel Phillips, (Mon Aug 27, 2:57 pm)
Re: [1/1] Block device throttling [Re: Distributed storage.], Evgeniy Polyakov, (Tue Aug 28, 2:35 am)
Re: Distributed storage., Evgeniy Polyakov, (Tue Aug 28, 10:19 am)
Re: [1/1] Block device throttling [Re: Distributed storage.], Daniel Phillips, (Tue Aug 28, 10:27 am)
Re: [1/1] Block device throttling [Re: Distributed storage.], Evgeniy Polyakov, (Tue Aug 28, 10:54 am)
Re: [1/1] Block device throttling [Re: Distributed storage.], Daniel Phillips, (Tue Aug 28, 2:08 pm)
Re: [1/1] Block device throttling [Re: Distributed storage.], Evgeniy Polyakov, (Wed Aug 29, 1:53 am)
Re: [1/1] Block device throttling [Re: Distributed storage.], Daniel Phillips, (Thu Aug 30, 4:20 pm)
Re: [1/1] Block device throttling [Re: Distributed storage.], Evgeniy Polyakov, (Fri Aug 31, 10:33 am)
Re: [1/1] Block device throttling [Re: Distributed storage.], Alasdair G Kergon, (Fri Aug 31, 2:41 pm)