[Linux-cluster] gnbd and caching

[Benjamin Marzinski]

On Thu, Jul 26, 2007 at 03:32:20PM +0200, Markus Hochholdinger wrote:
> Hi,
> 
> Am Donnerstag, 26. Juli 2007 02:12 schrieb Benjamin Marzinski:
> > On Tue, Jul 24, 2007 at 05:34:16PM +0200, Markus Hochholdinger wrote:
> > > i'm running some standalone gnbd servers (option -c) and i'm wondering
> > > where the (read-)caching of the gnbd happens and when/how the cache will
> > > be emptied.
> > > As far as i understand it, the caching happens on the gnbd client. Is
> > > this correct?
> > No. On the gnbd client side, the device works just like every other block
> > device. When the device is closed, the cache is automatically flushed.
> 
> so nothing on the gnbd-client side inside gnbd makes any caching. This is 
> good.
> 
> 
> > > If so, when will the cache of the gnbd client be emptied? Will it be
> > > emptied if no one accesses the device? Or will it be cached until the
> > > device is reimported (removed and imported)?
> > > Are there other possibilities to clear the read cache? Are there
> > > possibilities to see how much data is cached (allocated memory of
> > > gnbd_recvd)? Or have i to look for linux page cache?
> > Like I said, the caching option has nothing to do with the client side.
> > GNBD acts just like any other block device.
> > The caching option does only one thing.  I determines whether or not the
> > gnbd server daemon opens the exported block device with the O_DIRECT flag
> > or not.
> 
> Ah, so this caching option is only relevant on the server. So if one gnbd 
> client asks the server, it could serve data from the cache. If another 
> gnbd-client asks the same server, it also could serve from the same cache. So 
> as long as the exported block device will not change, my data would be 
> konsistent.
> And if i access the block device from the gnbd client, the normal linux 
> caching mechanismen will take place.
> 
> 
> > If the device is uncached, then the gnbd server will access the exported
> > device with O_DIRECT, which bypasses the cache. Otherwise, the gnbd server
> > will not use O_DIRECT, and use the page cache. If the exported device was
> > on a SAN, and being used by exported by multiple gnbd servers (which would
> > allow you to use dm-multipath for failover), you would want it uncached,
> > because otherwise changes on other machines might be missed, if you pulled
> > from cache.  Also, if you were say, exporting part of an LVM volume, which
> > you were also using locally, you would want to be uncached. Otherwise would
> > have one cache on top of the exported device, and another cache (for the
> > same device) on top of the LVM volume.
> 
> I think i understand it now. This gnbd caching thing is all about block device 
> caching on the gnbd server, not on the gnbd client.
> 
> If i access one exported block device from multiple gnbd clients with O_DIRECT 
> my data will be konsistent.
> This means, if one gnbd client reads (O_DIRECT) data from a cached gnbd 
> server, this data will be cached on the gnbd server. Then another gnbd client 
> writes data on the same position. If now the first gnbd client reads again 
> the same position, it will get the data written from the second client, 
> because the gnbd server serves both clients from the same cache.
> Are am I correct?

Yes. All the clients use the same cache on the server, so if they open the gnbd device O_DIRECT, they will all see eachother's writes.
 
> Many thanks for your explanations.
> 
> 
> -- 
> greetings
> 
> eMHa



> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster