[dm-devel] [PATCH] EnhanceIO ssd caching software
Darrick J. Wong
darrick.wong at oracle.com
Fri Feb 15 20:31:39 UTC 2013
[Resending with dm-devel, Kent, and Joe on cc. Sorry for the noise.]
On Fri, Feb 15, 2013 at 02:02:38PM +0800, OS Engineering wrote:
> Hi Greg, Jens,
>
> We are submitting EnhanceIO(TM) software driver for an inclusion in linux
> staging tree. Present state of this driver is beta. We have been posting it
> for a few weeks, while it was maintained at github. It is still being
> cleaned-up and is being tested by LKML members. Inclusion in linux staging
> tree will make testing and reviewing easier and help a future integration in
> Linux kernel.
>
> Could you please include it?
Mmm large patches, I'll try to review it... in pieces. Thanks for the work!
> Thanks.
> --
> Amit Kale
>
> From 31f636ffd63ce46c4a65ce622bcb7b18ce05f7c3 Mon Sep 17 00:00:00 2001
> From: sanoj <sunnikrishnan at stec-inc.com>
> Date: Tue, 12 Feb 2013 16:15:19 +0530
> Subject: [PATCH] Enhanceio Driver
>
> This driver is based on EnhanceIO(TM) SSD caching software product
> developed by STEC Inc. EnhanceIO(TM) software was derived from Facebook's open source
> Flashcache project. EnhanceIO(TM) software uses SSDs as cache devices for traditional
> rotating hard disk drives. EnhanceIO(TM) software can work with any block device, be it
> an entire physical disk, an individual disk partition, a RAIDed DAS device,
> a SAN volume, a device mapper volume or a software RAID (md) device.
>
> Signed-off-by:
> Amit Kale <akale at stec-inc.com>
> Sanoj Unnikrishnan <sunnikrishnan at stec-inc.com>
> Darrick J. Wong <darrick.wong at oracle.com>
> Jinpu Wang <jinpuwang at gmail.com>
Each of these email addresses needs to have the "S-o-b:" prefix attached.
Also, you ought to run this patch through scripts/checkpatch.pl, as there are
quite a lot of style errors.
> ---
> Documentation/enhanceio/94-Enhanceio.template | 32 +
> Documentation/enhanceio/Persistence.txt | 10 +
> Documentation/enhanceio/README.txt | 194 ++
> drivers/staging/Kconfig | 2 +
> drivers/staging/Makefile | 1 +
> drivers/staging/enhanceio/Kconfig | 20 +
> drivers/staging/enhanceio/Makefile | 16 +
> drivers/staging/enhanceio/eio.h | 1137 ++++++++
> drivers/staging/enhanceio/eio_conf.c | 2627 ++++++++++++++++++
> drivers/staging/enhanceio/eio_fifo.c | 240 ++
> drivers/staging/enhanceio/eio_ioctl.c | 157 ++
> drivers/staging/enhanceio/eio_ioctl.h | 89 +
> drivers/staging/enhanceio/eio_lru.c | 323 +++
> drivers/staging/enhanceio/eio_main.c | 3599 +++++++++++++++++++++++++
> drivers/staging/enhanceio/eio_mem.c | 235 ++
> drivers/staging/enhanceio/eio_policy.c | 146 +
> drivers/staging/enhanceio/eio_policy.h | 105 +
> drivers/staging/enhanceio/eio_procfs.c | 1939 +++++++++++++
> drivers/staging/enhanceio/eio_setlru.c | 170 ++
> drivers/staging/enhanceio/eio_setlru.h | 49 +
> drivers/staging/enhanceio/eio_subr.c | 451 ++++
> drivers/staging/enhanceio/eio_ttc.c | 1702 ++++++++++++
> drivers/staging/enhanceio/eio_ttc.h | 150 +
> tools/enhanceio/eio_cli | 344 +++
> 24 files changed, 13738 insertions(+), 0 deletions(-)
> create mode 100644 Documentation/enhanceio/94-Enhanceio.template
> create mode 100644 Documentation/enhanceio/Persistence.txt
> create mode 100644 Documentation/enhanceio/README.txt
> create mode 100644 drivers/staging/enhanceio/Kconfig
> create mode 100644 drivers/staging/enhanceio/Makefile
> create mode 100644 drivers/staging/enhanceio/eio.h
> create mode 100644 drivers/staging/enhanceio/eio_conf.c
> create mode 100644 drivers/staging/enhanceio/eio_fifo.c
> create mode 100644 drivers/staging/enhanceio/eio_ioctl.c
> create mode 100644 drivers/staging/enhanceio/eio_ioctl.h
> create mode 100644 drivers/staging/enhanceio/eio_lru.c
> create mode 100644 drivers/staging/enhanceio/eio_main.c
> create mode 100644 drivers/staging/enhanceio/eio_mem.c
> create mode 100644 drivers/staging/enhanceio/eio_policy.c
> create mode 100644 drivers/staging/enhanceio/eio_policy.h
> create mode 100644 drivers/staging/enhanceio/eio_procfs.c
> create mode 100644 drivers/staging/enhanceio/eio_setlru.c
> create mode 100644 drivers/staging/enhanceio/eio_setlru.h
> create mode 100644 drivers/staging/enhanceio/eio_subr.c
> create mode 100644 drivers/staging/enhanceio/eio_ttc.c
> create mode 100644 drivers/staging/enhanceio/eio_ttc.h
> create mode 100644 tools/enhanceio/eio_cli
>
> diff --git a/Documentation/enhanceio/94-Enhanceio.template b/Documentation/enhanceio/94-Enhanceio.template
> new file mode 100644
> index 0000000..ec4a685
> --- /dev/null
> +++ b/Documentation/enhanceio/94-Enhanceio.template
> @@ -0,0 +1,32 @@
> +ACTION!="add|change", GOTO="EIO_EOF"
> +SUBSYSTEM!="block", GOTO="EIO_EOF"
> +
> +<cache_match_expr>, GOTO="EIO_CACHE"
> +
> +<source_match_expr>, GOTO="EIO_SOURCE"
> +
> +# If none of the rules above matched then it isn't an EnhanceIO device so ignore it.
> +GOTO="EIO_EOF"
> +
> +# If we just found the cache device and the source already exists then we can setup
> +LABEL="EIO_CACHE"
> + TEST!="/dev/enhanceio/<cache_name>", PROGRAM="/bin/mkdir -p /dev/enhanceio/<cache_name>"
> + PROGRAM="/bin/sh -c 'echo $kernel > /dev/enhanceio/<cache_name>/.ssd_name'"
> +
> + TEST=="/dev/enhanceio/<cache_name>/.disk_name", GOTO="EIO_SETUP"
> +GOTO="EIO_EOF"
> +
> +# If we just found the source device and the cache already exists then we can setup
> +LABEL="EIO_SOURCE"
> + TEST!="/dev/enhanceio/<cache_name>", PROGRAM="/bin/mkdir -p /dev/enhanceio/<cache_name>"
> + PROGRAM="/bin/sh -c 'echo $kernel > /dev/enhanceio/<cache_name>/.disk_name'"
> +
> + TEST=="/dev/enhanceio/<cache_name>/.ssd_name", GOTO="EIO_SETUP"
If the cache is running in wb mode, perhaps we should make it ro until the SSD
shows up and we run eio_cli? Run blockdev --setro in the EIO_CACHE part, and
blockdev --setrw in the EIO_SOURCE part?
<shrug> not a udev developer, take that with a grain of salt.
> +GOTO="EIO_EOF"
> +
> +LABEL="EIO_SETUP"
> + PROGRAM="/bin/sh -c 'cat /dev/enhanceio/<cache_name>/.ssd_name'", ENV{ssd_name}="%c"
> + PROGRAM="/bin/sh -c 'cat /dev/enhanceio/<cache_name>/.disk_name'", ENV{disk_name}="%c"
> +
> + TEST!="/proc/enhanceio/<cache_name>", RUN+="/sbin/eio_cli enable -d /dev/$env{disk_name} -s /dev/$env{ssd_name} <cache_name>"
> +LABEL="EIO_EOF"
> diff --git a/Documentation/enhanceio/Persistence.txt b/Documentation/enhanceio/Persistence.txt
> new file mode 100644
> index 0000000..8b6e58f
> --- /dev/null
> +++ b/Documentation/enhanceio/Persistence.txt
> @@ -0,0 +1,10 @@
> +How to create persistent cache
> +==============================
> +
> +Use the 94-Enhanceio-template file to create a per cache udev-rule file named /etc/udev/rules.d/94-enhancio-<cache_name>.rules
> +
> +1) Change <cache_match_expr> to ENV{ID_SERIAL}=="<ID SERIAL OF YOUR CACHE DEVICE>", ENV{DEVTYPE}==<DEVICE TYPE OF YOUR CACHE DEVICE>
> +
> +2) Change <source_match_expr> to ENV{ID_SERIAL}=="<ID SERIAL OF YOUR HARD DISK>", ENV{DEVTYPE}==<DEVICE TYPE OF YOUR SOURCE DEVICE>
> +
> +3) Replace all instances of <cache_name> with the name of your cache
I wonder if there's a better way to do this than manually cutting and pasting
all these IDs into a udev rules file. Or, how about a quick script at cache
creation time that spits out files into /etc/udev/rules.d/ ?
> diff --git a/Documentation/enhanceio/README.txt b/Documentation/enhanceio/README.txt
> new file mode 100644
> index 0000000..6391dce
> --- /dev/null
> +++ b/Documentation/enhanceio/README.txt
> @@ -0,0 +1,194 @@
> + STEC EnhanceIO SSD Caching Software
> + 25th December, 2012
> +
> +
> +1. WHAT IS ENHANCEIO?
> +
> + EnhanceIO driver is based on EnhanceIO SSD caching software product
> + developed by STEC Inc. EnhanceIO was derived from Facebook's open source
> + Flashcache project. EnhanceIO uses SSDs as cache devices for
> + traditional rotating hard disk drives (referred to as source volumes
> + throughout this document).
> +
> + EnhanceIO can work with any block device, be it an entire physical
> + disk, an individual disk partition, a RAIDed DAS device, a SAN volume,
> + a device mapper volume or a software RAID (md) device.
> +
> + The source volume to SSD mapping is a set-associative mapping based on
> + the source volume sector number with a default set size
> + (aka associativity) of 512 blocks and a default block size of 4 KB.
> + Partial cache blocks are not used. The default value of 4 KB is chosen
> + because it is the common I/O block size of most storage systems. With
> + these default values, each cache set is 2 MB (512 * 4 KB). Therefore,
> + a 400 GB SSD will have a little less than 200,000 cache sets because a
> + little space is used for storing the meta data on the SSD.
> +
> + EnhanceIO supports three caching modes: read-only, write-through, and
> + write-back and three cache replacement policies: random, FIFO, and LRU.
> +
> + Read-only caching mode causes EnhanceIO to direct write IO requests only
> + to HDD. Read IO requests are issued to HDD and the data read from HDD is
> + stored on SSD. Subsequent Read requests for the same blocks are carried
> + out from SSD, thus reducing their latency by a substantial amount.
> +
> + In Write-through mode - reads are handled similar to Read-only mode.
> + Write-through mode causes EnhanceIO to write application data to both
> + HDD and SSD. Subsequent reads of the same data benefit because they can
> + be served from SSD.
> +
> + Write-back improves write latency by writing application requested data
> + only to SSD. This data, referred to as dirty data, is copied later to
How much later?
> + HDD asynchronously. Reads are handled similar to Read-only and
> + Write-through modes.
> +
> +2. WHAT HAS ENHANCEIO ADDED TO FLASHCACHE?
> +
> +2.1. A new write-back engine
> +
> + The write-back engine in EnhanceiO has been designed from scratch.
> + Several optimizations have been done. IO completion guarantees have
> + been improved. We have defined limits to let a user control the amount
> + of dirty data in a cache. Clean-up of dirty data is stopped by default
> + under a high load; this can be overridden if required. A user can
> + control the extent to which a single cache set can be filled with dirty
> + data. A background thread cleans-up dirty data at regular intervals.
> + Clean-up is also done at regular intevals by identifying cache sets
> + which have been written least recently.
> +
> +2.2. Transparent cache
> +
> + EnhanceIO does not use device mapper. This enables creation and
> + deletion of caches while a source volume is being used. It's possible
> + to either create or delete cache while a partition is mounted.
> +
> + EnhanceIO also supports creation of a cache for a device which contains
> + partitions. With this feature it's possible to create a cache without
> + worrying about having to create several SSD partitions and many
> + separate caches.
> +
> +
> +2.3. Large I/O Support
> +
> + Unlike Flashcache, EnhanceIO does not cause source volume I/O requests
> + to be split into cache block size pieces. For the typical SSD cache
> + block size of 4 KB, this means that a write I/O request size of, say,
> + 64 KB to the source volume is not split into 16 individual requests of
> + 4 KB each. This is a performance improvement over Flashcache. IO
> + codepaths have been substantially modified for this improvement.
> +
> +2.4. Small Memory Footprint
> +
> + Through a special compression algorithm, the meta data RAM usage has
> + been reduced to only 4 bytes for each SSD cache block (versus 16 bytes
> + in Flashcache). Since the most typical SSD cache block size is 4 KB,
> + this means that RAM usage is 0.1% (1/1000) of SSD capacity.
> + For example, for a 400 GB SSD, EnhanceIO will need only 400 MB to keep
> + all meta data in RAM.
> +
> + For an SSD cache block size of 8 KB, RAM usage is 0.05% (1/2000) of SSD
> + capacity.
> +
> + The compression algorithm needs at least 32,768 cache sets
> + (i.e., 16 bits to encode the set number). If the SSD capacity is small
> + and there are not at least 32,768 cache sets, EnhanceIO uses 8 bytes of
> + RAM for each SSD cache block. In this case, RAM usage is 0.2% (2/1000)
> + of SSD capacity for a cache block size of 4K.
> +
> +2.4. Loadable Replacement Policies
> +
> + Since the SSD cache size is typically 10%-20% of the source volume
> + size, the set-associative nature of EnhanceIO necessitates cache
> + block replacement.
> +
> + The main EnhanceIO kernel module that implements the caching engine
> + uses a random (actually, almost like round-robin) replacement policy
> + that does not require any additional RAM and has the least CPU
> + overhead. However, there are two additional kernel modules that
> + implement FIFO and LRU replacement policies. FIFO is the default cache
> + replacement policy because it uses less RAM than LRU. The FIFO and LRU
> + kernel modules are independent of each other and do not have to be
> + loaded if they are not needed.
> +
> + Since the replacement policy modules do not consume much RAM when not
> + used, both modules are typically loaded after the main caching engine
> + is loaded. RAM is used only after a cache has been instantiated to use
> + either the FIFO or the LRU replacement policy.
> +
> + Please note that the RAM used for replacement policies is in addition
> + to the RAM used for meta data (mentioned in Section 2.1). The table
> + below shows how much RAM each cache replacement policy uses:
> +
> + POLICY RAM USAGE
> + ------ ---------
> + Random 0
> + FIFO 4 bytes per cache set
> + LRU 4 bytes per cache set + 4 bytes per cache block
> +
> +2.5. Optimal Alignment of Data Blocks on SSD
> +
> + EnhanceIO writes all meta data and data blocks on 4K-aligned blocks
> + on the SSD. This minimizes write amplification and flash wear.
> + It also improves performance.
> +
> +2.6. Improved device failure handling
> +
> + Failure of an SSD device in read-only and write-through modes is
> + handled gracefully by allowing I/O to continue to/from the
> + source volume. An application may notice a drop in performance but it
> + will not receive any I/O errors.
> +
> + Failure of an SSD device in write-back mode obviously results in the
> + loss of dirty blocks in the cache. To guard against this data loss, two
> + SSD devices can be mirrored via RAID 1.
What happens to writes that happen after the SSD goes down? Are they simply
passed through to the slow disk?
> + EnhanceIO identifies device failures based on error codes. Depending on
> + whether the failure is likely to be intermittent or permanent, it takes
> + the best suited action.
> +
> +2.8. Coding optimizations
> +
> + Several coding optizations have been done to reduce CPU usage. These
> + include removing queues which are not required for write-through and
> + read-only cache modes, splitting of a single large spinlock, and more.
> + Most of the code paths in flashcache have been substantially
> + restructured.
> +
> +3. EnhanceIO usage
> +
> +3.1. Cache creation, deletion and editing properties
> +
> + eio_cli utility is used for creating and deleting caches and editing
> + their properties. Manpage for this utility eio_cli(8) provides more
> + information.
> +
> +3.2. Making a cache configuration persistent
> + It's essential that a cache be resumed before any applications or a
> + filesystem use the source volume during a bootup. If a cache is enabled
> + after a source volume is written to, stale data may be present in the
> + cache. It may cause data corruption. The document Persistent.txt
> + describes how to enable a cache during bootup using udev scripts.
> +
> + In case an SSD does not come up during a bootup, it's ok to allow read
> + and write access to HDD only in the case of a Write-through or a
> + read-only cache. A cache should be created again when SSD becomes
> + available. If a previous cache configuration is resumed, it may cause
> + stale data to be read.
> +
> +3.3. Using a Write-back cache
> + It's absolutely necessary to make a Write-back cache configuration
> + persistent. This is required particularly in the case of an OS crash or
> + a power failure. A Write-back cache may contain dirty blocks which
> + haven't been written to HDD yet. Reading the source volume without
> + enabling the cache will cause incorrect data to be read.
> +
> + In case an SSD does not come up during a bootup, access to HDD should
> + stopped. It should be enabled only after SSD comes-up and a cache is
> + enabled.
> +
> +4. ACKNOWLEDGEMENTS
> +
> + STEC acknowledges Facebook and in particular Mohan Srinivasan
> + for the design, development, and release of Flashcache as an
> + open source project.
> +
> + Flashcache, in turn, is based on DM-Cache by Ming Zhao.
> diff --git a/drivers/staging/Kconfig b/drivers/staging/Kconfig
> index 329bdb4..0e97141 100644
> --- a/drivers/staging/Kconfig
> +++ b/drivers/staging/Kconfig
> @@ -142,4 +142,6 @@ source "drivers/staging/sb105x/Kconfig"
>
> source "drivers/staging/fwserial/Kconfig"
>
> +source "drivers/staging/enhanceio/Kconfig"
> +
> endif # STAGING
> diff --git a/drivers/staging/Makefile b/drivers/staging/Makefile
> index c7ec486..81656de 100644
> --- a/drivers/staging/Makefile
> +++ b/drivers/staging/Makefile
> @@ -63,3 +63,4 @@ obj-$(CONFIG_DRM_IMX) += imx-drm/
> obj-$(CONFIG_DGRP) += dgrp/
> obj-$(CONFIG_SB105X) += sb105x/
> obj-$(CONFIG_FIREWIRE_SERIAL) += fwserial/
> +obj-$(CONFIG_ENHANCEIO) += enhanceio/
> diff --git a/drivers/staging/enhanceio/Kconfig b/drivers/staging/enhanceio/Kconfig
> new file mode 100644
> index 0000000..ca740e1
> --- /dev/null
> +++ b/drivers/staging/enhanceio/Kconfig
> @@ -0,0 +1,20 @@
> +#
> +# EnhanceIO caching solution by STEC INC.
> +#
> +
> +config ENHANCEIO
> + tristate "Enable EnhanceIO"
> + default n
> + ---help---
> + Based on Facebook's open source Flashcache project developed by
> + Mohan Srinivasan and hosted at "http://github.com", EnhanceIO is
Probably not necessary to mention github once this ends up in the kernel.
> + a collection of (currently three) loadable kernel modules for
> + using SSDs as cache devices for traditional rotating hard disk
> +
> + The caching engine is a loadable kernel module ("enhanceio.ko")
> + implemented as a device mapper target. The cache replacement
> + policies are implemented as loadable kernel modules
> + ("enhanceio_fifo.ko", "enhanceio_lru.ko") that register with
> + the caching engine module.
> +
> + If unsure, say N.
> diff --git a/drivers/staging/enhanceio/Makefile b/drivers/staging/enhanceio/Makefile
> new file mode 100644
> index 0000000..926aa71
> --- /dev/null
> +++ b/drivers/staging/enhanceio/Makefile
> @@ -0,0 +1,16 @@
> +#
> +# Makefile for EnhanceIO block device caching.
> +#
> +obj-$(CONFIG_ENHANCEIO) += enhanceio.o enhanceio_lru.o enhanceio_fifo.o
> +enhanceio-y += \
> + eio_conf.o \
> + eio_ioctl.o \
> + eio_main.o \
> + eio_mem.o \
> + eio_policy.o \
> + eio_procfs.o \
> + eio_setlru.o \
> + eio_subr.o \
> + eio_ttc.o
> +enhanceio_fifo-y += eio_fifo.o
> +enhanceio_lru-y += eio_lru.o
> diff --git a/drivers/staging/enhanceio/eio.h b/drivers/staging/enhanceio/eio.h
> new file mode 100644
> index 0000000..82bab6e
> --- /dev/null
> +++ b/drivers/staging/enhanceio/eio.h
> @@ -0,0 +1,1137 @@
> +/*
> + * eio.h
> + *
> + * Copyright (C) 2012 STEC, Inc. All rights not specifically granted
> + * under a license included herein are reserved
> + * Saied Kazemi <skazemi at stec-inc.com>
> + * Added EnhanceIO-specific code.
> + * Siddharth Choudhuri <schoudhuri at stec-inc.com>
> + * Common data structures and definitions between Windows and Linux.
> + * Amit Kale <akale at stec-inc.com>
> + * Restructured much of the io code to split bio within map function instead
> + * of letting dm do it.
> + * Amit Kale <akale at stec-inc.com>
> + * Harish Pujari <hpujari at stec-inc.com>
> + * Designed and implemented the writeback caching mode
> + * Copyright 2010 Facebook, Inc.
> + * Author: Mohan Srinivasan (mohan at facebook.com)
> + *
> + * Based on DM-Cache:
> + * Copyright (C) International Business Machines Corp., 2006
> + * Author: Ming Zhao (mingzhao at ufl.edu)
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; under version 2 of the License.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * along with this program. If not, see <http://www.gnu.org/licenses/>.
> + */
> +
> +#include <asm/atomic.h>
> +#include <linux/module.h>
> +#include <linux/init.h>
> +#include <linux/list.h>
> +#include <linux/blkdev.h>
> +#include <linux/bio.h>
> +#include <linux/slab.h>
> +#include <linux/hash.h>
> +#include <linux/spinlock.h>
> +#include <linux/workqueue.h>
> +#include <linux/pagemap.h>
> +#include <linux/random.h>
> +#include <linux/hardirq.h>
> +#include <linux/sysctl.h>
> +#include <linux/version.h>
> +#include <linux/reboot.h>
> +#include <linux/delay.h>
> +#include <linux/proc_fs.h>
> +#include <linux/seq_file.h>
> +#include <linux/device-mapper.h>
> +#include <linux/dm-kcopyd.h>
> +#include <linux/sort.h> /* required for eio_subr.c */
> +#include <linux/kthread.h>
> +#include <linux/jiffies.h>
> +#include <linux/vmalloc.h> /* for sysinfo (mem) variables */
> +#include <linux/mm.h>
> +#include <scsi/scsi_device.h> /* required for SSD failure handling */
> +/* resolve conflict with scsi/scsi_device.h */
> +#ifdef QUEUED
> +#undef QUEUED
> +#endif
> +
> +#if defined(__KERNEL__) && !defined(CONFIG_PROC_FS)
> +#error "EnhanceIO requires CONFIG_PROC_FS"
> +#endif /* __KERNEL__ && !CONFIG_PROC_FS */
This dependency should be stated in the Kconfig file. 'depends PROC_FS' or
something like that.
> +#ifndef EIO_INC_H
> +#define EIO_INC_H
> +
> +#define EIO_DBN_SET(dmc, index, dbn) ssdcache_dbn_set(dmc, index, dbn)
> +#define EIO_DBN_GET(dmc, index) ssdcache_dbn_get(dmc, index)
> +#define EIO_CACHE_STATE_SET(dmc, index, state) ssdcache_cache_state_set(dmc, index, state)
> +#define EIO_CACHE_STATE_GET(dmc, index) ssdcache_cache_state_get(dmc, index)
> +#define EIO_CACHE_STATE_OFF(dmc, index, bitmask) ssdcache_cache_state_off(dmc, index, bitmask)
> +#define EIO_CACHE_STATE_ON(dmc, index, bitmask) ssdcache_cache_state_on(dmc, index, bitmask)
> +
> +/* Bit offsets for wait_on_bit_lock() */
> +#define EIO_UPDATE_LIST 0
> +#define EIO_HANDLE_REBOOT 1
> +
> +struct eio_control_s {
> + volatile unsigned long synch_flags;
Are you sure that this volatile does what you think it does?
http://www.kernel.org/doc/Documentation/volatile-considered-harmful.txt
afaict all the uses of synch_flags seem to use atomic operations already...
> +};
> +
> +int eio_wait_schedule(void *unused);
> +
> +struct eio_event {
> + struct task_struct *process; /* handle of the sleeping process */
> +};
> +
> +typedef long int index_t;
> +
> +/*
> + * This file has three sections as follows:
> + *
> + * Section 1: User space only
> + * Section 2: User space and kernel
> + * Section 3: Kernel only
> + *
> + * Each section may contain its own subsections.
> + */
> +
> +/*
> + * Begin Section 1: User space only.
> + */
Empty?
> +/*
> + * End Section 1: User space only.
> + */
> +
> +/*
> + * Begin Section 2: User space and kernel.
> + */
> +
> +/* States of a cache block */
> +#define INVALID 0x0001
> +#define VALID 0x0002 /* Valid */
> +#define DISKREADINPROG 0x0004 /* Read from disk in progress */
> +#define DISKWRITEINPROG 0x0008 /* Write to disk in progress */
> +#define CACHEREADINPROG 0x0010 /* Read from cache in progress */
> +#define CACHEWRITEINPROG 0x0020 /* Write to cache in progress */
> +#define DIRTY 0x0040 /* Dirty, needs writeback to disk */
> +#define QUEUED 0x0080 /* Other requests are queued for this block */
> +
> +#define BLOCK_IO_INPROG (DISKREADINPROG | DISKWRITEINPROG | \
> + CACHEREADINPROG | CACHEWRITEINPROG)
> +#define DIRTY_INPROG (VALID | DIRTY | CACHEWRITEINPROG) /* block being dirtied */
> +#define CLEAN_INPROG (VALID | DIRTY | DISKWRITEINPROG) /* ongoing clean */
> +#define ALREADY_DIRTY (VALID | DIRTY) /* block which is dirty to begin with for an I/O */
These shouldn't go past 80 columns.
> +/*
> + * This is a special state used only in the following scenario as
> + * part of device (SSD) failure handling:
> + *
> + * ------| dev fail |------| dev resume |------------
> + * ...-<--- Tf --><- Td -><---- Tr ---><-- Tn ---...
> + * |---- Normal ----|-- Degraded -------|-- Normal ---|
> + *
> + * Tf: Time during device failure.
> + * Td: Time after failure when the cache is in degraded mode.
> + * Tr: Time when the SSD comes back online.
> + *
> + * When a failed SSD is added back again, it should be treated
> + * as a cold SSD.
> + *
> + * If Td is very small, then there can be IOs that were initiated
> + * before or during Tf, and did not finish until the end of Tr. From
> + * the IO's viewpoint, the SSD was there when the IO was initiated
> + * and it was there when the IO was finished. These IOs need special
> + * handling as described below.
> + *
> + * To add the SSD as a cold cache device, we initialize all blocks
> + * to INVALID, execept for the ones that had IOs in progress before
> + * or during Tf. We mark such blocks as both VALID and INVALID.
> + * These blocks will be marked INVALID when finished.
> + */
> +#define NO_SSD_IO_INPROG (VALID | INVALID)
> +
> +/*
> + * On Flash (cache metadata) Structures
> + */
> +#define CACHE_MD_STATE_DIRTY 0x55daddee
> +#define CACHE_MD_STATE_CLEAN 0xacceded1
> +#define CACHE_MD_STATE_FASTCLEAN 0xcafebabf
> +#define CACHE_MD_STATE_UNSTABLE 0xdeaddeee
> +
> +/* Do we have a read cache or a read-write cache */
> +#define CACHE_MODE_WB 1
> +#define CACHE_MODE_RO 2
> +#define CACHE_MODE_WT 3
> +#define CACHE_MODE_FIRST CACHE_MODE_WB
> +#define CACHE_MODE_LAST CACHE_MODE_WT
> +#define CACHE_MODE_DEFAULT CACHE_MODE_WT
> +
> +#define DEV_PATHLEN 128
> +#define EIO_SUPERBLOCK_SIZE 4096
> +
> +#define EIO_CLEAN_ABORT 0x00000000
> +#define EIO_CLEAN_START 0x00000001
> +#define EIO_CLEAN_KEEP 0x00000002
> +
> +/* EIO magic number */
> +#define EIO_MAGIC 0xE10CAC6E
> +#define EIO_BAD_MAGIC 0xBADCAC6E
> +
> +/* EIO version */
> +#define EIO_SB_VERSION 3 /* kernel superblock version */
> +#define EIO_SB_MAGIC_VERSION 3 /* version in which magic number was introduced */
> +
> +typedef union eio_superblock {
> + struct superblock_fields {
> + sector_t size; /* Cache size */
sector_t is 32 bits on !LBDAF 32-bit systems and 64 bits otherwise. This
structure seems reflect an on-disk format, which means that I can badly screw
things up if I move a cache disk between machines with differently configured
kernels. Plus, if we ever change the definition of sector_t then this
structure will be broken.
This field should be declared with an explicit size, i.e. __le64.
> + u_int32_t block_size; /* Cache block size */
Worse yet, these fields should use endianness notations (e.g. __le32) and when
you write out the superblock, you need to wrap the assignments with a
cpu_to_leXX() call. Otherwise, enhanceio caches created on ppc64 won't load on
a x64 box (and vice versa) because all the bytes are swapped.
These two grumblings also apply to any other on-disk-format structs in this
patch.
> + u_int32_t assoc; /* Cache associativity */
> + u_int32_t cache_sb_state; /* Clean shutdown ? */
> + char cache_devname[DEV_PATHLEN];
> + sector_t cache_devsize;
> + char disk_devname[DEV_PATHLEN];
> + sector_t disk_devsize;
> + u_int32_t cache_version;
> + char cache_name[DEV_PATHLEN];
> + u_int32_t mode;
> + u_int32_t repl_policy;
> + u_int32_t cache_flags;
> + /*
> + * Version 1.1 superblock ends here.
> + * Don't modify any of the above fields.
> + */
> + u_int32_t magic; /* Has to be the 1st field afer 1.1 superblock */
> + u_int32_t cold_boot; /* cache to be started as cold after boot */
> + char ssd_uuid[DEV_PATHLEN];
> + sector_t cache_md_start_sect; /* cache metadata start (8K aligned) */
> + sector_t cache_data_start_sect; /* cache data start (8K aligned) */
> + u_int32_t dirty_high_threshold;
> + u_int32_t dirty_low_threshold;
> + u_int32_t dirty_set_high_threshold;
> + u_int32_t dirty_set_low_threshold;
> + u_int32_t time_based_clean_interval;
> + u_int32_t autoclean_threshold;
> + } sbf;
> + u_int8_t padding[EIO_SUPERBLOCK_SIZE];
> +} eio_superblock_t;
Why does this in-memory data structure need to be 4096 bytes long? 'padding'
doesn't seem to be used anywhere.
> +
> +/*
> + * For EnhanceIO, we move the superblock from sector 0 to 128
> + * and give it a full 4K. Also, in addition to the single
> + * "red-zone" buffer that separates metadata sectors from the
> + * data sectors, we allocate extra sectors so that we can
> + * align the data sectors on a 4K boundary.
> + *
> + * 64K 4K variable variable 8K variable variable
> + * +--------+--+--------+---------+---+--------+---------+
> + * | unused |SB| align1 |metadata | Z | align2 | data... |
> + * +--------+--+--------+---------+---+--------+---------+
> + * <------------- dmc->md_sectors ------------>
> + */
> +#define EIO_UNUSED_SECTORS 128
> +#define EIO_SUPERBLOCK_SECTORS 8
> +#define EIO_REDZONE_SECTORS 16
> +#define EIO_START 0
> +
> +#define EIO_ALIGN1_SECTORS(index) ((index % 16) ? (24 - (index % 16)) : 8)
> +#define EIO_ALIGN2_SECTORS(index) ((index % 16) ? (16 - (index % 16)) : 0)
> +#define EIO_SUPERBLOCK_START (EIO_START + EIO_UNUSED_SECTORS)
> +#define EIO_METADATA_START(hd_start_sect) (EIO_SUPERBLOCK_START + \
> + EIO_SUPERBLOCK_SECTORS + \
> + EIO_ALIGN1_SECTORS(hd_start_sect))
> +
> +#define EIO_EXTRA_SECTORS(start_sect, md_sects) (EIO_METADATA_START(start_sect) + \
> + EIO_REDZONE_SECTORS + \
> + EIO_ALIGN2_SECTORS(md_sects))
> +
> +/*
> + * We do metadata updates only when a block trasitions from DIRTY -> CLEAN
> + * or from CLEAN -> DIRTY. Consequently, on an unclean shutdown, we only
> + * pick up blocks that are marked (DIRTY | CLEAN), we clean these and stick
> + * them in the cache.
> + * On a clean shutdown, we will sync the state for every block, and we will
> + * load every block back into cache on a restart.
> + */
> +struct flash_cacheblock {
> + sector_t dbn; /* Sector number of the cached block */
> +#ifdef DO_CHECKSUM
> + u_int64_t checksum;
> +#endif /* DO_CHECKSUM */
> + u_int32_t cache_state;
> +};
> +
> +/* blksize in terms of no. of sectors */
> +#define BLKSIZE_2K 4
> +#define BLKSIZE_4K 8
> +#define BLKSIZE_8K 16
> +
> +/*
> + * Give me number of pages to allocated for the
> + * iosize x specified in terms of bytes.
> + */
> +#define IO_PAGE_COUNT(x) (((x) + (PAGE_SIZE - 1)) / PAGE_SIZE)
> +
> +/*
> + * Macro that calculates number of biovecs to be
> + * allocated depending on the iosize and cache
> + * block size.
> + */
> +#define IO_BVEC_COUNT(x, blksize) ({ \
> + int count = IO_PAGE_COUNT(x); \
> + switch ((blksize)) { \
> + case BLKSIZE_2K: \
> + count = count * 2; \
> + break; \
> + case BLKSIZE_4K: \
> + case BLKSIZE_8K: \
> + break; \
> + } \
> + count; \
> + })
> +
> +#define MD_MAX_NR_PAGES 16
> +#define MD_BLOCKS_PER_PAGE ((PAGE_SIZE) / sizeof(struct flash_cacheblock))
> +#define INDEX_TO_MD_PAGE(INDEX) ((INDEX) / MD_BLOCKS_PER_PAGE)
> +#define INDEX_TO_MD_PAGE_OFFSET(INDEX) ((INDEX) % MD_BLOCKS_PER_PAGE)
> +
> +#define MD_BLOCKS_PER_SECTOR (512 / (sizeof(struct flash_cacheblock)))
> +#define INDEX_TO_MD_SECTOR(INDEX) ((INDEX) / MD_BLOCKS_PER_SECTOR)
> +#define INDEX_TO_MD_SECTOR_OFFSET(INDEX) ((INDEX) % MD_BLOCKS_PER_SECTOR)
> +#define MD_BLOCKS_PER_CBLOCK(dmc) (MD_BLOCKS_PER_SECTOR * (dmc)->block_size)
> +
> +#define METADATA_IO_BLOCKSIZE (256 * 1024)
> +#define METADATA_IO_BLOCKSIZE_SECT (METADATA_IO_BLOCKSIZE / 512)
> +#define SECTORS_PER_PAGE ((PAGE_SIZE) / 512)
> +
> +/*
> + * Cache persistence.
> + */
> +#define CACHE_RELOAD 1
> +#define CACHE_CREATE 2
> +#define CACHE_FORCECREATE 3
> +
> +/*
> + * Cache replacement policy.
> + */
> +#define CACHE_REPL_FIFO 1
> +#define CACHE_REPL_LRU 2
> +#define CACHE_REPL_RANDOM 3
> +#define CACHE_REPL_FIRST CACHE_REPL_FIFO
> +#define CACHE_REPL_LAST CACHE_REPL_RANDOM
> +#define CACHE_REPL_DEFAULT CACHE_REPL_FIFO
> +
> +/*
> + * Default cache parameters.
> + */
> +#define DEFAULT_CACHE_ASSOC 512
> +#define DEFAULT_CACHE_BLKSIZE 8 /* 4 KB */
> +
> +/*
> + * Valid commands that can be written to "control".
> + * NOTE: Update CACHE_CONTROL_FLAG_MAX value whenever a new control flag is added
> + */
> +#define CACHE_CONTROL_FLAG_MAX 7
> +#define CACHE_VERBOSE_OFF 0
> +#define CACHE_VERBOSE_ON 1
> +#define CACHE_WRITEBACK_ON 2 /* register write back variables */
> +#define CACHE_WRITEBACK_OFF 3
> +#define CACHE_INVALIDATE_ON 4 /* register invalidate variables */
> +#define CACHE_INVALIDATE_OFF 5
> +#define CACHE_FAST_REMOVE_ON 6 /* do not write MD when destroying cache */
> +#define CACHE_FAST_REMOVE_OFF 7
> +
> +/*
> + * Bit definitions in "cache_flags". These are exported in Linux as
> + * hex in the "flags" output line of /proc/enhanceio/<cache_name>/config.
> + */
> +
> +#define CACHE_FLAGS_VERBOSE (1 << 0)
> +#define CACHE_FLAGS_INVALIDATE (1 << 1)
> +#define CACHE_FLAGS_FAST_REMOVE (1 << 2)
> +#define CACHE_FLAGS_DEGRADED (1 << 3)
> +#define CACHE_FLAGS_SSD_ADD_INPROG (1 << 4)
> +#define CACHE_FLAGS_MD8 (1 << 5) /* using 8-byte metadata (instead of 4-byte md) */
> +#define CACHE_FLAGS_FAILED (1 << 6)
> +#define CACHE_FLAGS_STALE (1 << 7)
> +#define CACHE_FLAGS_SHUTDOWN_INPROG (1 << 8)
> +#define CACHE_FLAGS_MOD_INPROG (1 << 9) /* cache modification such as edit/delete in progress */
> +#define CACHE_FLAGS_DELETED (1 << 10)
> +#define CACHE_FLAGS_INCORE_ONLY (CACHE_FLAGS_DEGRADED | \
> + CACHE_FLAGS_SSD_ADD_INPROG | \
> + CACHE_FLAGS_FAILED | \
> + CACHE_FLAGS_SHUTDOWN_INPROG | \
> + CACHE_FLAGS_MOD_INPROG | \
> + CACHE_FLAGS_STALE | \
> + CACHE_FLAGS_DELETED) /* need a proper definition */
> +
> +/* flags that govern cold/warm enable after reboot */
> +#define BOOT_FLAG_COLD_ENABLE (1 << 0) /* enable the cache as cold */
> +#define BOOT_FLAG_FORCE_WARM (1 << 1) /* override the cold enable flag */
> +
> +typedef enum dev_notifier {
> + NOTIFY_INITIALIZER,
> + NOTIFY_SSD_ADD,
> + NOTIFY_SSD_REMOVED,
> + NOTIFY_SRC_REMOVED
> +} dev_notifier_t;
> +
> +/*
> + * End Section 2: User space and kernel.
> + */
> +
> +/*
> + * Begin Section 3: Kernel only.
> + */
> +#if defined(__KERNEL__)
> +
> +/*
> + * Subsection 3.1: Definitions.
> + */
> +
> +#define EIO_SB_VERSION 3 /* kernel superblock version */
> +
> +/* kcached/pending job states */
> +#define READCACHE 1
> +#define WRITECACHE 2
> +#define READDISK 3
> +#define WRITEDISK 4
> +#define READFILL 5 /* Read Cache Miss Fill */
> +#define INVALIDATE 6
> +
> +/* Cache persistence */
> +#define CACHE_RELOAD 1
> +#define CACHE_CREATE 2
> +#define CACHE_FORCECREATE 3
> +
> +/* Sysctl defined */
> +#define MAX_CLEAN_IOS_SET 2
> +#define MAX_CLEAN_IOS_TOTAL 4
> +
> +/*
> + * Harish: TBD
> + * Rethink on max, min, default values
> + */
> +#define DIRTY_HIGH_THRESH_DEF 30
> +#define DIRTY_LOW_THRESH_DEF 10
> +#define DIRTY_SET_HIGH_THRESH_DEF 100
> +#define DIRTY_SET_LOW_THRESH_DEF 30
What are the units of these values? I suspect that they're used to decide when
to start (and stop) flushing dirty blocks out of a wb cache, but please write
down in Documentation/enhanceio/README.txt or somewhere what the sysctl values
do, and in what units they are expressed.
> +
> +#define CLEAN_FACTOR(sectors) ((sectors) >> 25) /* in 16 GB multiples */
> +#define TIME_BASED_CLEAN_INTERVAL_DEF(dmc) (uint32_t)(CLEAN_FACTOR((dmc)->cache_size) ? \
> + CLEAN_FACTOR((dmc)->cache_size) : 1)
> +#define TIME_BASED_CLEAN_INTERVAL_MAX 720 /* in minutes */
> +
> +#define AUTOCLEAN_THRESH_DEF 128 /* Number of I/Os which puts a hold on time based cleaning */
> +#define AUTOCLEAN_THRESH_MAX 1024 /* Number of I/Os which puts a hold on time based cleaning */
> +
> +/* Inject a 5s delay between cleaning blocks and metadata */
> +#define CLEAN_REMOVE_DELAY 5000
> +
> +/*
> + * Subsection 2: Data structures.
> + */
> +
> +/*
> + * Block checksums :
> + * Block checksums seem a good idea (especially for debugging, I found a couple
> + * of bugs with this), but in practice there are a number of issues with this
> + * in production.
> + * 1) If a flash write fails, there is no guarantee that the failure was atomic.
> + * Some sectors may have been written to flash. If so, the checksum we have
> + * is wrong. We could re-read the flash block and recompute the checksum, but
> + * the read could fail too.
> + * 2) On a node crash, we could have crashed between the flash data write and the
> + * flash metadata update (which updates the new checksum to flash metadata). When
> + * we reboot, the checksum we read from metadata is wrong. This is worked around
> + * by having the cache load recompute checksums after an unclean shutdown.
> + * 3) Checksums require 4 or 8 more bytes per block in terms of metadata overhead.
> + * Especially because the metadata is wired into memory.
> + * 4) Checksums force us to do a flash metadata IO on a block re-dirty. If we
> + * didn't maintain checksums, we could avoid the metadata IO on a re-dirty.
> + * Therefore in production we disable block checksums.
> + *
> + * Use the Makefile to enable/disable DO_CHECKSUM
OH? I don't see any code that actually touches checksums (or enables
DO_CHECKSUM).
> + */
> +typedef void (*eio_notify_fn)(int error, void *context);
> +
> +/*
> + * 4-byte metadata support.
> + */
> +
> +#define EIO_MAX_SECTOR (((u_int64_t)1) << 40)
> +
> +struct md4 {
> + u_int16_t bytes1_2;
> + u_int8_t byte3;
> + u_int8_t cache_state;
> +};
> +
> +struct cacheblock {
> + union {
> + u_int32_t u_i_md4;
> + struct md4 u_s_md4;
> + } md4_u;
> +#ifdef DO_CHECKSUM
> + u_int64_t checksum;
> +#endif /* DO_CHECKSUM */
> +};
> +
> +#define md4_md md4_u.u_i_md4
> +#define md4_cache_state md4_u.u_s_md4.cache_state
> +#define EIO_MD4_DBN_BITS (32 - 8) /* 8 bits for state */
> +#define EIO_MD4_DBN_MASK ((1 << EIO_MD4_DBN_BITS) - 1)
> +#define EIO_MD4_INVALID (INVALID << EIO_MD4_DBN_BITS)
> +#define EIO_MD4_CACHE_STATE(dmc, index) (dmc->cache[index].md4_cache_state)
> +
> +/*
> + * 8-byte metadata support.
> + */
> +
> +struct md8 {
> + u_int32_t bytes1_4;
> + u_int16_t bytes5_6;
> + u_int8_t byte7;
> + u_int8_t cache_state;
> +};
> +
> +struct cacheblock_md8 {
> + union {
> + u_int64_t u_i_md8;
> + struct md8 u_s_md8;
> + } md8_u;
> +#ifdef DO_CHECKSUM
> + u_int64_t checksum;
> +#endif /* DO_CHECKSUM */
> +};
> +
> +#define md8_md md8_u.u_i_md8
> +#define md8_cache_state md8_u.u_s_md8.cache_state
> +#define EIO_MD8_DBN_BITS (64 - 8) /* 8 bits for state */
> +#define EIO_MD8_DBN_MASK ((((u_int64_t)1) << EIO_MD8_DBN_BITS) - 1)
> +#define EIO_MD8_INVALID (((u_int64_t)INVALID) << EIO_MD8_DBN_BITS)
> +#define EIO_MD8_CACHE_STATE(dmc, index) ((dmc)->cache_md8[index].md8_cache_state)
> +#define EIO_MD8(dmc) CACHE_MD8_IS_SET(dmc)
> +
> +/* Structure used for metadata update on-disk and in-core for writeback cache */
> +struct mdupdate_request {
> + struct list_head list; /* to build mdrequest chain */
> + struct work_struct work; /* work structure */
> + struct cache_c *dmc; /* cache pointer */
> + index_t set; /* set index */
> + unsigned md_size; /* metadata size */
> + unsigned mdbvec_count; /* count of bvecs allocated. */
> + struct bio_vec *mdblk_bvecs; /* bvecs for updating md_blocks */
> + atomic_t holdcount; /* I/O hold count */
> + struct eio_bio *pending_mdlist; /* ebios pending for md update */
> + struct eio_bio *inprog_mdlist; /* ebios processed for md update */
> + int error; /* error during md update */
> + struct mdupdate_request *next; /* next mdreq in the mdreq list .Harish: TBD. Deprecate */
> +};
> +
> +#define SETFLAG_CLEAN_INPROG 0x00000001 /* clean in progress on a set */
> +#define SETFLAG_CLEAN_WHOLE 0x00000002 /* clean the set fully */
> +
> +/* Structure used for doing operations and storing cache set level info */
> +struct cache_set {
> + struct list_head list;
> + u_int32_t nr_dirty; /* number of dirty blocks */
> + spinlock_t cs_lock; /* spin lock to protect struct fields */
> + struct rw_semaphore rw_lock; /* reader-writer lock used for clean */
> + unsigned int flags; /* misc cache set specific flags */
> + struct mdupdate_request *mdreq; /* metadata update request pointer */
> +};
> +
> +struct eio_errors {
> + int disk_read_errors;
> + int disk_write_errors;
> + int ssd_read_errors;
> + int ssd_write_errors;
> + int memory_alloc_errors;
> + int no_cache_dev;
> + int no_source_dev;
> +};
> +
> +/*
> + * Stats. Note that everything should be "atomic64_t" as
> + * code relies on it.
> + */
> +#define SECTOR_STATS(statval, io_size) \
> + atomic64_add(to_sector(io_size), &statval);
> +
> +struct eio_stats {
> + atomic64_t reads; /* Number of reads */
> + atomic64_t writes; /* Number of writes */
> + atomic64_t read_hits; /* Number of cache hits */
> + atomic64_t write_hits; /* Number of write hits (includes dirty write hits) */
> + atomic64_t dirty_write_hits; /* Number of "dirty" write hits */
> + atomic64_t cached_blocks; /* Number of cached blocks */
> + atomic64_t rd_replace; /* Number of read cache replacements. Harish: TBD modify def doc */
> + atomic64_t wr_replace; /* Number of write cache replacements. Harish: TBD modify def doc */
> + atomic64_t noroom; /* No room in set */
> + atomic64_t cleanings; /* blocks cleaned Harish: TBD modify def doc */
> + atomic64_t md_write_dirty; /* Metadata sector writes dirtying block */
> + atomic64_t md_write_clean; /* Metadata sector writes cleaning block */
> + atomic64_t md_ssd_writes; /* How many md ssd writes did we do ? */
> + atomic64_t uncached_reads;
> + atomic64_t uncached_writes;
> + atomic64_t uncached_map_size;
> + atomic64_t uncached_map_uncacheable;
> + atomic64_t disk_reads;
> + atomic64_t disk_writes;
> + atomic64_t ssd_reads;
> + atomic64_t ssd_writes;
> + atomic64_t ssd_readfills;
> + atomic64_t ssd_readfill_unplugs;
> + atomic64_t readdisk;
> + atomic64_t writedisk;
> + atomic64_t readcache;
> + atomic64_t readfill;
> + atomic64_t writecache;
> + atomic64_t wrtime_ms; /* total write time in ms */
> + atomic64_t rdtime_ms; /* total read time in ms */
> + atomic64_t readcount; /* total reads received so far */
> + atomic64_t writecount; /* total writes received so far */
> +};
> +
> +#define PENDING_JOB_HASH_SIZE 32
> +#define PENDING_JOB_HASH(index) ((index) % PENDING_JOB_HASH_SIZE)
> +#define SIZE_HIST (128 + 1)
> +#define EIO_COPY_PAGES 1024 /* Number of pages for I/O */
> +#define MIN_JOBS 1024
> +#define MIN_EIO_IO 4096
> +#define MIN_DMC_BIO_PAIR 8192
> +
> +/* Structure representing a sequence of sets(first to last set index) */
> +struct set_seq {
> + index_t first_set;
> + index_t last_set;
> + struct set_seq *next;
> +};
> +
> +/* EIO system control variables(tunables) */
> +/*
> + * vloatile are used here since the cost a strong synchonisation
"synchronization"
> + * is not worth the benefits.
> + */
> +struct eio_sysctl {
> + volatile uint32_t error_inject;
> + volatile int32_t fast_remove;
> + volatile int32_t zerostats;
> + volatile int32_t do_clean;
> + volatile uint32_t dirty_high_threshold;
> + volatile uint32_t dirty_low_threshold;
> + volatile uint32_t dirty_set_high_threshold;
> + volatile uint32_t dirty_set_low_threshold;
> + volatile uint32_t time_based_clean_interval; /* time after which dirty sets should clean */
> + volatile int32_t autoclean_threshold;
> + volatile int32_t mem_limit_pct;
> + volatile int32_t control;
> + volatile u_int64_t invalidate;
> +};
> +
> +/* forward declaration */
> +struct lru_ls;
> +
> +/* Replacement for 'struct dm_dev' */
> +struct eio_bdev {
> + struct block_device *bdev;
> + fmode_t mode;
> + char name[16];
> +};
> +
> +/* Replacement for 'struct dm_io_region */
> +struct eio_io_region {
> + struct block_device *bdev;
> + sector_t sector;
> + sector_t count; /* If zero the region is ignored */
> +};
> +
> +/*
> + * Cache context
> + */
> +struct cache_c {
> + struct list_head cachelist;
> + make_request_fn *origmfn;
> + char dev_info; /* partition or whole device */
> +
> + sector_t dev_start_sect;
> + sector_t dev_end_sect;
> + int cache_rdonly; /* protected by ttc_write lock */
> + struct eio_bdev *disk_dev; /* Source device */
> + struct eio_bdev *cache_dev; /* Cache device */
> + struct cacheblock *cache; /* Hash table for cache blocks */
> + struct cache_set *cache_sets;
> + struct cache_c *next_cache;
> + struct kcached_job *readfill_queue;
> + struct work_struct readfill_wq;
> +
> + struct list_head cleanq; /* queue of sets to awaiting clean */
> + struct eio_event clean_event; /* event to wait for, when cleanq is empty */
> + spinlock_t clean_sl; /* spinlock to protect cleanq etc */
> + void *clean_thread; /* OS specific thread object to handle cleanq */
> + int clean_thread_running; /* to indicate that clean thread is running */
> + atomic64_t clean_pendings; /* Number of sets pending to be cleaned */
> + struct bio_vec *clean_dbvecs; /* Data bvecs for clean set */
> + struct page **clean_mdpages; /* Metadata pages for clean set */
> + int dbvec_count;
> + int mdpage_count;
> + int clean_excess_dirty; /* Clean in progress to bring cache dirty blocks in limits */
> + atomic_t clean_index; /* set being cleaned, in case of force clean */
> +
> + u_int64_t md_start_sect; /* Sector no. at which Metadata starts */
> + u_int64_t md_sectors; /* Numbers of metadata sectors, including header */
> + u_int64_t disk_size; /* Source size */
> + u_int64_t size; /* Cache size */
> + u_int32_t assoc; /* Cache associativity */
> + u_int32_t block_size; /* Cache block size */
> + u_int32_t block_shift; /* Cache block size in bits */
> + u_int32_t block_mask; /* Cache block mask */
> + u_int32_t consecutive_shift; /* Consecutive blocks size in bits */
> + u_int32_t persistence; /* Create | Force create | Reload */
> + u_int32_t mode; /* CACHE_MODE_{WB, RO, WT} */
> + u_int32_t cold_boot; /* Cache should be started as cold after boot */
> + u_int32_t bio_nr_pages; /* number of hardware sectors supported by SSD in terms of PAGE_SIZE */
> +
> + spinlock_t cache_spin_lock;
> + long unsigned int cache_spin_lock_flags; /* See comments above spin_lock_irqsave_FLAGS */
> + atomic_t nr_jobs; /* Number of I/O jobs */
> +
> + volatile u_int32_t cache_flags;
> + u_int32_t sb_state; /* Superblock state */
> + u_int32_t sb_version; /* Superblock version */
> +
> + int readfill_in_prog;
> + struct eio_stats eio_stats; /* Run time stats */
> + struct eio_errors eio_errors; /* Error stats */
> + int max_clean_ios_set; /* Max cleaning IOs per set */
> + int max_clean_ios_total; /* Total max cleaning IOs */
> + int clean_inprog;
> + atomic64_t nr_dirty;
> + atomic64_t nr_ios;
> + atomic64_t size_hist[SIZE_HIST];
> +
> + void *sysctl_handle_common;
> + void *sysctl_handle_writeback;
> + void *sysctl_handle_invalidate;
> +
> + struct eio_sysctl sysctl_pending; /* sysctl values pending to become active */
> + struct eio_sysctl sysctl_active; /* sysctl currently active */
> +
> + char cache_devname[DEV_PATHLEN];
> + char disk_devname[DEV_PATHLEN];
> + char cache_name[DEV_PATHLEN];
> + char cache_gendisk_name[DEV_PATHLEN]; /* Used for SSD failure checks */
> + char cache_srcdisk_name[DEV_PATHLEN]; /* Used for SRC failure checks */
> + char ssd_uuid[DEV_PATHLEN];
> +
> + struct cacheblock_md8 *cache_md8;
> + sector_t cache_size; /* Cache size passed to ctr(), used by dmsetup info */
> + sector_t cache_dev_start_sect; /* starting sector of cache device */
> + u_int64_t index_zero; /* index of cache block with starting sector 0 */
> + u_int32_t num_sets; /* number of cache sets */
> + u_int32_t num_sets_bits; /* number of bits to encode "num_sets" */
> + u_int64_t num_sets_mask; /* mask value for bits in "num_sets" */
> +
> + struct eio_policy *policy_ops; /* Cache block Replacement policy */
> + u_int32_t req_policy; /* Policy requested by the user */
> + u_int32_t random; /* Use for random replacement policy */
> + void *sp_cache_blk; /* Per cache-block data structure */
> + void *sp_cache_set; /* Per cache-set data structure */
> + struct lru_ls *dirty_set_lru; /* lru for dirty sets : lru_list_t */
> + spinlock_t dirty_set_lru_lock; /* spinlock for dirty set lru */
> + struct delayed_work clean_aged_sets_work; /* work item for clean_aged_sets */
> + int is_clean_aged_sets_sched; /* to know whether clean aged sets is scheduled */
> + struct workqueue_struct *mdupdate_q; /* Workqueue to handle md updates */
> + struct workqueue_struct *callback_q; /* Workqueue to handle io callbacks */
> +};
> +
> +#define EIO_CACHE_IOSIZE 0
> +
> +#define EIO_ROUND_SECTOR(dmc, sector) (sector & (~(unsigned)(dmc->block_size - 1)))
> +#define EIO_ROUND_SET_SECTOR(dmc, sector) (sector & (~(unsigned)((dmc->block_size * dmc->assoc) - 1)))
> +
> +/*
> + * The bit definitions are exported to the user space and are in the very beginning of the file.
> + */
> +#define CACHE_VERBOSE_IS_SET(dmc) (((dmc)->cache_flags & CACHE_FLAGS_VERBOSE) ? 1 : 0)
> +#define CACHE_INVALIDATE_IS_SET(dmc) (((dmc)->cache_flags & CACHE_FLAGS_INVALIDATE) ? 1 : 0)
> +#define CACHE_FAST_REMOVE_IS_SET(dmc) (((dmc)->cache_flags & CACHE_FLAGS_FAST_REMOVE) ? 1 : 0)
> +#define CACHE_DEGRADED_IS_SET(dmc) (((dmc)->cache_flags & CACHE_FLAGS_DEGRADED) ? 1 : 0)
> +#define CACHE_SSD_ADD_INPROG_IS_SET(dmc) (((dmc)->cache_flags & CACHE_FLAGS_SSD_ADD_INPROG) ? 1 : 0)
> +#define CACHE_MD8_IS_SET(dmc) (((dmc)->cache_flags & CACHE_FLAGS_MD8) ? 1 : 0)
> +#define CACHE_FAILED_IS_SET(dmc) (((dmc)->cache_flags & CACHE_FLAGS_FAILED) ? 1 : 0)
> +#define CACHE_STALE_IS_SET(dmc) (((dmc)->cache_flags & CACHE_FLAGS_STALE) ? 1 : 0)
> +
> +/* Device failure handling. */
> +#define CACHE_SRC_IS_ABSENT(dmc) (((dmc)->eio_errors.no_source_dev == 1) ? 1 : 0)
> +
> +#define AUTOCLEAN_THRESHOLD_CROSSED(dmc) \
> + ((atomic64_read(&(dmc)->nr_ios) > (int64_t)(dmc)->sysctl_active.autoclean_threshold) || \
> + ((dmc)->sysctl_active.autoclean_threshold == 0))
> +
> +#define DIRTY_CACHE_THRESHOLD_CROSSED(dmc) \
> + (((atomic64_read(&(dmc)->nr_dirty) - atomic64_read(&(dmc)->clean_pendings)) >= \
> + (int64_t)((dmc)->sysctl_active.dirty_high_threshold * (dmc)->size) / 100) && \
> + ((dmc)->sysctl_active.dirty_high_threshold > (dmc)->sysctl_active.dirty_low_threshold))
> +
> +#define DIRTY_SET_THRESHOLD_CROSSED(dmc, set) \
> + (((dmc)->cache_sets[(set)].nr_dirty >= (u_int32_t)((dmc)->sysctl_active.dirty_set_high_threshold * (dmc)->assoc) / 100) && \
> + ((dmc)->sysctl_active.dirty_set_high_threshold > (dmc)->sysctl_active.dirty_set_low_threshold))
> +
> +/*
> + * Do not reverse the order of disk and cache! Code
> + * relies on this ordering. (Eg: eio_dm_io_async_bvec()).
> + */
> +struct job_io_regions {
> + struct eio_io_region disk; /* has to be the first member */
> + struct eio_io_region cache; /* has to be the second member */
> +};
> +
> +#define EB_MAIN_IO 1
> +#define EB_SUBORDINATE_IO 2
> +#define EB_INVAL 4
> +#define GET_BIO_FLAGS(ebio) ((ebio)->eb_bc->bc_bio->bi_rw)
> +#define VERIFY_BIO_FLAGS(ebio) VERIFY((ebio) && (ebio)->eb_bc && (ebio)->eb_bc->bc_bio)
> +
> +#define SET_BARRIER_FLAGS(rw_flags) (rw_flags |= (REQ_WRITE | REQ_FLUSH))
> +
> +struct eio_bio {
> + int eb_iotype;
> + struct bio_container *eb_bc;
> + unsigned eb_cacheset;
> + sector_t eb_sector; /*sector number*/
> + unsigned eb_size; /*size in bytes*/
> + struct bio_vec *eb_bv; /*bvec pointer*/
> + unsigned eb_nbvec; /*number of bio_vecs*/
> + int eb_dir; /* io direction*/
> + struct eio_bio *eb_next; /*used for splitting reads*/
> + index_t eb_index; /*for read bios*/
> + atomic_t eb_holdcount; /* ebio hold count, currently used only for dirty block I/O */
> + struct bio_vec eb_rbv[0];
> +};
> +
> +enum eio_io_dir {
> + EIO_IO_INVALID_DIR = 0,
> + CACHED_WRITE,
> + CACHED_READ,
> + UNCACHED_WRITE,
> + UNCACHED_READ,
> + UNCACHED_READ_AND_READFILL
> +};
> +
> +/* ASK
> + * Container for all eio_bio corresponding to a given bio
> + */
> +struct bio_container {
> + spinlock_t bc_lock; /* lock protecting the bc fields */
> + atomic_t bc_holdcount; /* number of ebios referencing bc */
> + struct bio *bc_bio; /* bio for the bc */
> + struct cache_c *bc_dmc; /* cache structure */
> + struct eio_bio *bc_mdlist; /* ebios waiting for md update */
> + int bc_mdwait; /* count of ebios that will do md update */
> + struct mdupdate_request *mdreqs; /* mdrequest structures required for md update */
> + struct set_seq *bc_setspan; /* sets spanned by the bc(used only for wb) */
> + struct set_seq bc_singlesspan; /* used(by wb) if bc spans a single set sequence */
> + enum eio_io_dir bc_dir; /* bc I/O direction */
> + int bc_error; /* error encountered during processing bc */
> + unsigned long bc_iotime; /* maintains i/o time in jiffies */
> + struct bio_container *bc_next; /* next bc in the chain */
> +};
> +
> +/* structure used as callback context during synchronous I/O */
> +struct sync_io_context {
> + struct rw_semaphore sio_lock;
> + unsigned long sio_error;
> +};
> +
> +struct kcached_job {
> + struct list_head list;
> + struct work_struct work;
> + struct cache_c *dmc;
> + struct eio_bio *ebio;
> + struct job_io_regions job_io_regions;
> + index_t index;
> + int action;
> + int error;
> + struct flash_cacheblock *md_sector;
> + struct bio_vec md_io_bvec;
> + struct kcached_job *next;
> +};
> +
> +struct ssd_rm_list {
> + struct cache_c *dmc;
> + int action;
> + dev_t devt;
> + dev_notifier_t note;
> + struct list_head list;
> +};
> +
> +struct dbn_index_pair {
> + sector_t dbn;
> + index_t index;
> +};
> +
> +/*
> + * Subsection 3: Function prototypes and definitions.
> + */
> +
> +struct kcached_job *eio_alloc_cache_job(void);
> +void eio_free_cache_job(struct kcached_job *job);
> +struct kcached_job *pop(struct list_head *jobs);
> +void push(struct list_head *jobs, struct kcached_job *job);
> +void do_work(struct work_struct *unused);
> +void update_job_cacheregion(struct kcached_job *job, struct cache_c *dmc,
> + struct eio_bio *bio);
> +void push_io(struct kcached_job *job);
> +void push_md_io(struct kcached_job *job);
> +void push_md_complete(struct kcached_job *job);
> +void push_uncached_io_complete(struct kcached_job *job);
> +int eio_io_empty(void);
> +int eio_md_io_empty(void);
> +int eio_md_complete_empty(void);
> +void eio_md_write_done(struct kcached_job *job);
> +void eio_ssderror_diskread(struct kcached_job *job);
> +void eio_md_write(struct kcached_job *job);
> +void eio_md_write_kickoff(struct kcached_job *job);
> +void eio_do_readfill(struct work_struct *work);
> +void eio_comply_dirty_thresholds(struct cache_c *dmc, index_t set);
> +void eio_clean_all(struct cache_c *dmc);
> +void eio_clean_for_reboot(struct cache_c *dmc);
> +void eio_clean_aged_sets(struct work_struct *work);
> +void eio_comply_dirty_thresholds(struct cache_c *dmc, index_t set);
> +#ifndef SSDCACHE
> +void eio_reclaim_lru_movetail(struct cache_c *dmc, index_t index,
> + struct eio_policy *);
> +#endif /* !SSDCACHE */
> +int eio_io_sync_vm(struct cache_c *dmc, struct eio_io_region *where, int rw,
> + struct bio_vec *bvec, int nbvec);
> +int eio_io_sync_pages(struct cache_c *dmc, struct eio_io_region *where, int rw,
> + struct page **pages, int num_bvecs);
> +void eio_update_sync_progress(struct cache_c *dmc);
> +void eio_plug_cache_device(struct cache_c *dmc);
> +void eio_unplug_cache_device(struct cache_c *dmc);
> +void eio_plug_disk_device(struct cache_c *dmc);
> +void eio_unplug_disk_device(struct cache_c *dmc);
> +int dm_io_async_bvec(unsigned int num_regions, struct eio_io_region *where,
> + int rw, struct bio_vec *bvec, eio_notify_fn fn,
> + void *context);
> +void eio_put_cache_device(struct cache_c *dmc);
> +void eio_suspend_caching(struct cache_c *dmc, dev_notifier_t note);
> +void eio_resume_caching(struct cache_c *dmc, char *dev);
> +int eio_ctr_ssd_add(struct cache_c *dmc, char *dev);
> +
> +/* procfs */
> +void eio_module_procfs_init(void);
> +void eio_module_procfs_exit(void);
> +void eio_procfs_ctr(struct cache_c *dmc);
> +void eio_procfs_dtr(struct cache_c *dmc);
> +
> +int eio_sb_store(struct cache_c *dmc);
> +
> +int eio_md_destroy(struct dm_target *tip, char *namep, char *srcp, char *cachep,
> + int force);
> +
> +/* eio_conf.c */
> +extern int eio_ctr(struct dm_target *ti, unsigned int argc, char **argv);
> +extern void eio_dtr(struct dm_target *ti);
> +extern int eio_md_destroy(struct dm_target *tip, char *namep, char *srcp,
> + char *cachep, int force);
> +extern int eio_ctr_ssd_add(struct cache_c *dmc, char *dev);
> +
> +/* thread related functions */
> +void *eio_create_thread(int (*func)(void *), void *context, char *name);
> +void eio_thread_exit(long exit_code);
> +void eio_wait_thread_exit(void *thrdptr, int *notifier);
> +
> +/* eio_main.c */
> +extern int eio_map(struct cache_c *, struct request_queue *, struct bio *);
> +extern void eio_md_write_done(struct kcached_job *job);
> +extern void eio_ssderror_diskread(struct kcached_job *job);
> +extern void eio_md_write(struct kcached_job *job);
> +extern void eio_md_write_kickoff(struct kcached_job *job);
> +extern void eio_do_readfill(struct work_struct *work);
> +extern void eio_check_dirty_thresholds(struct cache_c *dmc, index_t set);
> +extern void eio_clean_all(struct cache_c *dmc);
> +extern int eio_clean_thread_proc(void *context);
> +extern void eio_touch_set_lru(struct cache_c *dmc, index_t set);
> +extern void eio_inval_range(struct cache_c *dmc, sector_t iosector,
> + unsigned iosize);
> +extern int eio_invalidate_sanity_check(struct cache_c *dmc, u_int64_t iosector,
> + u_int64_t *iosize);
> +/*
> + * Invalidates all cached blocks without waiting for them to complete
> + * Should be called with incoming IO suspended
> + */
> +extern int eio_invalidate_cache(struct cache_c *dmc);
> +
> +/* eio_mem.c */
> +extern int eio_mem_init(struct cache_c *dmc);
> +extern u_int32_t eio_hash_block(struct cache_c *dmc, sector_t dbn);
> +extern unsigned int eio_shrink_dbn(struct cache_c *dmc, sector_t dbn);
> +extern sector_t eio_expand_dbn(struct cache_c *dmc, u_int64_t index);
> +extern void eio_invalidate_md(struct cache_c *dmc, u_int64_t index);
> +extern void eio_md4_dbn_set(struct cache_c *dmc, u_int64_t index,
> + u_int32_t dbn_24);
> +extern void eio_md8_dbn_set(struct cache_c *dmc, u_int64_t index, sector_t dbn);
> +
> +/* eio_procfs.c */
> +extern void eio_module_procfs_init(void);
> +extern void eio_module_procfs_exit(void);
> +extern void eio_procfs_ctr(struct cache_c *dmc);
> +extern void eio_procfs_dtr(struct cache_c *dmc);
> +extern int eio_version_query(size_t buf_sz, char *bufp);
> +
> +/* eio_subr.c */
> +extern void eio_free_cache_job(struct kcached_job *job);
> +extern void eio_do_work(struct work_struct *unused);
> +extern struct kcached_job *eio_new_job(struct cache_c *dmc, struct eio_bio *bio,
> + index_t index);
> +extern void eio_push_ssdread_failures(struct kcached_job *job);
> +extern void eio_push_md_io(struct kcached_job *job);
> +extern void eio_push_md_complete(struct kcached_job *job);
> +extern void eio_push_uncached_io_complete(struct kcached_job *job);
> +extern int eio_io_empty(void);
> +extern int eio_io_sync_vm(struct cache_c *dmc, struct eio_io_region *where,
> + int rw, struct bio_vec *bvec, int nbvec);
> +extern void eio_unplug_cache_device(struct cache_c *dmc);
> +extern void eio_put_cache_device(struct cache_c *dmc);
> +extern void eio_suspend_caching(struct cache_c *dmc, dev_notifier_t note);
> +extern void eio_resume_caching(struct cache_c *dmc, char *dev);
> +
> +static __inline__ void
> +EIO_DBN_SET(struct cache_c *dmc, u_int64_t index, sector_t dbn)
> +{
> + if (EIO_MD8(dmc))
> + eio_md8_dbn_set(dmc, index, dbn);
> + else
> + eio_md4_dbn_set(dmc, index, eio_shrink_dbn(dmc, dbn));
> + if (dbn == 0)
> + dmc->index_zero = index;
> +}
> +
> +static __inline__ u_int64_t EIO_DBN_GET(struct cache_c *dmc, u_int64_t index)
> +{
> + if (EIO_MD8(dmc))
> + return dmc->cache_md8[index].md8_md & EIO_MD8_DBN_MASK;
> +
> + return eio_expand_dbn(dmc, index);
> +}
> +
> +static __inline__ void
> +EIO_CACHE_STATE_SET(struct cache_c *dmc, u_int64_t index, u_int8_t cache_state)
> +{
> + if (EIO_MD8(dmc))
> + EIO_MD8_CACHE_STATE(dmc, index) = cache_state;
> + else
> + EIO_MD4_CACHE_STATE(dmc, index) = cache_state;
> +}
> +
> +static __inline__ u_int8_t
> +EIO_CACHE_STATE_GET(struct cache_c *dmc, u_int64_t index)
> +{
> + u_int8_t cache_state;
> +
> + if (EIO_MD8(dmc))
> + cache_state = EIO_MD8_CACHE_STATE(dmc, index);
> + else
> + cache_state = EIO_MD4_CACHE_STATE(dmc, index);
> + return cache_state;
> +}
> +
> +static __inline__ void
> +EIO_CACHE_STATE_OFF(struct cache_c *dmc, index_t index, u_int8_t bitmask)
> +{
> + u_int8_t cache_state = EIO_CACHE_STATE_GET(dmc, index);
> +
> + cache_state &= ~bitmask;
> + EIO_CACHE_STATE_SET(dmc, index, cache_state);
> +}
> +
> +static __inline__ void
> +EIO_CACHE_STATE_ON(struct cache_c *dmc, index_t index, u_int8_t bitmask)
> +{
> + u_int8_t cache_state = EIO_CACHE_STATE_GET(dmc, index);
> +
> + cache_state |= bitmask;
> + EIO_CACHE_STATE_SET(dmc, index, cache_state);
> +}
> +
> +void eio_set_warm_boot(void);
> +#endif /* defined(__KERNEL__) */
> +
> +#include "eio_ioctl.h"
> +
> +/* resolve conflict with scsi/scsi_device.h */
> +#ifdef __KERNEL__
> +#ifdef VERIFY
> +#undef VERIFY
> +#endif
> +#define ENABLE_VERIFY
> +#ifdef ENABLE_VERIFY
> +/* Like ASSERT() but always compiled in */
> +#define VERIFY(x) do { \
> + if (unlikely(!(x))) { \
> + dump_stack(); \
> + panic("VERIFY: assertion (%s) failed at %s (%d)\n", \
> + # x, __FILE__, __LINE__); \
> + } \
> +} while (0)
> +#else /* ENABLE_VERIFY */
> +#define VERIFY(x) do { } while (0);
> +#endif /* ENABLE_VERIFY */
BUG_ON()?
> +
> +extern sector_t eio_get_device_size(struct eio_bdev *);
> +extern sector_t eio_get_device_start_sect(struct eio_bdev *);
> +#endif /* __KERNEL__ */
> +
> +#define EIO_INIT_EVENT(ev) \
> + do { \
> + (ev)->process = NULL; \
> + } while (0)
> +
> +/*Assumes that the macro gets called under the same spinlock as in wait event*/
> +#define EIO_SET_EVENT_AND_UNLOCK(ev, sl, flags) \
> + do { \
> + struct task_struct *p = NULL; \
> + if ((ev)->process) { \
> + (p) = (ev)->process; \
> + (ev)->process = NULL; \
> + } \
> + spin_unlock_irqrestore((sl), flags); \
> + if (p) { \
> + (void)wake_up_process(p); \
> + } \
> + } while (0)
> +
> +/*Assumes that the spin lock sl is taken while calling this macro*/
> +#define EIO_WAIT_EVENT(ev, sl, flags) \
> + do { \
> + (ev)->process = current; \
> + set_current_state(TASK_INTERRUPTIBLE); \
> + spin_unlock_irqrestore((sl), flags); \
> + (void)schedule_timeout(10 * HZ); \
> + spin_lock_irqsave((sl), flags); \
> + (ev)->process = NULL; \
> + } while (0)
> +
> +#define EIO_CLEAR_EVENT(ev) \
> + do { \
> + (ev)->process = NULL; \
> + } while (0)
> +
> +#include "eio_setlru.h"
> +#include "eio_policy.h"
> +#define EIO_CACHE(dmc) (EIO_MD8(dmc) ? (void *)dmc->cache_md8 : (void *)dmc->cache)
> +
> +#endif /* !EIO_INC_H */
Ooookay, that's enough, I need a break, I'll review more later.
--D
More information about the dm-devel
mailing list