<div dir="ltr"><div>I propose a new DM target, dm-lc.</div><div><br></div><div>"lc" means Log-sturctured Caching and as the name implies</div><div>dm-lc is a caching software with only write-back mode.</div><div>
It batches random writes into a big sequential write to</div><div>the cache device.</div><div><br></div><div>It is now under development as a</div><div>portable kernel module in my Github repo</div><div><a href="https://github.com/akiradeveloper/dm-lc">https://github.com/akiradeveloper/dm-lc</a></div>
<div><br></div><div>Features:</div><div><br></div><div>(1) Simplicity</div><div>Since dm-lc only focuses on write-back caching,</div><div>the implementation is simpler, 3k LOC, than existing ones</div><div>such as dm-cache, bcache and enhanceIO</div>
<div>and thus less buggy. This is the point.</div><div>I think write-back cache software should be small and simple</div><div>because write-back cache may destroy the storage</div><div>in worst case scenario.</div><div><br>
</div><div>Stacking above the write-through/write-around mode of</div><div>existing caching software is a choice of usage.</div><div>dm-lc can co-exists with existing cache softwares.</div><div><br></div><div>(2) Superb Performance</div>
<div>Because the code is simple,</div><div>dm-lc performs better than existing ones</div><div>in random write throughput.</div><div>The benchmark is done using fio tool.</div><div>It only loses 3% throughtput compared to</div>
<div>cache device's sequential write throughput.</div><div>The detail is, 259MB/sec random writes by HDD+SSD</div><div>against 266MB/sec sequential write on SSD.</div><div><br></div><div>Plus, dm-lc is capable of 1.5GB/sec random write</div>
<div>throughput with a fast enough cache.</div><div>This means the code path is short and the locking is effective.</div><div><br></div><div>For more detail, please read</div><div><a href="https://github.com/akiradeveloper/dm-lc/tree/develop/performance">https://github.com/akiradeveloper/dm-lc/tree/develop/performance</a></div>
<div><br></div><div>(3) Durability for server crash</div><div>Servers sometimes crash but any data</div><div>on the storage should not be gone.</div><div>Some existing softwares have</div><div>cache metadata on memory in runtime</div>
<div>and lose them all in server crash.</div><div>dm-lc, as it write in log-structured manner</div><div>which means it writes metadata and data together</div><div>on the cache device, does't lose any data in such failures.</div>
<div><br></div><div>To answer to the expectation from upper layer,</div><div>dm-lc can handle REQ_FUA/REQ_FLUSH flags correctly</div><div>but not loses performance badly by lazy execution technique.</div><div><br></div><div>
(4) Autonomous migration</div><div>The algorithm of migration or writing back</div><div>is key factor in write-back cache.</div><div>dm-lc have a userland daemon to</div><div>autonomusly turn on and off migration</div><div>
according to the load of backing store.</div><div><br></div><div>For more detail, please read dm-lc.txt</div><div>that is now copy and paste on this mail below.</div><div><br></div><div>How to Taste?:</div><div>Clone the repository and follow the</div>
<div>"Quick Start" guideline.</div><div>It is almost automated and just 5 minutes work.</div><div>Benchmarkings and testings are prepared</div><div>in scripts and you can try them.</div><div><br></div><div>About Upstreaming:</div>
<div>Now, I think it's time to go upstreaming.</div><div>The softwares is well-tested and</div><div>shows nice performance boost in benchmarking using fio.</div><div>Merging into the mainline tree is the only way to</div>
<div>sophisticate the code and document more</div><div>for dm-lc at this point.</div><div><br></div><div>Before that, I think I need to discuss with you DM guys.</div><div>Q. Do you have some comments on the code and document?</div>
<div><br></div><div>Actually, I am not a programming newbie but</div><div>really new to Linux upstreaming community</div><div>so I have no clue to what to do next.</div><div><br></div><div>My plan is to first remove</div>
<div>all the code for version switch</div><div>and merge into some kernel tree in local, test</div><div>and then repost to this ML as a patch.</div><div>Q. What tree do I need to use for merging and testing?</div><div><br>
</div><div>Research Background:</div><div>The philosophy is based on Disk Caching Disk</div><div><a href="http://www.ele.uri.edu/research/hpcl/DCD/DCD.html">http://www.ele.uri.edu/research/hpcl/DCD/DCD.html</a></div><div>
My work is to build an efficient and thorough</div><div>implementation for Linux.</div><div><br></div><div>As a related work,</div><div>There is a recent study by Microsoft</div><div>"Extending SSD Lifetimes with Disk-Based Write Caches"</div>
<div>which is also on the philosophy of DCD.</div><div><a href="http://research.microsoft.com/apps/pubs/default.aspx?id=115352">http://research.microsoft.com/apps/pubs/default.aspx?id=115352</a></div><div><br></div><div><br>
</div><div><br></div><div>Belows are the 3k lines of kernel code (dm-lc.c) and document (dm-lc.txt).</div><div>The kernel code went through the <a href="http://checkpatch.pl">checkpatch.pl</a> and</div><div>only 17 warnings for version switch is remained at this time.</div>
<div><br></div><div>(dm-lc.c)</div><div><br></div><div>/*</div><div> * dm-lc.c : Log-structured Caching for Linux.</div><div> * Copyright (C) 2012-2013 Akira Hayakawa <<a href="mailto:ruby.wktk@gmail.com">ruby.wktk@gmail.com</a>></div>
<div> *</div><div> * This file is released under the GPL.</div><div> */</div><div><br></div><div>#define DM_MSG_PREFIX "lc"</div><div><br></div><div>#include <linux/module.h></div><div>#include <linux/version.h></div>
<div>#include <linux/list.h></div><div>#include <linux/slab.h></div><div>#include <linux/mutex.h></div><div>#include <linux/sched.h></div><div>#include <linux/device-mapper.h></div><div>#include <linux/dm-io.h></div>
<div>#include <linux/timer.h></div><div><br></div><div>/*</div><div> * Comments are described</div><div> * in the case the segment size order is 11</div><div> * which means the segment size is the maxium, 1MB.</div>
<div> */</div><div><br></div><div>/*</div><div> * (1 << x) sector.</div><div> * 4 <= x <= 11</div><div> */</div><div>#define LC_SEGMENTSIZE_ORDER 11</div><div>#define NR_CACHES_INSEG ((1 << (LC_SEGMENTSIZE_ORDER - 3)) - 1)</div>
<div><br></div><div>static void *do_kmalloc_retry(size_t size, gfp_t flags, int lineno)</div><div>{</div><div><span class="" style="white-space:pre"> </span>int count = 0;</div><div><span class="" style="white-space:pre">     </span>void *p;</div>
<div><br></div><div>retry_alloc:</div><div><span class="" style="white-space:pre">        </span>p = kmalloc(size, flags);</div><div><span class="" style="white-space:pre">  </span>if (!p) {</div><div><span class="" style="white-space:pre">          </span>count++;</div>
<div><span class="" style="white-space:pre">            </span>DMERR("L.%d: fail allocation(count:%d)", lineno, count);</div><div><span class="" style="white-space:pre">         </span>schedule_timeout_interruptible(msecs_to_jiffies(1));</div>
<div><span class="" style="white-space:pre">            </span>goto retry_alloc;</div><div><span class="" style="white-space:pre">  </span>}</div><div><span class="" style="white-space:pre">  </span>return p;</div><div>}</div><div>#define kmalloc_retry(size, flags) do_kmalloc_retry((size), (flags), __LINE__)</div>
<div><br></div><div>struct part {</div><div><span class="" style="white-space:pre">       </span>void *memory;</div><div>};</div><div><br></div><div>struct arr {</div><div><span class="" style="white-space:pre"> </span>struct part *parts;</div>
<div><span class="" style="white-space:pre">    </span>size_t nr_elems;</div><div><span class="" style="white-space:pre">   </span>size_t elemsize;</div><div>};</div><div><br></div><div>#define ALLOC_SIZE (1 << 16)</div>
<div>static size_t nr_elems_in_part(struct arr *arr)</div><div>{</div><div><span class="" style="white-space:pre">      </span>return ALLOC_SIZE / arr->elemsize;</div><div>};</div><div><br></div><div>static size_t nr_parts(struct arr *arr)</div>
<div>{</div><div><span class="" style="white-space:pre">    </span>return dm_div_up(arr->nr_elems, nr_elems_in_part(arr));</div><div>}</div><div><br></div><div>static struct arr *make_arr(size_t elemsize, size_t nr_elems)</div>
<div>{</div><div><span class="" style="white-space:pre">    </span>struct arr *arr = kmalloc(sizeof(*arr), GFP_KERNEL);</div><div><span class="" style="white-space:pre">       </span>arr->elemsize = elemsize;</div><div><span class="" style="white-space:pre">       </span>arr->nr_elems = nr_elems;</div>
<div><span class="" style="white-space:pre">    </span>arr->parts = kmalloc(sizeof(struct part) * nr_parts(arr), GFP_KERNEL);</div><div><br></div><div><span class="" style="white-space:pre"> </span>size_t i;</div><div><span class="" style="white-space:pre">  </span>for (i = 0; i < nr_parts(arr); i++) {</div>
<div><span class="" style="white-space:pre">            </span>struct part *part = arr->parts + i;</div><div><span class="" style="white-space:pre">             </span>part->memory = kmalloc(ALLOC_SIZE, GFP_KERNEL);</div><div><span class="" style="white-space:pre"> </span>}</div>
<div><span class="" style="white-space:pre">    </span>return arr;</div><div>}</div><div><br></div><div>static void kill_arr(struct arr *arr)</div><div>{</div><div><span class="" style="white-space:pre">   </span>size_t i;</div>
<div><span class="" style="white-space:pre">    </span>for (i = 0; i < nr_parts(arr); i++) {</div><div><span class="" style="white-space:pre">           </span>struct part *part = arr->parts + i;</div><div><span class="" style="white-space:pre">             </span>kfree(part->memory);</div>
<div><span class="" style="white-space:pre">    </span>}</div><div><span class="" style="white-space:pre">  </span>kfree(arr->parts);</div><div><span class="" style="white-space:pre">      </span>kfree(arr);</div><div>}</div><div>
<br></div><div>static void *arr_at(struct arr *arr, size_t i)</div><div>{</div><div><span class="" style="white-space:pre"> </span>size_t n = nr_elems_in_part(arr);</div><div><span class="" style="white-space:pre">  </span>size_t j = i / n;</div>
<div><span class="" style="white-space:pre">    </span>size_t k = i % n;</div><div><span class="" style="white-space:pre">  </span>struct part *part = arr->parts + j;</div><div><span class="" style="white-space:pre">     </span>return part->memory + (arr->elemsize * k);</div>
<div>}</div><div><br></div><div>static struct dm_io_client *lc_io_client;</div><div><br></div><div>struct safe_io {</div><div><span class="" style="white-space:pre">   </span>struct work_struct work;</div><div><span class="" style="white-space:pre">   </span>int err;</div>
<div><span class="" style="white-space:pre">    </span>unsigned long err_bits;</div><div><span class="" style="white-space:pre">    </span>struct dm_io_request *io_req;</div><div><span class="" style="white-space:pre">      </span>struct dm_io_region *region;</div>
<div><span class="" style="white-space:pre">    </span>unsigned num_regions;</div><div>};</div><div>static struct workqueue_struct *safe_io_wq;</div><div><br></div><div>static void safe_io_proc(struct work_struct *work)</div>
<div>{</div><div><span class="" style="white-space:pre">    </span>struct safe_io *io = container_of(work, struct safe_io, work);</div><div><span class="" style="white-space:pre">     </span>io->err_bits = 0;</div><div><span class="" style="white-space:pre">       </span>io->err = dm_io(io->io_req, io->num_regions, io->region, &io->err_bits);</div>
<div>}</div><div><br></div><div>/*</div><div> * dm_io wrapper.</div><div> * @thread run operation this in other thread to avoid deadlock.</div><div> */</div><div>static int dm_safe_io_internal(</div><div><span class="" style="white-space:pre">           </span>struct dm_io_request *io_req,</div>
<div><span class="" style="white-space:pre">            </span>struct dm_io_region *region, unsigned num_regions,</div><div><span class="" style="white-space:pre">         </span>unsigned long *err_bits, bool thread, int lineno)</div><div>
{</div><div><span class="" style="white-space:pre">       </span>int err;</div><div><span class="" style="white-space:pre">   </span>if (thread) {</div><div><span class="" style="white-space:pre">              </span>struct safe_io io = {</div>
<div><span class="" style="white-space:pre">                    </span>.io_req = io_req,</div><div><span class="" style="white-space:pre">                  </span>.region = region,</div><div><span class="" style="white-space:pre">                  </span>.num_regions = num_regions,</div>
<div><span class="" style="white-space:pre">            </span>};</div><div><br></div><div><span class="" style="white-space:pre">                </span>INIT_WORK_ONSTACK(&io.work, safe_io_proc);</div><div><br></div><div><span class="" style="white-space:pre">            </span>queue_work(safe_io_wq, &io.work);</div>
<div><span class="" style="white-space:pre">            </span>flush_work(&io.work);</div><div><br></div><div><span class="" style="white-space:pre">         </span>err = io.err;</div><div><span class="" style="white-space:pre">              </span>*err_bits = io.err_bits;</div>
<div><span class="" style="white-space:pre">    </span>} else {</div><div><span class="" style="white-space:pre">           </span>err = dm_io(io_req, num_regions, region, err_bits);</div><div><span class="" style="white-space:pre">        </span>}</div>
<div><br></div><div><span class="" style="white-space:pre">   </span>dev_t dev = region->bdev->bd_dev;</div><div><span class="" style="white-space:pre">    </span>if (err || *err_bits) {</div><div><span class="" style="white-space:pre">            </span>DMERR("L.%d: io err occurs err(%d), err_bits(%lu)",</div>
<div><span class="" style="white-space:pre">                            </span>lineno, err, *err_bits);</div><div><span class="" style="white-space:pre">           </span>DMERR("rw(%d), sector(%lu), dev(%u:%u)",</div><div><span class="" style="white-space:pre">                         </span>io_req->bi_rw, region->sector,</div>
<div><span class="" style="white-space:pre">                            </span>MAJOR(dev), MINOR(dev));</div><div><span class="" style="white-space:pre">   </span>}</div><div><br></div><div><span class="" style="white-space:pre"> </span>return err;</div>
<div>}</div><div>#define dm_safe_io(io_req, region, num_regions, thread) \</div><div><span class="" style="white-space:pre">    </span>dm_safe_io_internal((io_req), (region), (num_regions), \</div><div><span class="" style="white-space:pre">                   </span>(thread), __LINE__)</div>
<div><br></div><div>static void dm_safe_io_retry_internal(</div><div><span class="" style="white-space:pre">              </span>struct dm_io_request *io_req,</div><div><span class="" style="white-space:pre">              </span>struct dm_io_region *region, unsigned num_regions,</div>
<div><span class="" style="white-space:pre">            </span>bool thread, int lineno)</div><div>{</div><div><span class="" style="white-space:pre">   </span>int err;</div><div><span class="" style="white-space:pre">   </span>unsigned long err_bits;</div>
<div><br></div><div><span class="" style="white-space:pre">   </span>int count = 0;</div><div><br></div><div>retry_io:</div><div><span class="" style="white-space:pre">    </span>err_bits = 0;</div><div><span class="" style="white-space:pre">      </span>err = dm_safe_io_internal(io_req, region, num_regions, &err_bits,</div>
<div><span class="" style="white-space:pre">                    </span>thread, lineno);</div><div><br></div><div><span class="" style="white-space:pre">  </span>dev_t dev = region->bdev->bd_dev;</div><div><span class="" style="white-space:pre">    </span>if (err || err_bits) {</div>
<div><span class="" style="white-space:pre">            </span>count++;</div><div><span class="" style="white-space:pre">           </span>DMERR("failed io count(%d)", count);</div><div><br></div><div><span class="" style="white-space:pre">            </span>schedule_timeout_interruptible(msecs_to_jiffies(1000));</div>
<div><span class="" style="white-space:pre">            </span>goto retry_io;</div><div><span class="" style="white-space:pre">     </span>}</div><div><br></div><div><span class="" style="white-space:pre"> </span>if (count) {</div><div>
<span class="" style="white-space:pre">               </span>DMERR("L.%d: io has just turned fail to OK.", lineno);</div><div><span class="" style="white-space:pre">           </span>DMERR("rw(%d), sector(%lu), dev(%u:%u)",</div>
<div><span class="" style="white-space:pre">                    </span>io_req->bi_rw, region->sector, MAJOR(dev), MINOR(dev));</div><div><span class="" style="white-space:pre">      </span>}</div><div>}</div><div>#define dm_safe_io_retry(io_req, region, num_regions, thread) \</div>
<div><span class="" style="white-space:pre">    </span>dm_safe_io_retry_internal((io_req), (region), \</div><div><span class="" style="white-space:pre">                            </span>  (num_regions), (thread), __LINE__)</div><div><br></div><div>
/*</div><div> * device_id = 0</div><div> * is reserved for invalid cache block.</div><div> */</div><div>typedef u8 device_id;</div><div><br></div><div>struct lc_device {</div><div><span class="" style="white-space:pre"> </span>struct kobject kobj;</div>
<div><br></div><div><span class="" style="white-space:pre">   </span>unsigned char migrate_threshold;</div><div><br></div><div><span class="" style="white-space:pre">  </span>struct lc_cache *cache;</div><div><br></div><div>
<span class="" style="white-space:pre">       </span>device_id id;</div><div><span class="" style="white-space:pre">      </span>struct dm_dev *device;</div><div><br></div><div><span class="" style="white-space:pre">    </span>atomic64_t nr_dirty_caches;</div>
<div><br></div><div><span class="" style="white-space:pre">   </span>struct mapped_device *md;</div><div>};</div><div><br></div><div>/*</div><div> * cache_id = 0</div><div> * is reserved for no cache.</div><div> */</div><div>
typedef u8 cache_id;</div><div><br></div><div>/*</div><div> * dm-lc can't manange</div><div> * more than (1 << 8)</div><div> * virtual devices and cache devices.</div><div> */</div><div>#define LC_NR_SLOTS ((1 << 8) - 1)</div>
<div><br></div><div>cache_id cache_id_ptr;</div><div><br></div><div>struct lc_cache *lc_caches[LC_NR_SLOTS];</div><div><br></div><div>struct lc_device *lc_devices[LC_NR_SLOTS];</div><div><br></div><div>/*</div><div> * Type for cache line index.</div>
<div> *</div><div> * dm-lc can supoort a cache device</div><div> * with size less than 4KB * (1 << 32)</div><div> * that is 16TB.</div><div> * Needless to say, this is enough.</div><div> */</div><div>typedef u32 cache_nr;</div>
<div><br></div><div>/*</div><div> * Accounts for a 4KB cache line</div><div> * which consists of eight sectors</div><div> * that is managed by dirty bit for each.</div><div> *</div><div> * This allows partial writes</div>
<div> * that frees VFS layer from</div><div> * operating read-modify-write to</div><div> * commit full 4KB page to block layer.</div><div> */</div><div>struct metablock {</div><div><span class="" style="white-space:pre">        </span>sector_t sector;</div>
<div><br></div><div><span class="" style="white-space:pre">   </span>cache_nr idx; /* const. 4B. */</div><div><br></div><div><span class="" style="white-space:pre">    </span>struct hlist_node ht_list;</div><div><br></div><div>
<span class="" style="white-space:pre">       </span>/*</div><div><span class="" style="white-space:pre"> </span> * 8 bit flag for dirtiness</div><div><span class="" style="white-space:pre">        </span> * for each sector in cache line.</div>
<div><span class="" style="white-space:pre">    </span> *</div><div><span class="" style="white-space:pre"> </span> * Now we recover only dirty caches</div><div><span class="" style="white-space:pre">        </span> * in crash recovery.</div>
<div><span class="" style="white-space:pre">    </span> *</div><div><span class="" style="white-space:pre"> </span> * Adding recover flag</div><div><span class="" style="white-space:pre">     </span> * to recover clean caches</div>
<div><span class="" style="white-space:pre">    </span> * complicate the code.</div><div><span class="" style="white-space:pre">    </span> */</div><div><span class="" style="white-space:pre">        </span>u8 dirty_bits;</div><div><br>
</div><div><span class="" style="white-space:pre">        </span>device_id device_id;</div><div>};</div><div><br></div><div>static void inc_nr_dirty_caches(device_id id)</div><div>{</div><div><span class="" style="white-space:pre"> </span>struct lc_device *o = lc_devices[id];</div>
<div><span class="" style="white-space:pre">    </span>BUG_ON(!o);</div><div><span class="" style="white-space:pre">        </span>atomic64_inc(&o->nr_dirty_caches);</div><div>}</div><div><br></div><div>static void dec_nr_dirty_caches(device_id id)</div>
<div>{</div><div><span class="" style="white-space:pre">    </span>struct lc_device *o = lc_devices[id];</div><div><span class="" style="white-space:pre">      </span>BUG_ON(!o);</div><div><span class="" style="white-space:pre">        </span>atomic64_dec(&o->nr_dirty_caches);</div>
<div>}</div><div><br></div><div>struct metablock_device {</div><div><span class="" style="white-space:pre">   </span>sector_t sector;</div><div><span class="" style="white-space:pre">   </span>device_id device_id;</div><div><br>
</div><div><span class="" style="white-space:pre">        </span>u8 dirty_bits;</div><div><br></div><div><span class="" style="white-space:pre">    </span>u32 lap;</div><div>} __packed;</div><div><br></div><div>/*</div><div> * We preallocate 64 * 1MB writebuffers and use them cyclically.</div>
<div> * Dynamic allocation using kmalloc results in get_free_page path</div><div> * that may incur page reclaim which slowdown the system.</div><div> * This is why we statically preallocate these buffers.</div><div> *</div>
<div> * The number 64, though hueristically determined,</div><div> * is usually enough for any workload</div><div> * if having cache device with sufficient</div><div> * sequential write throughput, say 100MB/s.</div><div>
 */</div><div>#define NR_WB_POOL 64</div><div>struct writebuffer {</div><div><span class="" style="white-space:pre">      </span>void *data;</div><div><span class="" style="white-space:pre">        </span>struct completion done;</div>
<div>};</div><div><br></div><div>#define SZ_MAX (~(size_t)0) /* renamed backport */</div><div>struct segment_header {</div><div><span class="" style="white-space:pre">   </span>struct metablock mb_array[NR_CACHES_INSEG];</div>
<div><br></div><div><span class="" style="white-space:pre">   </span>/*</div><div><span class="" style="white-space:pre"> </span> * id is not circulated but uniformly increases.</div><div><span class="" style="white-space:pre">   </span> * id = 0 is used to tell that the segment is invalid</div>
<div><span class="" style="white-space:pre">    </span> * and valid id starts from 1.</div><div><span class="" style="white-space:pre">     </span> */</div><div><span class="" style="white-space:pre">        </span>size_t global_id;</div>
<div><br></div><div><span class="" style="white-space:pre">   </span>u8 length; /* Log length. <= NR_CACHES_INSEG */</div><div><br></div><div><span class="" style="white-space:pre">        </span>cache_nr start_idx; /* const */</div>
<div><span class="" style="white-space:pre">    </span>sector_t start_sector; /* const */</div><div><br></div><div><span class="" style="white-space:pre">        </span>struct completion flush_done;</div><div><br></div><div><span class="" style="white-space:pre">     </span>struct completion migrate_done;</div>
<div><br></div><div><span class="" style="white-space:pre">   </span>spinlock_t lock;</div><div><br></div><div><span class="" style="white-space:pre">  </span>atomic_t nr_inflight_ios;</div><div>};</div><div><br></div><div>#define lockseg(seg, flags) spin_lock_irqsave(&(seg)->lock, flags)</div>
<div>#define unlockseg(seg, flags) spin_unlock_irqrestore(&(seg)->lock, flags)</div><div><br></div><div>static u8 atomic_read_mb_dirtiness(struct segment_header *seg,</div><div><span class="" style="white-space:pre">                               </span>   struct metablock *mb)</div>
<div>{</div><div><span class="" style="white-space:pre">    </span>u8 r;</div><div><span class="" style="white-space:pre">      </span>unsigned long flags;</div><div><br></div><div><span class="" style="white-space:pre">      </span>lockseg(seg, flags);</div>
<div><span class="" style="white-space:pre">    </span>r = mb->dirty_bits;</div><div><span class="" style="white-space:pre">     </span>unlockseg(seg, flags);</div><div><br></div><div><span class="" style="white-space:pre">    </span>return r;</div>
<div>}</div><div><br></div><div>/* At most 4KB in total. */</div><div>struct segment_header_device {</div><div><span class="" style="white-space:pre">    </span>/* --- at most512 byte for atomicity. ---*/</div><div><span class="" style="white-space:pre">        </span>size_t global_id;</div>
<div><span class="" style="white-space:pre">    </span>u8 length;</div><div><span class="" style="white-space:pre"> </span>u32 lap; /* initially 0. 1 for the first lap. */</div><div><span class="" style="white-space:pre">   </span>/* -----------------------*/</div>
<div><span class="" style="white-space:pre">    </span>/* This array must locate at the tail */</div><div><span class="" style="white-space:pre">   </span>struct metablock_device mbarr[NR_CACHES_INSEG];</div><div>} __packed;</div>
<div><br></div><div>struct lookup_key {</div><div><span class="" style="white-space:pre"> </span>device_id device_id;</div><div><span class="" style="white-space:pre">       </span>sector_t sector;</div><div>};</div><div><br></div>
<div>enum STATFLAG {</div><div><span class="" style="white-space:pre">      </span>STAT_WRITE = 0,</div><div><span class="" style="white-space:pre">    </span>STAT_HIT,</div><div><span class="" style="white-space:pre">  </span>STAT_ON_BUFFER,</div>
<div><span class="" style="white-space:pre">    </span>STAT_FULLSIZE,</div><div>};</div><div>#define STATLEN (1 << 4)</div><div><br></div><div>struct ht_head {</div><div><span class="" style="white-space:pre">       </span>struct hlist_head ht_list;</div>
<div>};</div><div><br></div><div>struct lc_cache {</div><div><span class="" style="white-space:pre">  </span>struct kobject kobj;</div><div><br></div><div><span class="" style="white-space:pre">      </span>cache_id id;</div><div>
<span class="" style="white-space:pre">       </span>struct dm_dev *device;</div><div><span class="" style="white-space:pre">     </span>struct mutex io_lock;</div><div><span class="" style="white-space:pre">      </span>cache_nr nr_caches; /* const */</div>
<div><span class="" style="white-space:pre">    </span>size_t nr_segments; /* const */</div><div><span class="" style="white-space:pre">    </span>struct arr *segment_header_array;</div><div><br></div><div><span class="" style="white-space:pre"> </span>/*</div>
<div><span class="" style="white-space:pre">    </span> * Chained hashtable.</div><div><span class="" style="white-space:pre">      </span> */</div><div><span class="" style="white-space:pre">        </span>struct arr *htable;</div><div>
<span class="" style="white-space:pre">       </span>size_t htsize;</div><div><span class="" style="white-space:pre">     </span>struct ht_head *null_head;</div><div><br></div><div><span class="" style="white-space:pre">        </span>cache_nr cursor; /* Index that has done write */</div>
<div><span class="" style="white-space:pre">    </span>struct segment_header *current_seg;</div><div><span class="" style="white-space:pre">        </span>struct writebuffer *current_wb; /* Preallocated buffer. 1024KB */</div><div>
<span class="" style="white-space:pre">       </span>struct writebuffer *wb_pool;</div><div><br></div><div><span class="" style="white-space:pre">      </span>size_t last_migrated_segment_id;</div><div><span class="" style="white-space:pre">   </span>size_t last_flushed_segment_id;</div>
<div><span class="" style="white-space:pre">    </span>size_t reserving_segment_id;</div><div><br></div><div><span class="" style="white-space:pre">      </span>/*</div><div><span class="" style="white-space:pre"> </span> * For Flush daemon</div>
<div><span class="" style="white-space:pre">    </span> */</div><div><span class="" style="white-space:pre">        </span>spinlock_t flush_queue_lock;</div><div><span class="" style="white-space:pre">       </span>struct list_head flush_queue;</div>
<div><span class="" style="white-space:pre">    </span>struct work_struct flush_work;</div><div><span class="" style="white-space:pre">     </span>wait_queue_head_t flush_wait_queue;</div><div><span class="" style="white-space:pre">        </span>struct workqueue_struct *flush_wq;</div>
<div><br></div><div><span class="" style="white-space:pre">   </span>/*</div><div><span class="" style="white-space:pre"> </span> * For Migration daemon</div><div><span class="" style="white-space:pre">    </span> */</div><div>
<span class="" style="white-space:pre">       </span>bool allow_migrate;</div><div><span class="" style="white-space:pre">        </span>bool force_migrate;</div><div><span class="" style="white-space:pre">        </span>struct workqueue_struct *migrate_wq;</div>
<div><span class="" style="white-space:pre">    </span>struct work_struct migrate_work;</div><div><br></div><div><span class="" style="white-space:pre">  </span>/*</div><div><span class="" style="white-space:pre"> </span> * For migration I/O</div>
<div><span class="" style="white-space:pre">    </span> */</div><div><span class="" style="white-space:pre">        </span>wait_queue_head_t migrate_wait_queue;</div><div><span class="" style="white-space:pre">      </span>atomic_t migrate_fail_count;</div>
<div><span class="" style="white-space:pre">    </span>atomic_t migrate_io_count;</div><div><span class="" style="white-space:pre"> </span>u8 dirtiness_snapshot[NR_CACHES_INSEG];</div><div><span class="" style="white-space:pre">    </span>bool migrate_dests[LC_NR_SLOTS];</div>
<div><span class="" style="white-space:pre">    </span>void *migrate_buffer;</div><div><br></div><div><span class="" style="white-space:pre">     </span>/*</div><div><span class="" style="white-space:pre"> </span> * For deferred flush/FUA handling.</div>
<div><span class="" style="white-space:pre">    </span> */</div><div><span class="" style="white-space:pre">        </span>struct timer_list barrier_deadline_timer;</div><div><span class="" style="white-space:pre">  </span>struct bio_list barrier_ios;</div>
<div><span class="" style="white-space:pre">    </span>unsigned long barrier_deadline_ms;</div><div><span class="" style="white-space:pre"> </span>struct work_struct barrier_deadline_work;</div><div><br></div><div><span class="" style="white-space:pre"> </span>bool on_terminate;</div>
<div><br></div><div><span class="" style="white-space:pre">   </span>atomic64_t stat[STATLEN];</div><div><br></div><div><span class="" style="white-space:pre"> </span>unsigned long update_interval;</div><div><span class="" style="white-space:pre">     </span>unsigned long commit_super_block_interval;</div>
<div><span class="" style="white-space:pre">    </span>unsigned long flush_current_buffer_interval;</div><div>};</div><div><br></div><div>static void inc_stat(struct lc_cache *cache,</div><div><span class="" style="white-space:pre">          </span>     int rw, bool found, bool on_buffer, bool fullsize)</div>
<div>{</div><div><span class="" style="white-space:pre">    </span>int i = 0;</div><div><span class="" style="white-space:pre"> </span>if (rw)</div><div><span class="" style="white-space:pre">            </span>i |= (1 << STAT_WRITE);</div>
<div><span class="" style="white-space:pre">    </span>if (found)</div><div><span class="" style="white-space:pre">         </span>i |= (1 << STAT_HIT);</div><div><span class="" style="white-space:pre">        </span>if (on_buffer)</div>
<div><span class="" style="white-space:pre">            </span>i |= (1 << STAT_ON_BUFFER);</div><div><span class="" style="white-space:pre">  </span>if (fullsize)</div><div><span class="" style="white-space:pre">              </span>i |= (1 << STAT_FULLSIZE);</div>
<div><br></div><div><span class="" style="white-space:pre">   </span>atomic64_t *v = &cache->stat[i];</div><div><span class="" style="white-space:pre">    </span>atomic64_inc(v);</div><div>}</div><div><br></div><div>static void clear_stat(struct lc_cache *cache)</div>
<div>{</div><div><span class="" style="white-space:pre">    </span>int i;</div><div><span class="" style="white-space:pre">     </span>for (i = 0; i < STATLEN; i++) {</div><div><span class="" style="white-space:pre">         </span>atomic64_t *v = &cache->stat[i];</div>
<div><span class="" style="white-space:pre">            </span>atomic64_set(v, 0);</div><div><span class="" style="white-space:pre">        </span>}</div><div>}</div><div><br></div><div>static struct metablock *mb_at(struct lc_cache *cache, cache_nr idx)</div>
<div>{</div><div><span class="" style="white-space:pre">    </span>size_t seg_idx = idx / NR_CACHES_INSEG;</div><div><span class="" style="white-space:pre">    </span>struct segment_header *seg =</div><div><span class="" style="white-space:pre">               </span>arr_at(cache->segment_header_array, seg_idx);</div>
<div><span class="" style="white-space:pre">    </span>cache_nr idx_inseg = idx % NR_CACHES_INSEG;</div><div><span class="" style="white-space:pre">        </span>return seg->mb_array + idx_inseg;</div><div>}</div><div><br></div>
<div>static void mb_array_empty_init(struct lc_cache *cache)</div><div>{</div><div><span class="" style="white-space:pre">      </span>size_t i;</div><div><span class="" style="white-space:pre">  </span>for (i = 0; i < cache->nr_caches; i++) {</div>
<div><span class="" style="white-space:pre">            </span>struct metablock *mb = mb_at(cache, i);</div><div><span class="" style="white-space:pre">            </span>INIT_HLIST_NODE(&mb->ht_list);</div><div><br></div><div><span class="" style="white-space:pre">             </span>mb->idx = i;</div>
<div><span class="" style="white-space:pre">            </span>mb->dirty_bits = 0;</div><div><span class="" style="white-space:pre">     </span>}</div><div>}</div><div><br></div><div>static void ht_empty_init(struct lc_cache *cache)</div>
<div>{</div><div><span class="" style="white-space:pre">    </span>cache->htsize = cache->nr_caches;</div><div><br></div><div><span class="" style="white-space:pre">   </span>size_t nr_heads = (cache->htsize + 1);</div>
<div><span class="" style="white-space:pre">    </span>struct arr *arr = make_arr(sizeof(struct ht_head), nr_heads);</div><div><br></div><div><span class="" style="white-space:pre">     </span>cache->htable = arr;</div><div><br>
</div><div><span class="" style="white-space:pre">        </span>size_t i;</div><div><span class="" style="white-space:pre">  </span>for (i = 0; i < nr_heads; i++) {</div><div><span class="" style="white-space:pre">                </span>struct ht_head *hd = arr_at(arr, i);</div>
<div><span class="" style="white-space:pre">            </span>INIT_HLIST_HEAD(&hd->ht_list);</div><div><span class="" style="white-space:pre">      </span>}</div><div><br></div><div><span class="" style="white-space:pre"> </span>/*</div>
<div><span class="" style="white-space:pre">    </span> * Our hashtable has one special bucket called null head.</div><div><span class="" style="white-space:pre">  </span> * A metablock is linked to the null head</div><div><span class="" style="white-space:pre">  </span> * if it is not counted in hashtable search.</div>
<div><span class="" style="white-space:pre">    </span> */</div><div><span class="" style="white-space:pre">        </span>cache->null_head = arr_at(cache->htable, cache->htsize);</div><div><br></div><div><span class="" style="white-space:pre"> </span>cache_nr idx;</div>
<div><span class="" style="white-space:pre">    </span>for (idx = 0; idx < cache->nr_caches; idx++) {</div><div><span class="" style="white-space:pre">               </span>struct metablock *mb = mb_at(cache, idx);</div><div><span class="" style="white-space:pre">          </span>hlist_add_head(&mb->ht_list, &cache->null_head->ht_list);</div>
<div><span class="" style="white-space:pre">    </span>}</div><div>}</div><div><br></div><div>static cache_nr ht_hash(struct lc_cache *cache, struct lookup_key *key)</div><div>{</div><div><span class="" style="white-space:pre">   </span>return key->sector % cache->htsize;</div>
<div>}</div><div><br></div><div>static bool mb_hit(struct metablock *mb, struct lookup_key *key)</div><div>{</div><div><span class="" style="white-space:pre">    </span>return (mb->sector == key->sector) && (mb->device_id == key->device_id);</div>
<div>}</div><div><br></div><div>static void ht_del(struct lc_cache *cache, struct metablock *mb)</div><div>{</div><div><span class="" style="white-space:pre">    </span>hlist_del(&mb->ht_list);</div><div><br></div><div>
<span class="" style="white-space:pre">       </span>struct ht_head *null_head = cache->null_head;</div><div><span class="" style="white-space:pre">   </span>hlist_add_head(&mb->ht_list, &null_head->ht_list);</div>
<div>}</div><div><br></div><div>static void ht_register(struct lc_cache *cache, struct ht_head *head,</div><div><span class="" style="white-space:pre">                       </span>struct lookup_key *key, struct metablock *mb)</div><div>{</div>
<div><span class="" style="white-space:pre">    </span>hlist_del(&mb->ht_list);</div><div><span class="" style="white-space:pre">    </span>hlist_add_head(&mb->ht_list, &head->ht_list);</div><div><br></div>
<div><span class="" style="white-space:pre">    </span>mb->device_id = key->device_id;</div><div><span class="" style="white-space:pre">      </span>mb->sector = key->sector;</div><div>};</div><div><br></div><div>static struct metablock *ht_lookup(struct lc_cache *cache,</div>
<div><span class="" style="white-space:pre">                            </span>   struct ht_head *head, struct lookup_key *key)</div><div>{</div><div><span class="" style="white-space:pre">   </span>struct metablock *found = NULL;</div><div><span class="" style="white-space:pre">    </span>struct metablock *mb;</div>
<div><br></div><div>#if LINUX_VERSION_CODE >= KERNEL_VERSION(3, 9, 0)</div><div><span class="" style="white-space:pre">        </span>hlist_for_each_entry(mb, &head->ht_list, ht_list)</div><div>#else</div><div><span class="" style="white-space:pre">       </span>struct hlist_node *pos;</div>
<div><span class="" style="white-space:pre">    </span>hlist_for_each_entry(mb, pos, &head->ht_list, ht_list)</div><div>#endif</div><div><span class="" style="white-space:pre"> </span>{</div><div><span class="" style="white-space:pre">          </span>if (mb_hit(mb, key)) {</div>
<div><span class="" style="white-space:pre">                    </span>found = mb;</div><div><span class="" style="white-space:pre">                        </span>break;</div><div><span class="" style="white-space:pre">             </span>}</div><div><span class="" style="white-space:pre">  </span>}</div>
<div><span class="" style="white-space:pre">    </span>return found;</div><div>}</div><div><br></div><div>static void discard_caches_inseg(struct lc_cache *cache,</div><div><span class="" style="white-space:pre">                              </span> struct segment_header *seg)</div>
<div>{</div><div><span class="" style="white-space:pre">    </span>u8 i;</div><div><span class="" style="white-space:pre">      </span>for (i = 0; i < NR_CACHES_INSEG; i++) {</div><div><span class="" style="white-space:pre">         </span>struct metablock *mb = seg->mb_array + i;</div>
<div><span class="" style="white-space:pre">            </span>ht_del(cache, mb);</div><div><span class="" style="white-space:pre"> </span>}</div><div>}</div><div><br></div><div>static void init_segment_header_array(struct lc_cache *cache)</div>
<div>{</div><div><span class="" style="white-space:pre">    </span>size_t nr_segments = cache->nr_segments;</div><div><br></div><div><span class="" style="white-space:pre">       </span>cache->segment_header_array =</div><div>
<span class="" style="white-space:pre">               </span>make_arr(sizeof(struct segment_header), nr_segments);</div><div><br></div><div><span class="" style="white-space:pre">     </span>size_t segment_idx;</div><div><span class="" style="white-space:pre">        </span>for (segment_idx = 0; segment_idx < nr_segments; segment_idx++) {</div>
<div><span class="" style="white-space:pre">            </span>struct segment_header *seg =</div><div><span class="" style="white-space:pre">                       </span>arr_at(cache->segment_header_array, segment_idx);</div><div><span class="" style="white-space:pre">               </span>seg->start_idx = NR_CACHES_INSEG * segment_idx;</div>
<div><span class="" style="white-space:pre">            </span>seg->start_sector =</div><div><span class="" style="white-space:pre">                     </span>((segment_idx % nr_segments) + 1) *</div><div><span class="" style="white-space:pre">                        </span>(1 << LC_SEGMENTSIZE_ORDER);</div>
<div><br></div><div><span class="" style="white-space:pre">           </span>seg->length = 0;</div><div><br></div><div><span class="" style="white-space:pre">               </span>atomic_set(&seg->nr_inflight_ios, 0);</div><div><br></div>
<div><span class="" style="white-space:pre">            </span>spin_lock_init(&seg->lock);</div><div><br></div><div><span class="" style="white-space:pre">                </span>init_completion(&seg->flush_done);</div><div><span class="" style="white-space:pre">          </span>complete_all(&seg->flush_done);</div>
<div><br></div><div><span class="" style="white-space:pre">           </span>init_completion(&seg->migrate_done);</div><div><span class="" style="white-space:pre">                </span>complete_all(&seg->migrate_done);</div><div>
<span class="" style="white-space:pre">       </span>}</div><div>}</div><div><br></div><div>static struct segment_header *get_segment_header_by_id(struct lc_cache *cache,</div><div><span class="" style="white-space:pre">                                            </span>       size_t segment_id)</div>
<div>{</div><div><span class="" style="white-space:pre">    </span>struct segment_header *r =</div><div><span class="" style="white-space:pre">         </span>arr_at(cache->segment_header_array,</div><div><span class="" style="white-space:pre">             </span>       (segment_id - 1) % cache->nr_segments);</div>
<div><span class="" style="white-space:pre">    </span>return r;</div><div>}</div><div><br></div><div>static u32 calc_segment_lap(struct lc_cache *cache, size_t segment_id)</div><div>{</div><div><span class="" style="white-space:pre">    </span>u32 a = (segment_id - 1) / cache->nr_segments;</div>
<div><span class="" style="white-space:pre">    </span>return a + 1;</div><div>};</div><div><br></div><div>static sector_t calc_mb_start_sector(struct segment_header *seg,</div><div><span class="" style="white-space:pre">                             </span>     cache_nr mb_idx)</div>
<div>{</div><div><span class="" style="white-space:pre">    </span>size_t k = 1 + (mb_idx % NR_CACHES_INSEG);</div><div><span class="" style="white-space:pre"> </span>return seg->start_sector + (k << 3);</div><div>}</div>
<div><br></div><div>static u8 count_dirty_caches_remained(struct segment_header *seg)</div><div>{</div><div><span class="" style="white-space:pre">   </span>u8 count = 0;</div><div><span class="" style="white-space:pre">      </span>u8 i;</div>
<div><span class="" style="white-space:pre">    </span>struct metablock *mb;</div><div><span class="" style="white-space:pre">      </span>for (i = 0; i < seg->length; i++) {</div><div><span class="" style="white-space:pre">          </span>mb = seg->mb_array + i;</div>
<div><span class="" style="white-space:pre">            </span>if (mb->dirty_bits)</div><div><span class="" style="white-space:pre">                     </span>count++;</div><div><span class="" style="white-space:pre">   </span>}</div><div><span class="" style="white-space:pre">  </span>return count;</div>
<div>}</div><div><br></div><div>static void prepare_segment_header_device(</div><div><span class="" style="white-space:pre">          </span>struct segment_header_device *dest,</div><div><span class="" style="white-space:pre">                </span>struct lc_cache *cache, struct segment_header *src)</div>
<div>{</div><div><span class="" style="white-space:pre">    </span>dest->global_id = src->global_id;</div><div><span class="" style="white-space:pre">    </span>dest->length = src->length;</div><div><span class="" style="white-space:pre">  </span>dest->lap = calc_segment_lap(cache, src->global_id);</div>
<div><br></div><div><span class="" style="white-space:pre">   </span>u8 left = src->length - 1;</div><div><span class="" style="white-space:pre">      </span>u8 right = (cache->cursor) % NR_CACHES_INSEG;</div><div><span class="" style="white-space:pre">   </span>BUG_ON(left != right);</div>
<div><br></div><div><span class="" style="white-space:pre">   </span>cache_nr i;</div><div><span class="" style="white-space:pre">        </span>for (i = 0; i < src->length; i++) {</div><div><span class="" style="white-space:pre">          </span>struct metablock *mb = src->mb_array + i;</div>
<div><span class="" style="white-space:pre">            </span>struct metablock_device *mbdev = &dest->mbarr[i];</div><div><span class="" style="white-space:pre">           </span>mbdev->device_id = mb->device_id;</div><div><span class="" style="white-space:pre">            </span>mbdev->sector = mb->sector;</div>
<div><span class="" style="white-space:pre">            </span>mbdev->dirty_bits = mb->dirty_bits;</div><div><span class="" style="white-space:pre">          </span>mbdev->lap = dest->lap;</div><div><span class="" style="white-space:pre">      </span>}</div>
<div>}</div><div><br></div><div>struct flush_context {</div><div><span class="" style="white-space:pre">      </span>struct list_head flush_queue;</div><div><span class="" style="white-space:pre">      </span>struct segment_header *seg;</div>
<div><span class="" style="white-space:pre">    </span>struct writebuffer *wb;</div><div><span class="" style="white-space:pre">    </span>struct bio_list barrier_ios;</div><div>};</div><div><br></div><div>static void flush_proc(struct work_struct *work)</div>
<div>{</div><div><span class="" style="white-space:pre">    </span>unsigned long flags;</div><div><span class="" style="white-space:pre">       </span>struct lc_cache *cache =</div><div><span class="" style="white-space:pre">           </span>container_of(work, struct lc_cache, flush_work);</div>
<div><br></div><div><span class="" style="white-space:pre">   </span>while (true) {</div><div><span class="" style="white-space:pre">             </span>spin_lock_irqsave(&cache->flush_queue_lock, flags);</div><div><span class="" style="white-space:pre">         </span>while (list_empty(&cache->flush_queue)) {</div>
<div><span class="" style="white-space:pre">                    </span>spin_unlock_irqrestore(&cache->flush_queue_lock, flags);</div><div><span class="" style="white-space:pre">                    </span>wait_event_interruptible_timeout(</div><div>
<span class="" style="white-space:pre">                               </span>cache->flush_wait_queue,</div><div><span class="" style="white-space:pre">                                </span>(!list_empty(&cache->flush_queue)),</div><div><span class="" style="white-space:pre">                         </span>msecs_to_jiffies(100));</div>
<div><span class="" style="white-space:pre">                    </span>spin_lock_irqsave(&cache->flush_queue_lock, flags);</div><div><br></div><div><span class="" style="white-space:pre">                        </span>if (cache->on_terminate)</div><div>
<span class="" style="white-space:pre">                               </span>return;</div><div><span class="" style="white-space:pre">            </span>}</div><div><br></div><div><span class="" style="white-space:pre">         </span>/* Pop the first entry */</div>
<div><span class="" style="white-space:pre">            </span>struct flush_context *ctx;</div><div><span class="" style="white-space:pre">         </span>ctx = list_first_entry(</div><div><span class="" style="white-space:pre">                    </span>&cache->flush_queue, struct flush_context, flush_queue);</div>
<div><span class="" style="white-space:pre">            </span>list_del(&ctx->flush_queue);</div><div><span class="" style="white-space:pre">                </span>spin_unlock_irqrestore(&cache->flush_queue_lock, flags);</div><div>
<br></div><div><span class="" style="white-space:pre">              </span>struct segment_header *seg = ctx->seg;</div><div><br></div><div><span class="" style="white-space:pre">         </span>struct dm_io_request io_req = {</div><div><span class="" style="white-space:pre">                    </span>.client = lc_io_client,</div>
<div><span class="" style="white-space:pre">                    </span>.bi_rw = WRITE,</div><div><span class="" style="white-space:pre">                    </span>.notify.fn = NULL,</div><div><span class="" style="white-space:pre">                 </span>.mem.type = DM_IO_KMEM,</div>
<div><span class="" style="white-space:pre">                    </span>.mem.ptr.addr = ctx->wb->data,</div><div><span class="" style="white-space:pre">               </span>};</div><div><span class="" style="white-space:pre">         </span>struct dm_io_region region = {</div>
<div><span class="" style="white-space:pre">                    </span>.bdev = cache->device->bdev,</div><div><span class="" style="white-space:pre">                 </span>.sector = seg->start_sector,</div><div><span class="" style="white-space:pre">                    </span>.count = (seg->length + 1) << 3,</div>
<div><span class="" style="white-space:pre">            </span>};</div><div><span class="" style="white-space:pre">         </span>dm_safe_io_retry(&io_req, &region, 1, false);</div><div><br></div><div><span class="" style="white-space:pre">             </span>cache->last_flushed_segment_id = seg->global_id;</div>
<div><br></div><div><span class="" style="white-space:pre">           </span>complete_all(&seg->flush_done);</div><div><br></div><div><span class="" style="white-space:pre">            </span>complete_all(&ctx->wb->done);</div>
<div><br></div><div><span class="" style="white-space:pre">           </span>if (!bio_list_empty(&ctx->barrier_ios)) {</div><div><span class="" style="white-space:pre">                   </span>blkdev_issue_flush(cache->device->bdev, GFP_NOIO, NULL);</div>
<div><span class="" style="white-space:pre">                    </span>struct bio *bio;</div><div><span class="" style="white-space:pre">                   </span>while ((bio = bio_list_pop(&ctx->barrier_ios)))</div><div><span class="" style="white-space:pre">                             </span>bio_endio(bio, 0);</div>
<div><br></div><div><span class="" style="white-space:pre">                   </span>mod_timer(&cache->barrier_deadline_timer,</div><div><span class="" style="white-space:pre">                           </span>  msecs_to_jiffies(cache->barrier_deadline_ms));</div>
<div><span class="" style="white-space:pre">            </span>}</div><div><br></div><div><span class="" style="white-space:pre">         </span>kfree(ctx);</div><div><span class="" style="white-space:pre">        </span>}</div><div>}</div><div>
<br></div><div>static void prepare_meta_writebuffer(void *writebuffer, struct lc_cache *cache,</div><div><span class="" style="white-space:pre">                                </span>     struct segment_header *seg)</div><div>{</div><div><span class="" style="white-space:pre">   </span>prepare_segment_header_device(writebuffer, cache, seg);</div>
<div>}</div><div><br></div><div>static void queue_flushing(struct lc_cache *cache)</div><div>{</div><div><span class="" style="white-space:pre">  </span>unsigned long flags;</div><div><span class="" style="white-space:pre">       </span>struct segment_header *current_seg = cache->current_seg;</div>
<div><br></div><div><span class="" style="white-space:pre">   </span>size_t n1 = 0;</div><div><span class="" style="white-space:pre">     </span>while (atomic_read(&current_seg->nr_inflight_ios)) {</div><div><span class="" style="white-space:pre">                </span>n1++;</div>
<div><span class="" style="white-space:pre">            </span>if (n1 == 100)</div><div><span class="" style="white-space:pre">                     </span>DMWARN(</div><div><span class="" style="white-space:pre">                    </span>"Too long to wait for current_seg ios to finish.");</div>
<div><span class="" style="white-space:pre">            </span>schedule_timeout_interruptible(msecs_to_jiffies(1));</div><div><span class="" style="white-space:pre">       </span>}</div><div><br></div><div><span class="" style="white-space:pre"> </span>prepare_meta_writebuffer(cache->current_wb->data, cache,</div>
<div><span class="" style="white-space:pre">                            </span> cache->current_seg);</div><div><br></div><div><span class="" style="white-space:pre">  </span>INIT_COMPLETION(current_seg->migrate_done);</div><div><span class="" style="white-space:pre">     </span>INIT_COMPLETION(current_seg->flush_done);</div>
<div><br></div><div><span class="" style="white-space:pre">   </span>struct flush_context *ctx = kmalloc_retry(sizeof(*ctx), GFP_NOIO);</div><div><span class="" style="white-space:pre"> </span>INIT_LIST_HEAD(&ctx->flush_queue);</div>
<div><span class="" style="white-space:pre">    </span>ctx->seg = current_seg;</div><div><span class="" style="white-space:pre"> </span>ctx->wb = cache->current_wb;</div><div><br></div><div><span class="" style="white-space:pre">        </span>bio_list_init(&ctx->barrier_ios);</div>
<div><span class="" style="white-space:pre">    </span>bio_list_merge(&ctx->barrier_ios, &cache->barrier_ios);</div><div><span class="" style="white-space:pre">      </span>bio_list_init(&cache->barrier_ios);</div>
<div><br></div><div><span class="" style="white-space:pre">   </span>spin_lock_irqsave(&cache->flush_queue_lock, flags);</div><div><span class="" style="white-space:pre"> </span>bool empty = list_empty(&cache->flush_queue);</div>
<div><span class="" style="white-space:pre">    </span>list_add_tail(&ctx->flush_queue, &cache->flush_queue);</div><div><span class="" style="white-space:pre">       </span>spin_unlock_irqrestore(&cache->flush_queue_lock, flags);</div>
<div><span class="" style="white-space:pre">    </span>if (empty)</div><div><span class="" style="white-space:pre">         </span>wake_up_interruptible(&cache->flush_wait_queue);</div><div><br></div><div><span class="" style="white-space:pre">   </span>size_t next_id = current_seg->global_id + 1;</div>
<div><span class="" style="white-space:pre">    </span>struct segment_header *new_seg =</div><div><span class="" style="white-space:pre">           </span>get_segment_header_by_id(cache, next_id);</div><div><span class="" style="white-space:pre">  </span>new_seg->global_id = next_id;</div>
<div><br></div><div><span class="" style="white-space:pre">   </span>size_t n2 = 0;</div><div><span class="" style="white-space:pre">     </span>while (atomic_read(&new_seg->nr_inflight_ios)) {</div><div><span class="" style="white-space:pre">            </span>n2++;</div>
<div><span class="" style="white-space:pre">            </span>if (n2 == 100)</div><div><span class="" style="white-space:pre">                     </span>DMWARN(</div><div><span class="" style="white-space:pre">                    </span>"Too long to wait for new_seg ios to finish.");</div>
<div><span class="" style="white-space:pre">            </span>schedule_timeout_interruptible(msecs_to_jiffies(1));</div><div><span class="" style="white-space:pre">       </span>}</div><div><br></div><div><span class="" style="white-space:pre"> </span>u8 nr_new = count_dirty_caches_remained(new_seg);</div>
<div><span class="" style="white-space:pre">    </span>BUG_ON(nr_new);</div><div><br></div><div><span class="" style="white-space:pre">   </span>discard_caches_inseg(cache, new_seg);</div><div><br></div><div><span class="" style="white-space:pre">     </span>/* Set the cursor to the last of the flushed segment. */</div>
<div><span class="" style="white-space:pre">    </span>cache->cursor = current_seg->start_idx + (NR_CACHES_INSEG - 1);</div><div><span class="" style="white-space:pre">      </span>new_seg->length = 0;</div><div><br></div>
<div><span class="" style="white-space:pre">    </span>struct writebuffer *next_wb = cache->wb_pool + (next_id % NR_WB_POOL);</div><div><span class="" style="white-space:pre">  </span>wait_for_completion(&next_wb->done);</div>
<div><span class="" style="white-space:pre">    </span>INIT_COMPLETION(next_wb->done);</div><div><br></div><div><span class="" style="white-space:pre">        </span>cache->current_wb = next_wb;</div><div><br></div><div><span class="" style="white-space:pre">   </span>cache->current_seg = new_seg;</div>
<div>}</div><div><br></div><div>static void migrate_mb(</div><div><span class="" style="white-space:pre">             </span>struct lc_cache *cache, struct segment_header *seg,</div><div><span class="" style="white-space:pre">                </span>struct metablock *mb, u8 dirty_bits, bool thread)</div>
<div>{</div><div><span class="" style="white-space:pre">    </span>struct lc_device *lc = lc_devices[mb->device_id];</div><div><br></div><div><span class="" style="white-space:pre">      </span>if (!dirty_bits)</div><div><span class="" style="white-space:pre">           </span>return;</div>
<div><br></div><div><span class="" style="white-space:pre">   </span>if (dirty_bits == 255) {</div><div><span class="" style="white-space:pre">           </span>void *buf = kmalloc_retry(1 << 12, GFP_NOIO);</div><div><br></div>
<div><span class="" style="white-space:pre">            </span>struct dm_io_request io_req_r = {</div><div><span class="" style="white-space:pre">                  </span>.client = lc_io_client,</div><div><span class="" style="white-space:pre">                    </span>.bi_rw = READ,</div>
<div><span class="" style="white-space:pre">                    </span>.notify.fn = NULL,</div><div><span class="" style="white-space:pre">                 </span>.mem.type = DM_IO_KMEM,</div><div><span class="" style="white-space:pre">                    </span>.mem.ptr.addr = buf,</div>
<div><span class="" style="white-space:pre">            </span>};</div><div><span class="" style="white-space:pre">         </span>struct dm_io_region region_r = {</div><div><span class="" style="white-space:pre">                   </span>.bdev = cache->device->bdev,</div>
<div><span class="" style="white-space:pre">                    </span>.sector = calc_mb_start_sector(seg, mb->idx),</div><div><span class="" style="white-space:pre">                   </span>.count = (1 << 3),</div><div><span class="" style="white-space:pre">           </span>};</div>
<div><span class="" style="white-space:pre">            </span>dm_safe_io_retry(&io_req_r, &region_r, 1, thread);</div><div><br></div><div><span class="" style="white-space:pre">                </span>struct dm_io_request io_req_w = {</div>
<div><span class="" style="white-space:pre">                    </span>.client = lc_io_client,</div><div><span class="" style="white-space:pre">                    </span>.bi_rw = WRITE_FUA,</div><div><span class="" style="white-space:pre">                        </span>.notify.fn = NULL,</div>
<div><span class="" style="white-space:pre">                    </span>.mem.type = DM_IO_KMEM,</div><div><span class="" style="white-space:pre">                    </span>.mem.ptr.addr = buf,</div><div><span class="" style="white-space:pre">               </span>};</div>
<div><span class="" style="white-space:pre">            </span>struct dm_io_region region_w = {</div><div><span class="" style="white-space:pre">                   </span>.bdev = lc->device->bdev,</div><div><span class="" style="white-space:pre">                    </span>.sector = mb->sector,</div>
<div><span class="" style="white-space:pre">                    </span>.count = (1 << 3),</div><div><span class="" style="white-space:pre">           </span>};</div><div><span class="" style="white-space:pre">         </span>dm_safe_io_retry(&io_req_w, &region_w, 1, thread);</div>
<div><br></div><div><span class="" style="white-space:pre">           </span>kfree(buf);</div><div><span class="" style="white-space:pre">        </span>} else {</div><div><span class="" style="white-space:pre">           </span>void *buf = kmalloc_retry(1 << SECTOR_SHIFT, GFP_NOIO);</div>
<div><span class="" style="white-space:pre">            </span>size_t i;</div><div><span class="" style="white-space:pre">          </span>for (i = 0; i < 8; i++) {</div><div><span class="" style="white-space:pre">                       </span>bool bit_on = dirty_bits & (1 << i);</div>
<div><span class="" style="white-space:pre">                    </span>if (!bit_on)</div><div><span class="" style="white-space:pre">                               </span>continue;</div><div><br></div><div><span class="" style="white-space:pre">                 </span>struct dm_io_request io_req_r = {</div>
<div><span class="" style="white-space:pre">                            </span>.client = lc_io_client,</div><div><span class="" style="white-space:pre">                            </span>.bi_rw = READ,</div><div><span class="" style="white-space:pre">                             </span>.notify.fn = NULL,</div>
<div><span class="" style="white-space:pre">                            </span>.mem.type = DM_IO_KMEM,</div><div><span class="" style="white-space:pre">                            </span>.mem.ptr.addr = buf,</div><div><span class="" style="white-space:pre">                       </span>};</div>
<div><span class="" style="white-space:pre">                    </span>/* A tmp variable just to avoid 80 cols rule */</div><div><span class="" style="white-space:pre">                    </span>sector_t src = calc_mb_start_sector(seg, mb->idx) + i;</div>
<div><span class="" style="white-space:pre">                    </span>struct dm_io_region region_r = {</div><div><span class="" style="white-space:pre">                           </span>.bdev = cache->device->bdev,</div><div><span class="" style="white-space:pre">                         </span>.sector = src,</div>
<div><span class="" style="white-space:pre">                            </span>.count = 1,</div><div><span class="" style="white-space:pre">                        </span>};</div><div><span class="" style="white-space:pre">                 </span>dm_safe_io_retry(&io_req_r, &region_r, 1, thread);</div>
<div><br></div><div><span class="" style="white-space:pre">                   </span>struct dm_io_request io_req_w = {</div><div><span class="" style="white-space:pre">                          </span>.client = lc_io_client,</div><div><span class="" style="white-space:pre">                            </span>.bi_rw = WRITE,</div>
<div><span class="" style="white-space:pre">                            </span>.notify.fn = NULL,</div><div><span class="" style="white-space:pre">                         </span>.mem.type = DM_IO_KMEM,</div><div><span class="" style="white-space:pre">                            </span>.mem.ptr.addr = buf,</div>
<div><span class="" style="white-space:pre">                    </span>};</div><div><span class="" style="white-space:pre">                 </span>struct dm_io_region region_w = {</div><div><span class="" style="white-space:pre">                           </span>.bdev = lc->device->bdev,</div>
<div><span class="" style="white-space:pre">                            </span>.sector = mb->sector + 1 * i,</div><div><span class="" style="white-space:pre">                           </span>.count = 1,</div><div><span class="" style="white-space:pre">                        </span>};</div>
<div><span class="" style="white-space:pre">                    </span>dm_safe_io_retry(&io_req_w, &region_w, 1, thread);</div><div><span class="" style="white-space:pre">         </span>}</div><div><span class="" style="white-space:pre">          </span>kfree(buf);</div>
<div><span class="" style="white-space:pre">    </span>}</div><div>}</div><div><br></div><div>static void migrate_endio(unsigned long error, void *__context)</div><div>{</div><div><span class="" style="white-space:pre">   </span>struct lc_cache *cache = __context;</div>
<div><br></div><div><span class="" style="white-space:pre">   </span>if (error)</div><div><span class="" style="white-space:pre">         </span>atomic_inc(&cache->migrate_fail_count);</div><div><br></div><div><span class="" style="white-space:pre">    </span>if (atomic_dec_and_test(&cache->migrate_io_count))</div>
<div><span class="" style="white-space:pre">            </span>wake_up_interruptible(&cache->migrate_wait_queue);</div><div>}</div><div><br></div><div>static void migrate_whole_segment(struct lc_cache *cache,</div><div><span class="" style="white-space:pre">                         </span>  struct segment_header *seg)</div>
<div>{</div><div><span class="" style="white-space:pre">    </span>struct dm_io_request io_req_r = {</div><div><span class="" style="white-space:pre">          </span>.client = lc_io_client,</div><div><span class="" style="white-space:pre">            </span>.bi_rw = READ,</div>
<div><span class="" style="white-space:pre">            </span>.notify.fn = NULL,</div><div><span class="" style="white-space:pre">         </span>.mem.type = DM_IO_KMEM,</div><div><span class="" style="white-space:pre">            </span>.mem.ptr.addr = cache->migrate_buffer,</div>
<div><span class="" style="white-space:pre">    </span>};</div><div><span class="" style="white-space:pre"> </span>struct dm_io_region region_r = {</div><div><span class="" style="white-space:pre">           </span>.bdev = cache->device->bdev,</div>
<div><span class="" style="white-space:pre">            </span>.sector = seg->start_sector,</div><div><span class="" style="white-space:pre">            </span>.count = (seg->length + 1) << 3,</div><div><span class="" style="white-space:pre">  </span>};</div>
<div><span class="" style="white-space:pre">    </span>dm_safe_io_retry(&io_req_r, &region_r, 1, false);</div><div><br></div><div>migrate_write:</div><div><span class="" style="white-space:pre">    </span>;</div><div><span class="" style="white-space:pre">  </span>unsigned long flags;</div>
<div><span class="" style="white-space:pre">    </span>struct metablock *mb;</div><div><span class="" style="white-space:pre">      </span>u8 i, j;</div><div><br></div><div><span class="" style="white-space:pre">  </span>atomic_set(&cache->migrate_io_count, 0);</div>
<div><span class="" style="white-space:pre">    </span>atomic_set(&cache->migrate_fail_count, 0);</div><div><br></div><div><span class="" style="white-space:pre"> </span>for (i = 0; i < LC_NR_SLOTS; i++)</div><div>
<span class="" style="white-space:pre">               </span>*(cache->migrate_dests + i) = false;</div><div><br></div><div><span class="" style="white-space:pre">   </span>for (i = 0; i < seg->length; i++) {</div><div><span class="" style="white-space:pre">          </span>mb = seg->mb_array + i;</div>
<div><span class="" style="white-space:pre">            </span>*(cache->dirtiness_snapshot + i) =</div><div><span class="" style="white-space:pre">                      </span>atomic_read_mb_dirtiness(seg, mb);</div><div><span class="" style="white-space:pre"> </span>}</div>
<div><br></div><div><span class="" style="white-space:pre">   </span>for (i = 0; i < seg->length; i++) {</div><div><span class="" style="white-space:pre">          </span>mb = seg->mb_array + i;</div><div><br></div><div><span class="" style="white-space:pre">                </span>u8 dirty_bits = *(cache->dirtiness_snapshot + i);</div>
<div><br></div><div><span class="" style="white-space:pre">           </span>if (!dirty_bits)</div><div><span class="" style="white-space:pre">                   </span>continue;</div><div><br></div><div><span class="" style="white-space:pre">         </span>*(cache->migrate_dests + mb->device_id) = true;</div>
<div><br></div><div><span class="" style="white-space:pre">           </span>if (dirty_bits == 255) {</div><div><span class="" style="white-space:pre">                   </span>atomic_inc(&cache->migrate_io_count);</div><div><span class="" style="white-space:pre">               </span>} else {</div>
<div><span class="" style="white-space:pre">                    </span>for (j = 0; j < 8; j++) {</div><div><span class="" style="white-space:pre">                               </span>if (dirty_bits & (1 << j))</div><div><span class="" style="white-space:pre">                                       </span>atomic_inc(&cache->migrate_io_count);</div>
<div><span class="" style="white-space:pre">                    </span>}</div><div><span class="" style="white-space:pre">          </span>}</div><div><span class="" style="white-space:pre">  </span>}</div><div><br></div><div><span class="" style="white-space:pre"> </span>struct lc_device *lc;</div>
<div><span class="" style="white-space:pre">    </span>for (i = 0; i < seg->length; i++) {</div><div><span class="" style="white-space:pre">          </span>mb = seg->mb_array + i;</div><div><br></div><div><span class="" style="white-space:pre">                </span>lc = lc_devices[mb->device_id];</div>
<div><span class="" style="white-space:pre">            </span>u8 dirty_bits = *(cache->dirtiness_snapshot + i);</div><div><br></div><div><span class="" style="white-space:pre">              </span>if (!dirty_bits)</div><div><span class="" style="white-space:pre">                   </span>continue;</div>
<div><br></div><div><span class="" style="white-space:pre">           </span>unsigned long diff = ((1 + i) << 3) << SECTOR_SHIFT;</div><div><span class="" style="white-space:pre">           </span>void *base = cache->migrate_buffer + diff;</div>
<div><br></div><div><span class="" style="white-space:pre">           </span>void *addr;</div><div><span class="" style="white-space:pre">                </span>if (dirty_bits == 255) {</div><div><span class="" style="white-space:pre">                   </span>addr = base;</div>
<div><span class="" style="white-space:pre">                    </span>struct dm_io_request io_req_w = {</div><div><span class="" style="white-space:pre">                          </span>.client = lc_io_client,</div><div><span class="" style="white-space:pre">                            </span>.bi_rw = WRITE,</div>
<div><span class="" style="white-space:pre">                            </span>.notify.fn = migrate_endio,</div><div><span class="" style="white-space:pre">                                </span>.notify.context = cache,</div><div><span class="" style="white-space:pre">                           </span>.mem.type = DM_IO_KMEM,</div>
<div><span class="" style="white-space:pre">                            </span>.mem.ptr.addr = addr,</div><div><span class="" style="white-space:pre">                      </span>};</div><div><span class="" style="white-space:pre">                 </span>struct dm_io_region region_w = {</div>
<div><span class="" style="white-space:pre">                            </span>.bdev = lc->device->bdev,</div><div><span class="" style="white-space:pre">                            </span>.sector = mb->sector,</div><div><span class="" style="white-space:pre">                           </span>.count = (1 << 3),</div>
<div><span class="" style="white-space:pre">                    </span>};</div><div><span class="" style="white-space:pre">                 </span>dm_safe_io_retry(&io_req_w, &region_w, 1, false);</div><div><span class="" style="white-space:pre">          </span>} else {</div>
<div><span class="" style="white-space:pre">                    </span>for (j = 0; j < 8; j++) {</div><div><span class="" style="white-space:pre">                               </span>bool b = dirty_bits & (1 << j);</div><div><span class="" style="white-space:pre">                          </span>if (!b)</div>
<div><span class="" style="white-space:pre">                                    </span>continue;</div><div><br></div><div><span class="" style="white-space:pre">                         </span>addr = base + (j << SECTOR_SHIFT);</div><div><span class="" style="white-space:pre">                           </span>struct dm_io_request io_req_w = {</div>
<div><span class="" style="white-space:pre">                                    </span>.client = lc_io_client,</div><div><span class="" style="white-space:pre">                                    </span>.bi_rw = WRITE,</div><div><span class="" style="white-space:pre">                                    </span>.notify.fn = migrate_endio,</div>
<div><span class="" style="white-space:pre">                                    </span>.notify.context = cache,</div><div><span class="" style="white-space:pre">                                   </span>.mem.type = DM_IO_KMEM,</div><div><span class="" style="white-space:pre">                                    </span>.mem.ptr.addr = addr,</div>
<div><span class="" style="white-space:pre">                            </span>};</div><div><span class="" style="white-space:pre">                         </span>struct dm_io_region region_w = {</div><div><span class="" style="white-space:pre">                                   </span>.bdev = lc->device->bdev,</div>
<div><span class="" style="white-space:pre">                                    </span>.sector = mb->sector + j,</div><div><span class="" style="white-space:pre">                                       </span>.count = 1,</div><div><span class="" style="white-space:pre">                                </span>};</div>
<div><span class="" style="white-space:pre">                            </span>dm_safe_io_retry(</div><div><span class="" style="white-space:pre">                                  </span>&io_req_w, &region_w, 1, false);</div><div><span class="" style="white-space:pre">                   </span>}</div>
<div><span class="" style="white-space:pre">            </span>}</div><div><span class="" style="white-space:pre">  </span>}</div><div><br></div><div><span class="" style="white-space:pre"> </span>wait_event_interruptible(cache->migrate_wait_queue,</div>
<div><span class="" style="white-space:pre">                            </span> (atomic_read(&cache->migrate_io_count) == 0));</div><div><br></div><div><span class="" style="white-space:pre">    </span>if (atomic_read(&cache->migrate_fail_count)) {</div>
<div><span class="" style="white-space:pre">            </span>DMERR("migrate failed. %u writebacks failed. redo.",</div><div><span class="" style="white-space:pre">                             </span>atomic_read(&cache->migrate_fail_count));</div>
<div><span class="" style="white-space:pre">            </span>goto migrate_write;</div><div><span class="" style="white-space:pre">        </span>}</div><div><br></div><div><span class="" style="white-space:pre"> </span>BUG_ON(atomic_read(&cache->migrate_io_count));</div>
<div><br></div><div><span class="" style="white-space:pre">   </span>for (i = 0; i < seg->length; i++) {</div><div><span class="" style="white-space:pre">          </span>mb = seg->mb_array + i;</div><div><br></div><div><span class="" style="white-space:pre">                </span>bool b = false;</div>
<div><span class="" style="white-space:pre">            </span>lockseg(seg, flags);</div><div><span class="" style="white-space:pre">               </span>if (mb->dirty_bits) {</div><div><span class="" style="white-space:pre">                   </span>mb->dirty_bits = 0;</div>
<div><span class="" style="white-space:pre">                    </span>b = true;</div><div><span class="" style="white-space:pre">          </span>}</div><div><span class="" style="white-space:pre">          </span>unlockseg(seg, flags);</div><div><br></div>
<div><span class="" style="white-space:pre">            </span>if (b)</div><div><span class="" style="white-space:pre">                     </span>dec_nr_dirty_caches(mb->device_id);</div><div><span class="" style="white-space:pre">     </span>}</div>
<div><br></div><div><span class="" style="white-space:pre">   </span>for (i = 1; i < LC_NR_SLOTS; i++) {</div><div><span class="" style="white-space:pre">             </span>bool b = *(cache->migrate_dests + i);</div><div><span class="" style="white-space:pre">           </span>if (!b)</div>
<div><span class="" style="white-space:pre">                    </span>continue;</div><div><br></div><div><span class="" style="white-space:pre">         </span>lc = lc_devices[i];</div><div><span class="" style="white-space:pre">                </span>blkdev_issue_flush(lc->device->bdev, GFP_NOIO, NULL);</div>
<div><span class="" style="white-space:pre">    </span>}</div><div><br></div><div><span class="" style="white-space:pre"> </span>/*</div><div><span class="" style="white-space:pre"> </span> * Discarding the migrated regions</div>
<div><span class="" style="white-space:pre">    </span> * can avoid unnecessary wear amplifier in the future.</div><div><span class="" style="white-space:pre">     </span> * But note that we should not discard</div><div><span class="" style="white-space:pre">     </span> * the metablock region because</div>
<div><span class="" style="white-space:pre">    </span> * it varies across flash devices whether to ensure</div><div><span class="" style="white-space:pre">        </span> * the discarded block returns certain value</div><div><span class="" style="white-space:pre">       </span> * and discarding the metablock may lead to</div>
<div><span class="" style="white-space:pre">    </span> * integrity collapsion.</div><div><span class="" style="white-space:pre">   </span> */</div><div><span class="" style="white-space:pre">        </span>blkdev_issue_discard(</div>
<div><span class="" style="white-space:pre">            </span>cache->device->bdev,</div><div><span class="" style="white-space:pre">         </span>seg->start_sector + (1 << 3),</div><div><span class="" style="white-space:pre">             </span>seg->length << 3,</div>
<div><span class="" style="white-space:pre">            </span>GFP_NOIO, 0);</div><div>}</div><div><br></div><div>static void migrate_proc(struct work_struct *work)</div><div>{</div><div><span class="" style="white-space:pre">    </span>struct lc_cache *cache =</div>
<div><span class="" style="white-space:pre">            </span>container_of(work, struct lc_cache, migrate_work);</div><div><br></div><div><span class="" style="white-space:pre">        </span>while (true) {</div><div><span class="" style="white-space:pre">             </span>if (cache->on_terminate)</div>
<div><span class="" style="white-space:pre">                    </span>return;</div><div><br></div><div><span class="" style="white-space:pre">           </span>/*</div><div><span class="" style="white-space:pre">         </span> * reserving_id > 0 means</div>
<div><span class="" style="white-space:pre">            </span> * that migration is immediate.</div><div><span class="" style="white-space:pre">            </span> */</div><div><span class="" style="white-space:pre">                </span>bool allow_migrate =</div>
<div><span class="" style="white-space:pre">                    </span>cache->reserving_segment_id || cache->allow_migrate;</div><div><br></div><div><span class="" style="white-space:pre">                </span>if (!allow_migrate) {</div><div><span class="" style="white-space:pre">                      </span>schedule_timeout_interruptible(msecs_to_jiffies(1000));</div>
<div><span class="" style="white-space:pre">                    </span>continue;</div><div><span class="" style="white-space:pre">          </span>}</div><div><br></div><div><span class="" style="white-space:pre">         </span>bool need_migrate =</div>
<div><span class="" style="white-space:pre">                    </span>(cache->last_migrated_segment_id <</div><div><span class="" style="white-space:pre">                   </span> cache->last_flushed_segment_id);</div><div><br></div><div><span class="" style="white-space:pre">              </span>if (!need_migrate) {</div>
<div><span class="" style="white-space:pre">                    </span>schedule_timeout_interruptible(msecs_to_jiffies(1000));</div><div><span class="" style="white-space:pre">                    </span>continue;</div><div><span class="" style="white-space:pre">          </span>}</div>
<div><br></div><div><span class="" style="white-space:pre">           </span>struct segment_header *seg =</div><div><span class="" style="white-space:pre">                       </span>get_segment_header_by_id(cache,</div><div><span class="" style="white-space:pre">                            </span>cache->last_migrated_segment_id + 1);</div>
<div><br></div><div><span class="" style="white-space:pre">           </span>migrate_whole_segment(cache, seg);</div><div><br></div><div><span class="" style="white-space:pre">                </span>/*</div><div><span class="" style="white-space:pre">         </span> * (Locking)</div>
<div><span class="" style="white-space:pre">            </span> * Only this line alter last_migrate_segment_id in runtime.</div><div><span class="" style="white-space:pre">                </span> */</div><div><span class="" style="white-space:pre">                </span>cache->last_migrated_segment_id++;</div>
<div><br></div><div><span class="" style="white-space:pre">           </span>complete_all(&seg->migrate_done);</div><div><span class="" style="white-space:pre">   </span>}</div><div>}</div><div><br></div><div>static void wait_for_migration(struct lc_cache *cache, size_t id)</div>
<div>{</div><div><span class="" style="white-space:pre">    </span>struct segment_header *seg = get_segment_header_by_id(cache, id);</div><div><br></div><div><span class="" style="white-space:pre"> </span>cache->reserving_segment_id = id;</div>
<div><span class="" style="white-space:pre">    </span>wait_for_completion(&seg->migrate_done);</div><div><span class="" style="white-space:pre">    </span>cache->reserving_segment_id = 0;</div><div>}</div><div><br></div>
<div>struct superblock_device {</div><div><span class="" style="white-space:pre">   </span>size_t last_migrated_segment_id;</div><div>} __packed;</div><div><br></div><div>static void commit_super_block(struct lc_cache *cache)</div>
<div>{</div><div><span class="" style="white-space:pre">    </span>struct superblock_device o;</div><div><br></div><div><span class="" style="white-space:pre">       </span>o.last_migrated_segment_id = cache->last_migrated_segment_id;</div>
<div><br></div><div><span class="" style="white-space:pre">   </span>void *buf = kmalloc_retry(1 << SECTOR_SHIFT, GFP_NOIO);</div><div><span class="" style="white-space:pre">      </span>memcpy(buf, &o, sizeof(o));</div>
<div><br></div><div><span class="" style="white-space:pre">   </span>struct dm_io_request io_req = {</div><div><span class="" style="white-space:pre">            </span>.client = lc_io_client,</div><div><span class="" style="white-space:pre">            </span>.bi_rw = WRITE_FUA,</div>
<div><span class="" style="white-space:pre">            </span>.notify.fn = NULL,</div><div><span class="" style="white-space:pre">         </span>.mem.type = DM_IO_KMEM,</div><div><span class="" style="white-space:pre">            </span>.mem.ptr.addr = buf,</div>
<div><span class="" style="white-space:pre">    </span>};</div><div><span class="" style="white-space:pre"> </span>struct dm_io_region region = {</div><div><span class="" style="white-space:pre">             </span>.bdev = cache->device->bdev,</div>
<div><span class="" style="white-space:pre">            </span>.sector = 0,</div><div><span class="" style="white-space:pre">               </span>.count = 1,</div><div><span class="" style="white-space:pre">        </span>};</div><div><span class="" style="white-space:pre"> </span>dm_safe_io_retry(&io_req, &region, 1, true);</div>
<div><span class="" style="white-space:pre">    </span>kfree(buf);</div><div>}</div><div><br></div><div>static void read_superblock_device(struct superblock_device *dest,</div><div><span class="" style="white-space:pre">                              </span>   struct lc_cache *cache)</div>
<div>{</div><div><span class="" style="white-space:pre">    </span>void *buf = kmalloc(1 << SECTOR_SHIFT, GFP_KERNEL);</div><div><span class="" style="white-space:pre">  </span>struct dm_io_request io_req = {</div><div>
<span class="" style="white-space:pre">               </span>.client = lc_io_client,</div><div><span class="" style="white-space:pre">            </span>.bi_rw = READ,</div><div><span class="" style="white-space:pre">             </span>.notify.fn = NULL,</div>
<div><span class="" style="white-space:pre">            </span>.mem.type = DM_IO_KMEM,</div><div><span class="" style="white-space:pre">            </span>.mem.ptr.addr = buf,</div><div><span class="" style="white-space:pre">       </span>};</div>
<div><span class="" style="white-space:pre">    </span>struct dm_io_region region = {</div><div><span class="" style="white-space:pre">             </span>.bdev = cache->device->bdev,</div><div><span class="" style="white-space:pre">         </span>.sector = 0,</div>
<div><span class="" style="white-space:pre">            </span>.count = 1,</div><div><span class="" style="white-space:pre">        </span>};</div><div><span class="" style="white-space:pre"> </span>dm_safe_io_retry(&io_req, &region, 1, true);</div>
<div><span class="" style="white-space:pre">    </span>memcpy(dest, buf, sizeof(*dest));</div><div><span class="" style="white-space:pre">  </span>kfree(buf);</div><div>}</div><div><br></div><div>static sector_t calc_segment_header_start(size_t segment_idx)</div>
<div>{</div><div><span class="" style="white-space:pre">    </span>return (1 << LC_SEGMENTSIZE_ORDER) * (segment_idx + 1);</div><div>}</div><div><br></div><div>static void read_segment_header_device(</div><div><span class="" style="white-space:pre">               </span>struct segment_header_device *dest,</div>
<div><span class="" style="white-space:pre">            </span>struct lc_cache *cache, size_t segment_idx)</div><div>{</div><div><span class="" style="white-space:pre">        </span>void *buf = kmalloc(1 << 12, GFP_KERNEL);</div><div>
<span class="" style="white-space:pre">       </span>struct dm_io_request io_req = {</div><div><span class="" style="white-space:pre">            </span>.client = lc_io_client,</div><div><span class="" style="white-space:pre">            </span>.bi_rw = READ,</div>
<div><span class="" style="white-space:pre">            </span>.notify.fn = NULL,</div><div><span class="" style="white-space:pre">         </span>.mem.type = DM_IO_KMEM,</div><div><span class="" style="white-space:pre">            </span>.mem.ptr.addr = buf,</div>
<div><span class="" style="white-space:pre">    </span>};</div><div><span class="" style="white-space:pre"> </span>struct dm_io_region region = {</div><div><span class="" style="white-space:pre">             </span>.bdev = cache->device->bdev,</div>
<div><span class="" style="white-space:pre">            </span>.sector = calc_segment_header_start(segment_idx),</div><div><span class="" style="white-space:pre">          </span>.count = (1 << 3),</div><div><span class="" style="white-space:pre">   </span>};</div>
<div><span class="" style="white-space:pre">    </span>dm_safe_io_retry(&io_req, &region, 1, true);</div><div><span class="" style="white-space:pre">       </span>memcpy(dest, buf, sizeof(*dest));</div><div><span class="" style="white-space:pre">  </span>kfree(buf);</div>
<div>}</div><div><br></div><div>static void update_by_segment_header_device(struct lc_cache *cache,</div><div><span class="" style="white-space:pre">                                 </span>    struct segment_header_device *src)</div><div>{</div><div>
<span class="" style="white-space:pre">       </span>struct segment_header *seg =</div><div><span class="" style="white-space:pre">               </span>get_segment_header_by_id(cache, src->global_id);</div><div><span class="" style="white-space:pre">        </span>seg->length = src->length;</div>
<div><br></div><div><span class="" style="white-space:pre">   </span>INIT_COMPLETION(seg->migrate_done);</div><div><br></div><div><span class="" style="white-space:pre">    </span>cache_nr i;</div><div><span class="" style="white-space:pre">        </span>for (i = 0 ; i < src->length; i++) {</div>
<div><span class="" style="white-space:pre">            </span>struct metablock *mb = seg->mb_array + i;</div><div><br></div><div><span class="" style="white-space:pre">              </span>struct metablock_device *mbdev = &src->mbarr[i];</div>
<div><br></div><div><span class="" style="white-space:pre">           </span>if (!mbdev->dirty_bits)</div><div><span class="" style="white-space:pre">                 </span>continue;</div><div><br></div><div><span class="" style="white-space:pre">         </span>mb->sector = mbdev->sector;</div>
<div><span class="" style="white-space:pre">            </span>mb->device_id = mbdev->device_id;</div><div><span class="" style="white-space:pre">            </span>mb->dirty_bits = mbdev->dirty_bits;</div><div><br></div><div><span class="" style="white-space:pre">         </span>inc_nr_dirty_caches(mb->device_id);</div>
<div><br></div><div><span class="" style="white-space:pre">           </span>struct lookup_key key = {</div><div><span class="" style="white-space:pre">                  </span>.device_id = mb->device_id,</div><div><span class="" style="white-space:pre">                     </span>.sector = mb->sector,</div>
<div><span class="" style="white-space:pre">            </span>};</div><div><br></div><div><span class="" style="white-space:pre">                </span>cache_nr k = ht_hash(cache, &key);</div><div><span class="" style="white-space:pre">             </span>struct ht_head *head = arr_at(cache->htable, k);</div>
<div><br></div><div><span class="" style="white-space:pre">           </span>struct metablock *found = ht_lookup(cache, head, &key);</div><div><span class="" style="white-space:pre">                </span>if (found)</div><div><span class="" style="white-space:pre">                 </span>ht_del(cache, found);</div>
<div><span class="" style="white-space:pre">            </span>ht_register(cache, head, &key, mb);</div><div><span class="" style="white-space:pre">    </span>}</div><div>}</div><div><br></div><div>static bool checkup_atomicity(struct segment_header_device *header)</div>
<div>{</div><div><span class="" style="white-space:pre">    </span>size_t i;</div><div><span class="" style="white-space:pre">  </span>struct metablock_device *o;</div><div><span class="" style="white-space:pre">        </span>for (i = 0; i < header->length; i++) {</div>
<div><span class="" style="white-space:pre">            </span>o = header->mbarr + i;</div><div><span class="" style="white-space:pre">          </span>if (o->lap != header->lap)</div><div><span class="" style="white-space:pre">                   </span>return false;</div>
<div><span class="" style="white-space:pre">    </span>}</div><div><span class="" style="white-space:pre">  </span>return true;</div><div>}</div><div><br></div><div><br></div><div>static void recover_cache(struct lc_cache *cache)</div>
<div>{</div><div><span class="" style="white-space:pre">    </span>struct superblock_device sup;</div><div><span class="" style="white-space:pre">      </span>read_superblock_device(&sup, cache);</div><div><br></div><div><span class="" style="white-space:pre">  </span>size_t i;</div>
<div><span class="" style="white-space:pre">    </span>size_t nr_segments = cache->nr_segments;</div><div><br></div><div><span class="" style="white-space:pre">       </span>struct segment_header_device *header =</div><div><span class="" style="white-space:pre">             </span>kmalloc(sizeof(*header), GFP_KERNEL);</div>
<div><br></div><div><span class="" style="white-space:pre">   </span>/*</div><div><span class="" style="white-space:pre"> </span> * Finding the oldest, non-zero id and its index.</div><div><span class="" style="white-space:pre">  </span> */</div>
<div><br></div><div><span class="" style="white-space:pre">   </span>size_t max_id = SZ_MAX; /* This global_id is forbidden. */</div><div><span class="" style="white-space:pre"> </span>size_t oldest_id = max_id;</div><div><span class="" style="white-space:pre"> </span>size_t oldest_idx = 0;</div>
<div><span class="" style="white-space:pre">    </span>for (i = 0; i < nr_segments; i++) {</div><div><span class="" style="white-space:pre">             </span>read_segment_header_device(header, cache, i);</div><div><br></div><div><span class="" style="white-space:pre">             </span>if (header->global_id < 1)</div>
<div><span class="" style="white-space:pre">                    </span>continue;</div><div><br></div><div><span class="" style="white-space:pre">         </span>if (header->global_id < oldest_id) {</div><div><span class="" style="white-space:pre">                 </span>oldest_idx = i;</div>
<div><span class="" style="white-space:pre">                    </span>oldest_id = header->global_id;</div><div><span class="" style="white-space:pre">          </span>}</div><div><span class="" style="white-space:pre">  </span>}</div><div><br>
</div><div><span class="" style="white-space:pre">        </span>size_t last_flushed_id = 0;</div><div><br></div><div><span class="" style="white-space:pre">       </span>/*</div><div><span class="" style="white-space:pre"> </span> * This is an invariant.</div>
<div><span class="" style="white-space:pre">    </span> * We always start from the segment</div><div><span class="" style="white-space:pre">        </span> * that is right after the last_flush_id.</div><div><span class="" style="white-space:pre">  </span> */</div>
<div><span class="" style="white-space:pre">    </span>size_t init_segment_id = last_flushed_id + 1;</div><div><br></div><div><span class="" style="white-space:pre">     </span>/*</div><div><span class="" style="white-space:pre"> </span> * If no segment was flushed</div>
<div><span class="" style="white-space:pre">    </span> * then there is nothing to recover.</div><div><span class="" style="white-space:pre">       </span> */</div><div><span class="" style="white-space:pre">        </span>if (oldest_id == max_id)</div>
<div><span class="" style="white-space:pre">            </span>goto setup_init_segment;</div><div><br></div><div><span class="" style="white-space:pre">  </span>/*</div><div><span class="" style="white-space:pre"> </span> * What we have to do in the next loop is to</div>
<div><span class="" style="white-space:pre">    </span> * reincarnate the segments that are</div><div><span class="" style="white-space:pre">       </span> * flushed but yet not migrated.</div><div><span class="" style="white-space:pre">   </span> */</div>
<div><br></div><div><span class="" style="white-space:pre">   </span>/*</div><div><span class="" style="white-space:pre"> </span> * Example:</div><div><span class="" style="white-space:pre">        </span> * There are only 5 segments.</div>
<div><span class="" style="white-space:pre">    </span> * The segments we will consider is of id k+2 and k+3.</div><div><span class="" style="white-space:pre">     </span> *</div><div><span class="" style="white-space:pre"> </span> * id: [   k+3   ][    k+4   ][     k     ][   k+1    ][   K+2   ]</div>
<div><span class="" style="white-space:pre">    </span> *       flushed    init_seg    last         migrated    flushed</div><div><span class="" style="white-space:pre">   </span> *                              _migrated</div>
<div><span class="" style="white-space:pre">    </span> */</div><div><span class="" style="white-space:pre">        </span>size_t j;</div><div><span class="" style="white-space:pre">  </span>for (i = oldest_idx; i < (nr_segments + oldest_idx); i++) {</div>
<div><span class="" style="white-space:pre">            </span>j = i % nr_segments;</div><div><span class="" style="white-space:pre">               </span>read_segment_header_device(header, cache, j);</div><div><br></div><div><span class="" style="white-space:pre">             </span>/*</div>
<div><span class="" style="white-space:pre">            </span> * global_id must uniformly increase from 1.</div><div><span class="" style="white-space:pre">               </span> * Since last_flush_id is 0 at first,</div><div><span class="" style="white-space:pre">              </span> * if global_id is 0,</div>
<div><span class="" style="white-space:pre">            </span> * we consider this and the subsequents</div><div><span class="" style="white-space:pre">            </span> * are all invalid.</div><div><span class="" style="white-space:pre">                </span> */</div>
<div><span class="" style="white-space:pre">            </span>if (header->global_id <= last_flushed_id)</div><div><span class="" style="white-space:pre">                    </span>break;</div><div><br></div><div><span class="" style="white-space:pre">            </span>if (!checkup_atomicity(header))</div>
<div><span class="" style="white-space:pre">                    </span>break;</div><div><br></div><div><span class="" style="white-space:pre">            </span>/*</div><div><span class="" style="white-space:pre">         </span> * Now the header is proven valid.</div>
<div><span class="" style="white-space:pre">            </span> */</div><div><br></div><div><span class="" style="white-space:pre">               </span>last_flushed_id = header->global_id;</div><div><span class="" style="white-space:pre">            </span>init_segment_id = last_flushed_id + 1;</div>
<div><br></div><div><span class="" style="white-space:pre">           </span>/*</div><div><span class="" style="white-space:pre">         </span> * If the data is already on the backing store,</div><div><span class="" style="white-space:pre">            </span> * we ignore the segment.</div>
<div><span class="" style="white-space:pre">            </span> */</div><div><span class="" style="white-space:pre">                </span>if (header->global_id <= sup.last_migrated_segment_id)</div><div><span class="" style="white-space:pre">                       </span>continue;</div>
<div><br></div><div><span class="" style="white-space:pre">           </span>/*</div><div><span class="" style="white-space:pre">         </span> * Only those to be migrated are counted in.</div><div><span class="" style="white-space:pre">               </span> * These segments will not be available until migrated.</div>
<div><span class="" style="white-space:pre">            </span> */</div><div><span class="" style="white-space:pre">                </span>update_by_segment_header_device(cache, header);</div><div><span class="" style="white-space:pre">    </span>}</div>
<div><br></div><div>setup_init_segment:</div><div><span class="" style="white-space:pre"> </span>kfree(header);</div><div><br></div><div><span class="" style="white-space:pre">    </span>struct segment_header *seg =</div><div>
<span class="" style="white-space:pre">               </span>get_segment_header_by_id(cache, init_segment_id);</div><div><span class="" style="white-space:pre">  </span>seg->global_id = init_segment_id;</div><div><span class="" style="white-space:pre">       </span>atomic_set(&seg->nr_inflight_ios, 0);</div>
<div><br></div><div><span class="" style="white-space:pre">   </span>cache->last_flushed_segment_id = seg->global_id - 1;</div><div><br></div><div><span class="" style="white-space:pre">        </span>cache->last_migrated_segment_id =</div>
<div><span class="" style="white-space:pre">            </span>cache->last_flushed_segment_id > cache->nr_segments ?</div><div><span class="" style="white-space:pre">             </span>cache->last_flushed_segment_id - cache->nr_segments : 0;</div>
<div><br></div><div><span class="" style="white-space:pre">   </span>if (sup.last_migrated_segment_id > cache->last_migrated_segment_id)</div><div><span class="" style="white-space:pre">          </span>cache->last_migrated_segment_id = sup.last_migrated_segment_id;</div>
<div><br></div><div><span class="" style="white-space:pre">   </span>wait_for_migration(cache, seg->global_id);</div><div><br></div><div><span class="" style="white-space:pre">     </span>discard_caches_inseg(cache, seg);</div>
<div><br></div><div><span class="" style="white-space:pre">   </span>/*</div><div><span class="" style="white-space:pre"> </span> * cursor is set to the first element of the segment.</div><div><span class="" style="white-space:pre">      </span> * This means that we will not use the element.</div>
<div><span class="" style="white-space:pre">    </span> * I believe this is the simplest implementation.</div><div><span class="" style="white-space:pre">  </span> */</div><div><span class="" style="white-space:pre">        </span>cache->cursor = seg->start_idx;</div>
<div><span class="" style="white-space:pre">    </span>seg->length = 1;</div><div><br></div><div><span class="" style="white-space:pre">       </span>cache->current_seg = seg;</div><div>}</div><div><br></div><div>static sector_t dm_devsize(struct dm_dev *dev)</div>
<div>{</div><div><span class="" style="white-space:pre">    </span>return i_size_read(dev->bdev->bd_inode) >> SECTOR_SHIFT;</div><div>}</div><div><br></div><div>static size_t calc_nr_segments(struct dm_dev *dev)</div>
<div>{</div><div><span class="" style="white-space:pre">    </span>sector_t devsize = dm_devsize(dev);</div><div><br></div><div><span class="" style="white-space:pre">       </span>/*</div><div><span class="" style="white-space:pre"> </span> * disk format:</div>
<div><span class="" style="white-space:pre">    </span> * superblock(512B/1024KB) [segment(1024KB)]+</div><div><span class="" style="white-space:pre">      </span> * We reserve the first segment (1MB) as the superblock.</div><div>
<span class="" style="white-space:pre">       </span> *</div><div><span class="" style="white-space:pre"> </span> * segment:</div><div><span class="" style="white-space:pre">        </span> * segment_header(4KB) metablock(4KB)*NR_CACHES_INSEG</div>
<div><span class="" style="white-space:pre">    </span> */</div><div><span class="" style="white-space:pre">        </span>return devsize / (1 << LC_SEGMENTSIZE_ORDER) - 1;</div><div>}</div><div><br></div><div>struct format_segmd_context {</div>
<div><span class="" style="white-space:pre">    </span>atomic64_t count;</div><div>};</div><div><br></div><div>static void format_segmd_endio(unsigned long error, void *__context)</div><div>{</div><div><span class="" style="white-space:pre">     </span>struct format_segmd_context *context = __context;</div>
<div><span class="" style="white-space:pre">    </span>atomic64_dec(&context->count);</div><div>}</div><div><br></div><div>static void format_cache_device(struct dm_dev *dev)</div><div>{</div><div><span class="" style="white-space:pre">   </span>size_t nr_segments = calc_nr_segments(dev);</div>
<div><span class="" style="white-space:pre">    </span>void *buf;</div><div><br></div><div><span class="" style="white-space:pre">        </span>buf = kzalloc(1 << SECTOR_SHIFT, GFP_KERNEL);</div><div><span class="" style="white-space:pre">        </span>struct dm_io_request io_req_sup = {</div>
<div><span class="" style="white-space:pre">            </span>.client = lc_io_client,</div><div><span class="" style="white-space:pre">            </span>.bi_rw = WRITE_FUA,</div><div><span class="" style="white-space:pre">                </span>.notify.fn = NULL,</div>
<div><span class="" style="white-space:pre">            </span>.mem.type = DM_IO_KMEM,</div><div><span class="" style="white-space:pre">            </span>.mem.ptr.addr = buf,</div><div><span class="" style="white-space:pre">       </span>};</div>
<div><span class="" style="white-space:pre">    </span>struct dm_io_region region_sup = {</div><div><span class="" style="white-space:pre">         </span>.bdev = dev->bdev,</div><div><span class="" style="white-space:pre">              </span>.sector = 0,</div>
<div><span class="" style="white-space:pre">            </span>.count = 1,</div><div><span class="" style="white-space:pre">        </span>};</div><div><span class="" style="white-space:pre"> </span>dm_safe_io_retry(&io_req_sup, &region_sup, 1, false);</div>
<div><span class="" style="white-space:pre">    </span>kfree(buf);</div><div><br></div><div><span class="" style="white-space:pre">       </span>struct format_segmd_context context;</div><div><span class="" style="white-space:pre">       </span>atomic64_set(&context.count, nr_segments);</div>
<div><br></div><div><span class="" style="white-space:pre">   </span>size_t i;</div><div><span class="" style="white-space:pre">  </span>buf = kzalloc(1 << 12, GFP_KERNEL);</div><div><span class="" style="white-space:pre">  </span>for (i = 0; i < nr_segments; i++) {</div>
<div><span class="" style="white-space:pre">            </span>struct dm_io_request io_req_seg = {</div><div><span class="" style="white-space:pre">                        </span>.client = lc_io_client,</div><div><span class="" style="white-space:pre">                    </span>.bi_rw = WRITE,</div>
<div><span class="" style="white-space:pre">                    </span>.notify.fn = format_segmd_endio,</div><div><span class="" style="white-space:pre">                   </span>.notify.context = &context,</div><div><span class="" style="white-space:pre">                    </span>.mem.type = DM_IO_KMEM,</div>
<div><span class="" style="white-space:pre">                    </span>.mem.ptr.addr = buf,</div><div><span class="" style="white-space:pre">               </span>};</div><div><span class="" style="white-space:pre">         </span>struct dm_io_region region_seg = {</div>
<div><span class="" style="white-space:pre">                    </span>.bdev = dev->bdev,</div><div><span class="" style="white-space:pre">                      </span>.sector = calc_segment_header_start(i),</div><div><span class="" style="white-space:pre">                    </span>.count = (1 << 3),</div>
<div><span class="" style="white-space:pre">            </span>};</div><div><span class="" style="white-space:pre">         </span>dm_safe_io_retry(&io_req_seg, &region_seg, 1, false);</div><div><span class="" style="white-space:pre">      </span>}</div>
<div><span class="" style="white-space:pre">    </span>kfree(buf);</div><div><br></div><div><span class="" style="white-space:pre">       </span>while (atomic64_read(&context.count))</div><div><span class="" style="white-space:pre">          </span>schedule_timeout_interruptible(msecs_to_jiffies(100));</div>
<div><br></div><div><span class="" style="white-space:pre">   </span>blkdev_issue_flush(dev->bdev, GFP_KERNEL, NULL);</div><div>}</div><div><br></div><div>static bool is_on_buffer(struct lc_cache *cache, cache_nr mb_idx)</div>
<div>{</div><div><span class="" style="white-space:pre">    </span>cache_nr start = cache->current_seg->start_idx;</div><div><span class="" style="white-space:pre">      </span>if (mb_idx < start)</div><div><span class="" style="white-space:pre">             </span>return false;</div>
<div><br></div><div><span class="" style="white-space:pre">   </span>if (mb_idx >= (start + NR_CACHES_INSEG))</div><div><span class="" style="white-space:pre">                </span>return false;</div><div><br></div><div><span class="" style="white-space:pre">     </span>return true;</div>
<div>}</div><div><br></div><div>static void bio_remap(struct bio *bio, struct dm_dev *dev, sector_t sector)</div><div>{</div><div><span class="" style="white-space:pre"> </span>bio->bi_bdev = dev->bdev;</div><div><span class="" style="white-space:pre">    </span>bio->bi_sector = sector;</div>
<div>}</div><div><br></div><div>static sector_t calc_cache_alignment(struct lc_cache *cache,</div><div><span class="" style="white-space:pre">                                </span>     sector_t bio_sector)</div><div>{</div><div><span class="" style="white-space:pre">  </span>return (bio_sector / (1 << 3)) * (1 << 3);</div>
<div>}</div><div><br></div><div>static void migrate_buffered_mb(struct lc_cache *cache,</div><div><span class="" style="white-space:pre">                             </span>struct metablock *mb, u8 dirty_bits)</div><div>{</div><div><span class="" style="white-space:pre">       </span>u8 k = 1 + (mb->idx % NR_CACHES_INSEG);</div>
<div><span class="" style="white-space:pre">    </span>sector_t offset = (k << 3);</div><div><br></div><div><span class="" style="white-space:pre"> </span>u8 i;</div><div><span class="" style="white-space:pre">      </span>void *buf = kmalloc_retry(1 << SECTOR_SHIFT, GFP_NOIO);</div>
<div><span class="" style="white-space:pre">    </span>for (i = 0; i < 8; i++) {</div><div><span class="" style="white-space:pre">               </span>bool bit_on = dirty_bits & (1 << i);</div><div><span class="" style="white-space:pre">             </span>if (!bit_on)</div>
<div><span class="" style="white-space:pre">                    </span>continue;</div><div><br></div><div><span class="" style="white-space:pre">         </span>void *src = cache->current_wb->data +</div><div><span class="" style="white-space:pre">                        </span>((offset + i) << SECTOR_SHIFT);</div>
<div><span class="" style="white-space:pre">            </span>memcpy(buf, src, 1 << SECTOR_SHIFT);</div><div><br></div><div><span class="" style="white-space:pre">                </span>struct dm_io_request io_req = {</div><div><span class="" style="white-space:pre">                    </span>.client = lc_io_client,</div>
<div><span class="" style="white-space:pre">                    </span>.bi_rw = WRITE_FUA,</div><div><span class="" style="white-space:pre">                        </span>.notify.fn = NULL,</div><div><span class="" style="white-space:pre">                 </span>.mem.type = DM_IO_KMEM,</div>
<div><span class="" style="white-space:pre">                    </span>.mem.ptr.addr = buf,</div><div><span class="" style="white-space:pre">               </span>};</div><div><br></div><div><span class="" style="white-space:pre">                </span>struct lc_device *lc = lc_devices[mb->device_id];</div>
<div><span class="" style="white-space:pre">            </span>sector_t dest = mb->sector + 1 * i;</div><div><span class="" style="white-space:pre">             </span>struct dm_io_region region = {</div><div><span class="" style="white-space:pre">                     </span>.bdev = lc->device->bdev,</div>
<div><span class="" style="white-space:pre">                    </span>.sector = dest,</div><div><span class="" style="white-space:pre">                    </span>.count = 1,</div><div><span class="" style="white-space:pre">                </span>};</div><div><br></div>
<div><span class="" style="white-space:pre">            </span>dm_safe_io_retry(&io_req, &region, 1, true);</div><div><span class="" style="white-space:pre">       </span>}</div><div><span class="" style="white-space:pre">  </span>kfree(buf);</div>
<div>}</div><div><br></div><div>static void queue_current_buffer(struct lc_cache *cache)</div><div>{</div><div><span class="" style="white-space:pre">    </span>/*</div><div><span class="" style="white-space:pre"> </span> * Before we get the next segment</div>
<div><span class="" style="white-space:pre">    </span> * We must wait until the segment is all clean.</div><div><span class="" style="white-space:pre">    </span> * A clean segment desn't have</div><div><span class="" style="white-space:pre"> </span> * log to flush and dirties to migrate.</div>
<div><span class="" style="white-space:pre">    </span> */</div><div><span class="" style="white-space:pre">        </span>size_t next_id = cache->current_seg->global_id + 1;</div><div><br></div><div><span class="" style="white-space:pre"> </span>struct segment_header *next_seg =</div>
<div><span class="" style="white-space:pre">            </span>get_segment_header_by_id(cache, next_id);</div><div><br></div><div><span class="" style="white-space:pre"> </span>wait_for_completion(&next_seg->flush_done);</div>
<div><br></div><div><span class="" style="white-space:pre">   </span>wait_for_migration(cache, next_id);</div><div><br></div><div><span class="" style="white-space:pre">       </span>queue_flushing(cache);</div><div>}</div><div><br>
</div><div>static void flush_current_buffer_sync(struct lc_cache *cache)</div><div>{</div><div><span class="" style="white-space:pre">    </span>mutex_lock(&cache->io_lock);</div><div><span class="" style="white-space:pre">        </span>struct segment_header *old_seg = cache->current_seg;</div>
<div><br></div><div><span class="" style="white-space:pre">   </span>queue_current_buffer(cache);</div><div><span class="" style="white-space:pre">       </span>cache->cursor = (cache->cursor + 1) % cache->nr_caches;</div>
<div><span class="" style="white-space:pre">    </span>cache->current_seg->length = 1;</div><div><span class="" style="white-space:pre">      </span>mutex_unlock(&cache->io_lock);</div><div><br></div><div><span class="" style="white-space:pre">     </span>wait_for_completion(&old_seg->flush_done);</div>
<div>}</div><div><br></div><div>static void flush_barrier_ios(struct work_struct *work)</div><div>{</div><div><span class="" style="white-space:pre">     </span>struct lc_cache *cache =</div><div><span class="" style="white-space:pre">           </span>container_of(work, struct lc_cache,</div>
<div><span class="" style="white-space:pre">                    </span>     barrier_deadline_work);</div><div><br></div><div><span class="" style="white-space:pre">      </span>if (bio_list_empty(&cache->barrier_ios))</div><div><span class="" style="white-space:pre">            </span>return;</div>
<div><br></div><div><span class="" style="white-space:pre">   </span>flush_current_buffer_sync(cache);</div><div>}</div><div><br></div><div>static void barrier_deadline_proc(unsigned long data)</div><div>{</div><div><span class="" style="white-space:pre">     </span>struct lc_cache *cache = (struct lc_cache *) data;</div>
<div><span class="" style="white-space:pre">    </span>schedule_work(&cache->barrier_deadline_work);</div><div>}</div><div><br></div><div>static void queue_barrier_io(struct lc_cache *cache, struct bio *bio)</div><div>
{</div><div><span class="" style="white-space:pre">       </span>mutex_lock(&cache->io_lock);</div><div><span class="" style="white-space:pre">        </span>bio_list_add(&cache->barrier_ios, bio);</div><div><span class="" style="white-space:pre">     </span>mutex_unlock(&cache->io_lock);</div>
<div><br></div><div><span class="" style="white-space:pre">   </span>if (!timer_pending(&cache->barrier_deadline_timer))</div><div><span class="" style="white-space:pre">         </span>mod_timer(&cache->barrier_deadline_timer,</div>
<div><span class="" style="white-space:pre">                    </span>  msecs_to_jiffies(cache->barrier_deadline_ms));</div><div>}</div><div><br></div><div>#define PER_BIO_VERSION KERNEL_VERSION(3, 8, 0)</div><div>#if LINUX_VERSION_CODE >= PER_BIO_VERSION</div>
<div>struct per_bio_data {</div><div><span class="" style="white-space:pre">        </span>void *ptr;</div><div>};</div><div>#endif</div><div><br></div><div>static int lc_map(struct dm_target *ti, struct bio *bio</div><div>#if LINUX_VERSION_CODE < PER_BIO_VERSION</div>
<div><span class="" style="white-space:pre">            </span>, union map_info *map_context</div><div>#endif</div><div><span class="" style="white-space:pre">         </span>)</div><div>{</div><div><span class="" style="white-space:pre">  </span>struct lc_device *lc = ti->private;</div>
<div><span class="" style="white-space:pre">    </span>struct dm_dev *orig = lc->device;</div><div><br></div><div><span class="" style="white-space:pre">      </span>if (!lc->cache) {</div><div><span class="" style="white-space:pre">               </span>bio_remap(bio, orig, bio->bi_sector);</div>
<div><span class="" style="white-space:pre">            </span>return DM_MAPIO_REMAPPED;</div><div><span class="" style="white-space:pre">  </span>}</div><div><br></div><div><span class="" style="white-space:pre"> </span>/*</div><div>
<span class="" style="white-space:pre">       </span> * We only discard the backing store.</div><div><span class="" style="white-space:pre">      </span> * 1. 3.4 kernel doesn't support split_discard_requests.</div><div><span class="" style="white-space:pre">       </span> *    Hence, it is close to impossible to discard blocks on cache.</div>
<div><span class="" style="white-space:pre">    </span> * 2. Discarding the blocks on cache is meaningless.</div><div><span class="" style="white-space:pre">       </span> *    Because they will be overwritten eventually.</div><div>
<span class="" style="white-space:pre">       </span> */</div><div><span class="" style="white-space:pre">        </span>if (bio->bi_rw & REQ_DISCARD) {</div><div><span class="" style="white-space:pre">             </span>bio_remap(bio, orig, bio->bi_sector);</div>
<div><span class="" style="white-space:pre">            </span>return DM_MAPIO_REMAPPED;</div><div><span class="" style="white-space:pre">  </span>}</div><div><br></div><div><span class="" style="white-space:pre"> </span>struct lc_cache *cache = lc->cache;</div>
<div><br></div><div><span class="" style="white-space:pre">   </span>if (bio->bi_rw & REQ_FLUSH) {</div><div><span class="" style="white-space:pre">               </span>BUG_ON(bio->bi_size);</div><div><span class="" style="white-space:pre">           </span>queue_barrier_io(cache, bio);</div>
<div><span class="" style="white-space:pre">            </span>return DM_MAPIO_SUBMITTED;</div><div><span class="" style="white-space:pre"> </span>}</div><div><br></div><div><span class="" style="white-space:pre"> </span>unsigned long flags;</div>
<div><br></div><div>#if LINUX_VERSION_CODE >= PER_BIO_VERSION</div><div><span class="" style="white-space:pre">        </span>struct per_bio_data *map_context =</div><div><span class="" style="white-space:pre">         </span>dm_per_bio_data(bio, ti->per_bio_data_size);</div>
<div>#endif</div><div><span class="" style="white-space:pre">       </span>map_context->ptr = NULL;</div><div><br></div><div><span class="" style="white-space:pre">       </span>sector_t bio_count = bio->bi_size >> SECTOR_SHIFT;</div>
<div><span class="" style="white-space:pre">    </span>bool bio_fullsize = (bio_count == (1 << 3));</div><div><span class="" style="white-space:pre"> </span>sector_t bio_offset = bio->bi_sector % (1 << 3);</div>
<div><br></div><div><span class="" style="white-space:pre">   </span>int rw = bio_data_dir(bio);</div><div><br></div><div><span class="" style="white-space:pre">       </span>struct lookup_key key = {</div><div><span class="" style="white-space:pre">          </span>.sector = calc_cache_alignment(cache, bio->bi_sector),</div>
<div><span class="" style="white-space:pre">            </span>.device_id = lc->id,</div><div><span class="" style="white-space:pre">    </span>};</div><div><br></div><div><span class="" style="white-space:pre">        </span>cache_nr k = ht_hash(cache, &key);</div>
<div><span class="" style="white-space:pre">    </span>struct ht_head *head = arr_at(cache->htable, k);</div><div><br></div><div><span class="" style="white-space:pre">       </span>struct segment_header *seg;</div><div><span class="" style="white-space:pre">        </span>struct metablock *mb;</div>
<div><br></div><div><span class="" style="white-space:pre">   </span>mutex_lock(&cache->io_lock);</div><div><span class="" style="white-space:pre">        </span>mb = ht_lookup(cache, head, &key);</div><div><span class="" style="white-space:pre">     </span>if (mb) {</div>
<div><span class="" style="white-space:pre">            </span>seg = ((void *) mb) - ((mb->idx % NR_CACHES_INSEG) *</div><div><span class="" style="white-space:pre">                            </span>       sizeof(struct metablock));</div><div><span class="" style="white-space:pre">          </span>atomic_inc(&seg->nr_inflight_ios);</div>
<div><span class="" style="white-space:pre">    </span>}</div><div><br></div><div><span class="" style="white-space:pre"> </span>bool found = (mb != NULL);</div><div><span class="" style="white-space:pre"> </span>bool on_buffer = false;</div>
<div><span class="" style="white-space:pre">    </span>if (found)</div><div><span class="" style="white-space:pre">         </span>on_buffer = is_on_buffer(cache, mb->idx);</div><div><br></div><div><span class="" style="white-space:pre">      </span>inc_stat(cache, rw, found, on_buffer, bio_fullsize);</div>
<div><br></div><div><span class="" style="white-space:pre">   </span>if (!rw) {</div><div><span class="" style="white-space:pre">         </span>mutex_unlock(&cache->io_lock);</div><div><br></div><div><span class="" style="white-space:pre">             </span>if (!found) {</div>
<div><span class="" style="white-space:pre">                    </span>bio_remap(bio, orig, bio->bi_sector);</div><div><span class="" style="white-space:pre">                   </span>return DM_MAPIO_REMAPPED;</div><div><span class="" style="white-space:pre">          </span>}</div>
<div><br></div><div><span class="" style="white-space:pre">           </span>u8 dirty_bits = atomic_read_mb_dirtiness(seg, mb);</div><div><br></div><div><span class="" style="white-space:pre">                </span>if (unlikely(on_buffer)) {</div>
<div><br></div><div><span class="" style="white-space:pre">                   </span>if (dirty_bits)</div><div><span class="" style="white-space:pre">                            </span>migrate_buffered_mb(cache, mb, dirty_bits);</div><div><br></div><div><span class="" style="white-space:pre">                       </span>/*</div>
<div><span class="" style="white-space:pre">                    </span> * Dirtiness of a live cache</div><div><span class="" style="white-space:pre">                       </span> *</div><div><span class="" style="white-space:pre">                 </span> * We can assume dirtiness of a cache only increase</div>
<div><span class="" style="white-space:pre">                    </span> * when it is on the buffer, we call this cache is live.</div><div><span class="" style="white-space:pre">                   </span> * This eases the locking because</div><div><span class="" style="white-space:pre">                  </span> * we don't worry the dirtiness of</div>
<div><span class="" style="white-space:pre">                    </span> * a live cache fluctuates.</div><div><span class="" style="white-space:pre">                        </span> */</div><div><br></div><div><span class="" style="white-space:pre">                       </span>atomic_dec(&seg->nr_inflight_ios);</div>
<div><span class="" style="white-space:pre">                    </span>bio_remap(bio, orig, bio->bi_sector);</div><div><span class="" style="white-space:pre">                   </span>return DM_MAPIO_REMAPPED;</div><div><span class="" style="white-space:pre">          </span>}</div>
<div><br></div><div><span class="" style="white-space:pre">           </span>wait_for_completion(&seg->flush_done);</div><div><span class="" style="white-space:pre">              </span>if (likely(dirty_bits == 255)) {</div><div><span class="" style="white-space:pre">                   </span>bio_remap(</div>
<div><span class="" style="white-space:pre">                            </span>bio, cache->device,</div><div><span class="" style="white-space:pre">                             </span>calc_mb_start_sector(seg, mb->idx)</div><div><span class="" style="white-space:pre">                              </span>+ bio_offset);</div>
<div><span class="" style="white-space:pre">                    </span>map_context->ptr = seg;</div><div><span class="" style="white-space:pre">         </span>} else {</div><div><br></div><div><span class="" style="white-space:pre">                  </span>/*</div>
<div><span class="" style="white-space:pre">                    </span> * Dirtiness of a stable cache</div><div><span class="" style="white-space:pre">                     </span> *</div><div><span class="" style="white-space:pre">                 </span> * Unlike the live caches doesn't</div>
<div><span class="" style="white-space:pre">                    </span> * fluctuate the dirtiness,</div><div><span class="" style="white-space:pre">                        </span> * stable caches which is not on the buffer</div><div><span class="" style="white-space:pre">                        </span> * but on the cache device</div>
<div><span class="" style="white-space:pre">                    </span> * may decrease the dirtiness by other processes</div><div><span class="" style="white-space:pre">                   </span> * other than the migrate daemon.</div><div><span class="" style="white-space:pre">                  </span> * This works fine</div>
<div><span class="" style="white-space:pre">                    </span> * because migrating the same cache twice</div><div><span class="" style="white-space:pre">                  </span> * doesn't destroy the cache concistency.</div><div><span class="" style="white-space:pre">                      </span> */</div>
<div><br></div><div><span class="" style="white-space:pre">                   </span>migrate_mb(cache, seg, mb, dirty_bits, true);</div><div><br></div><div><span class="" style="white-space:pre">                     </span>bool b = false;</div><div><span class="" style="white-space:pre">                    </span>lockseg(seg, flags);</div>
<div><span class="" style="white-space:pre">                    </span>if (mb->dirty_bits) {</div><div><span class="" style="white-space:pre">                           </span>mb->dirty_bits = 0;</div><div><span class="" style="white-space:pre">                             </span>b = true;</div>
<div><span class="" style="white-space:pre">                    </span>}</div><div><span class="" style="white-space:pre">                  </span>unlockseg(seg, flags);</div><div><br></div><div><span class="" style="white-space:pre">                    </span>if (b)</div>
<div><span class="" style="white-space:pre">                            </span>dec_nr_dirty_caches(mb->device_id);</div><div><br></div><div><span class="" style="white-space:pre">                    </span>atomic_dec(&seg->nr_inflight_ios);</div><div><span class="" style="white-space:pre">                  </span>bio_remap(bio, orig, bio->bi_sector);</div>
<div><span class="" style="white-space:pre">            </span>}</div><div><span class="" style="white-space:pre">          </span>return DM_MAPIO_REMAPPED;</div><div><span class="" style="white-space:pre">  </span>}</div><div><br></div><div>
<span class="" style="white-space:pre">       </span>cache_nr update_mb_idx;</div><div><span class="" style="white-space:pre">    </span>if (found) {</div><div><br></div><div><span class="" style="white-space:pre">              </span>if (unlikely(on_buffer)) {</div>
<div><span class="" style="white-space:pre">                    </span>mutex_unlock(&cache->io_lock);</div><div><br></div><div><span class="" style="white-space:pre">                     </span>update_mb_idx = mb->idx;</div><div><span class="" style="white-space:pre">                        </span>goto write_on_buffer;</div>
<div><span class="" style="white-space:pre">            </span>} else {</div><div><br></div><div><span class="" style="white-space:pre">                  </span>u8 dirty_bits = atomic_read_mb_dirtiness(seg, mb);</div><div><br></div><div><span class="" style="white-space:pre">                        </span>/*</div>
<div><span class="" style="white-space:pre">                    </span> * First clean up the previous cache</div><div><span class="" style="white-space:pre">                       </span> * and migrate the cache if needed.</div><div><span class="" style="white-space:pre">                        </span> */</div>
<div><span class="" style="white-space:pre">                    </span>bool needs_cleanup_prev_cache =</div><div><span class="" style="white-space:pre">                            </span>!bio_fullsize || !(dirty_bits == 255);</div><div><br></div><div><span class="" style="white-space:pre">                    </span>if (unlikely(needs_cleanup_prev_cache)) {</div>
<div><span class="" style="white-space:pre">                            </span>wait_for_completion(&seg->flush_done);</div><div><span class="" style="white-space:pre">                              </span>migrate_mb(cache, seg, mb, dirty_bits, true);</div><div><span class="" style="white-space:pre">                      </span>}</div>
<div><br></div><div><span class="" style="white-space:pre">                   </span>/*</div><div><span class="" style="white-space:pre">                 </span> * Fullsize dirty cache</div><div><span class="" style="white-space:pre">                    </span> * can be discarded without migration.</div>
<div><span class="" style="white-space:pre">                    </span> */</div><div><span class="" style="white-space:pre">                        </span>bool b = false;</div><div><span class="" style="white-space:pre">                    </span>lockseg(seg, flags);</div><div>
<span class="" style="white-space:pre">                       </span>if (mb->dirty_bits) {</div><div><span class="" style="white-space:pre">                           </span>mb->dirty_bits = 0;</div><div><span class="" style="white-space:pre">                             </span>b = true;</div>
<div><span class="" style="white-space:pre">                    </span>}</div><div><span class="" style="white-space:pre">                  </span>unlockseg(seg, flags);</div><div><br></div><div><span class="" style="white-space:pre">                    </span>if (b)</div>
<div><span class="" style="white-space:pre">                            </span>dec_nr_dirty_caches(mb->device_id);</div><div><br></div><div><span class="" style="white-space:pre">                    </span>ht_del(cache, mb);</div><div><br></div><div><span class="" style="white-space:pre">                        </span>atomic_dec(&seg->nr_inflight_ios);</div>
<div><span class="" style="white-space:pre">                    </span>goto write_not_found;</div><div><span class="" style="white-space:pre">              </span>}</div><div><span class="" style="white-space:pre">  </span>}</div><div><br></div><div>write_not_found:</div>
<div><span class="" style="white-space:pre">    </span>;</div><div><br></div><div><span class="" style="white-space:pre"> </span>/*</div><div><span class="" style="white-space:pre"> </span> * If cache->cursor is 254, 509, ...</div>
<div><span class="" style="white-space:pre">    </span> * that is the last cache line in the segment.</div><div><span class="" style="white-space:pre">     </span> * We must flush the current segment and</div><div><span class="" style="white-space:pre">   </span> * get the new one.</div>
<div><span class="" style="white-space:pre">    </span> */</div><div><span class="" style="white-space:pre">        </span>bool refresh_segment = !((cache->cursor + 1) % NR_CACHES_INSEG);</div><div><br></div><div><span class="" style="white-space:pre">       </span>if (refresh_segment)</div>
<div><span class="" style="white-space:pre">            </span>queue_current_buffer(cache);</div><div><br></div><div><span class="" style="white-space:pre">      </span>cache->cursor = (cache->cursor + 1) % cache->nr_caches;</div>
<div><br></div><div><span class="" style="white-space:pre">   </span>/*</div><div><span class="" style="white-space:pre"> </span> * update_mb_idx is the cache line index to update.</div><div><span class="" style="white-space:pre">        </span> */</div>
<div><span class="" style="white-space:pre">    </span>update_mb_idx = cache->cursor;</div><div><br></div><div><span class="" style="white-space:pre"> </span>seg = cache->current_seg;</div><div><span class="" style="white-space:pre">       </span>atomic_inc(&seg->nr_inflight_ios);</div>
<div><br></div><div><span class="" style="white-space:pre">   </span>struct metablock *new_mb =</div><div><span class="" style="white-space:pre">         </span>seg->mb_array + (update_mb_idx % NR_CACHES_INSEG);</div><div><span class="" style="white-space:pre">      </span>new_mb->dirty_bits = 0;</div>
<div><span class="" style="white-space:pre">    </span>ht_register(cache, head, &key, new_mb);</div><div><span class="" style="white-space:pre">        </span>mutex_unlock(&cache->io_lock);</div><div><br></div><div><span class="" style="white-space:pre">     </span>mb = new_mb;</div>
<div><br></div><div>write_on_buffer:</div><div><span class="" style="white-space:pre">    </span>;</div><div><span class="" style="white-space:pre">  </span>cache_nr idx_inseg = update_mb_idx % NR_CACHES_INSEG;</div><div><span class="" style="white-space:pre">      </span>sector_t s = (idx_inseg + 1) << 3;</div>
<div><br></div><div><span class="" style="white-space:pre">   </span>bool b = false;</div><div><span class="" style="white-space:pre">    </span>lockseg(seg, flags);</div><div><span class="" style="white-space:pre">       </span>if (!mb->dirty_bits) {</div>
<div><span class="" style="white-space:pre">            </span>seg->length++;</div><div><span class="" style="white-space:pre">          </span>BUG_ON(seg->length >  NR_CACHES_INSEG);</div><div><span class="" style="white-space:pre">              </span>b = true;</div>
<div><span class="" style="white-space:pre">    </span>}</div><div><br></div><div><span class="" style="white-space:pre"> </span>if (likely(bio_fullsize)) {</div><div><span class="" style="white-space:pre">                </span>mb->dirty_bits = 255;</div>
<div><span class="" style="white-space:pre">    </span>} else {</div><div><span class="" style="white-space:pre">           </span>s += bio_offset;</div><div><span class="" style="white-space:pre">           </span>u8 i;</div><div><span class="" style="white-space:pre">              </span>u8 acc_bits = 0;</div>
<div><span class="" style="white-space:pre">            </span>for (i = bio_offset; i < (bio_offset+bio_count); i++)</div><div><span class="" style="white-space:pre">                   </span>acc_bits += (1 << i);</div><div><br></div><div>
<span class="" style="white-space:pre">               </span>mb->dirty_bits |= acc_bits;</div><div><span class="" style="white-space:pre">     </span>}</div><div><br></div><div><span class="" style="white-space:pre"> </span>BUG_ON(!mb->dirty_bits);</div>
<div><br></div><div><span class="" style="white-space:pre">   </span>unlockseg(seg, flags);</div><div><br></div><div><span class="" style="white-space:pre">    </span>if (b)</div><div><span class="" style="white-space:pre">             </span>inc_nr_dirty_caches(mb->device_id);</div>
<div><br></div><div><span class="" style="white-space:pre">   </span>size_t start = s << SECTOR_SHIFT;</div><div><span class="" style="white-space:pre">    </span>void *data = bio_data(bio);</div><div><br></div><div><span class="" style="white-space:pre">       </span>memcpy(cache->current_wb->data + start, data, bio->bi_size);</div>
<div><span class="" style="white-space:pre">    </span>atomic_dec(&seg->nr_inflight_ios);</div><div><br></div><div><span class="" style="white-space:pre"> </span>if (bio->bi_rw & REQ_FUA) {</div><div><span class="" style="white-space:pre">         </span>queue_barrier_io(cache, bio);</div>
<div><span class="" style="white-space:pre">            </span>return DM_MAPIO_SUBMITTED;</div><div><span class="" style="white-space:pre"> </span>}</div><div><br></div><div><span class="" style="white-space:pre"> </span>bio_endio(bio, 0);</div>
<div><span class="" style="white-space:pre">    </span>return DM_MAPIO_SUBMITTED;</div><div>}</div><div><br></div><div>static int lc_end_io(struct dm_target *ti, struct bio *bio, int error</div><div>#if LINUX_VERSION_CODE < PER_BIO_VERSION</div>
<div><span class="" style="white-space:pre">            </span>, union map_info *map_context</div><div>#endif</div><div><span class="" style="white-space:pre">         </span>)</div><div>{</div><div>#if LINUX_VERSION_CODE >= PER_BIO_VERSION</div>
<div><span class="" style="white-space:pre">    </span>struct per_bio_data *map_context =</div><div><span class="" style="white-space:pre">         </span>dm_per_bio_data(bio, ti->per_bio_data_size);</div><div>#endif</div><div><span class="" style="white-space:pre">       </span>if (!map_context->ptr)</div>
<div><span class="" style="white-space:pre">            </span>return 0;</div><div><br></div><div><span class="" style="white-space:pre"> </span>struct segment_header *seg = map_context->ptr;</div><div><span class="" style="white-space:pre">  </span>atomic_dec(&seg->nr_inflight_ios);</div>
<div><br></div><div><span class="" style="white-space:pre">   </span>return 0;</div><div>}</div><div><br></div><div><br></div><div>static ssize_t var_show(unsigned long var, char *page)</div><div>{</div><div><span class="" style="white-space:pre">   </span>return sprintf(page, "%lu\n", var);</div>
<div>}</div><div><br></div><div>static ssize_t var_store(unsigned long *var, const char *page, size_t len)</div><div>{</div><div><span class="" style="white-space:pre">  </span>char *p = (char *) page;</div><div><span class="" style="white-space:pre">   </span>int r = kstrtoul(p, 10, var);</div>
<div><span class="" style="white-space:pre">    </span>if (r)</div><div><span class="" style="white-space:pre">             </span>return r;</div><div><span class="" style="white-space:pre">  </span>return len;</div><div>}</div><div><br>
</div><div>static struct kobject *devices_kobj;</div><div><br></div><div>struct device_sysfs_entry {</div><div><span class="" style="white-space:pre">  </span>struct attribute attr;</div><div><span class="" style="white-space:pre">     </span>ssize_t (*show)(struct lc_device *, char *);</div>
<div><span class="" style="white-space:pre">    </span>ssize_t (*store)(struct lc_device *, const char *, size_t);</div><div>};</div><div><br></div><div>#define to_device(attr) container_of((attr), struct device_sysfs_entry, attr)</div>
<div>static ssize_t device_attr_show(struct kobject *kobj, struct attribute *attr,</div><div><span class="" style="white-space:pre">                                </span>char *page)</div><div>{</div><div><span class="" style="white-space:pre">        </span>struct device_sysfs_entry *entry = to_device(attr);</div>
<div><span class="" style="white-space:pre">    </span>struct lc_device *device =</div><div><span class="" style="white-space:pre">         </span>container_of(kobj, struct lc_device, kobj);</div><div><br></div><div><span class="" style="white-space:pre">       </span>return entry->show(device, page);</div>
<div>}</div><div><br></div><div>static ssize_t device_attr_store(struct kobject *kobj, struct attribute *attr,</div><div><span class="" style="white-space:pre">                              </span> const char *page, size_t len)</div><div>{</div><div>
<span class="" style="white-space:pre">       </span>struct device_sysfs_entry *entry = to_device(attr);</div><div><span class="" style="white-space:pre">        </span>if (!entry->store)</div><div><span class="" style="white-space:pre">              </span>return -EIO;</div>
<div><br></div><div><span class="" style="white-space:pre">   </span>struct lc_device *device = container_of(kobj, struct lc_device, kobj);</div><div><span class="" style="white-space:pre">     </span>return entry->store(device, page, len);</div>
<div>}</div><div><br></div><div>static cache_id cache_id_of(struct lc_device *device)</div><div>{</div><div><span class="" style="white-space:pre">       </span>cache_id id;</div><div><span class="" style="white-space:pre">       </span>if (!device->cache)</div>
<div><span class="" style="white-space:pre">            </span>id = 0;</div><div><span class="" style="white-space:pre">    </span>else</div><div><span class="" style="white-space:pre">               </span>id = device->cache->id;</div><div>
<span class="" style="white-space:pre">       </span>return id;</div><div>}</div><div><br></div><div>static ssize_t cache_id_show(struct lc_device *device, char *page)</div><div>{</div><div><span class="" style="white-space:pre">       </span>return var_show(cache_id_of(device), (page));</div>
<div>}</div><div><br></div><div>static struct device_sysfs_entry cache_id_entry = {</div><div><span class="" style="white-space:pre"> </span>.attr = { .name = "cache_id", .mode = S_IRUGO },</div><div><span class="" style="white-space:pre"> </span>.show = cache_id_show,</div>
<div>};</div><div><br></div><div>static ssize_t dev_show(struct lc_device *device, char *page)</div><div>{</div><div><span class="" style="white-space:pre">      </span>return sprintf(page, "%s\n", dm_device_name(device->md));</div>
<div>}</div><div><br></div><div>static struct device_sysfs_entry dev_entry = {</div><div><span class="" style="white-space:pre">      </span>.attr = { .name = "dev", .mode = S_IRUGO },</div><div><span class="" style="white-space:pre">      </span>.show = dev_show,</div>
<div>};</div><div><br></div><div>static ssize_t migrate_threshold_show(struct lc_device *device, char *page)</div><div>{</div><div><span class="" style="white-space:pre">        </span>return var_show(device->migrate_threshold, (page));</div>
<div>}</div><div><br></div><div>static ssize_t migrate_threshold_store(struct lc_device *device,</div><div><span class="" style="white-space:pre">                            </span>       const char *page, size_t count)</div><div>{</div><div><span class="" style="white-space:pre">     </span>unsigned long x;</div>
<div><span class="" style="white-space:pre">    </span>ssize_t r = var_store(&x, page, count);</div><div><span class="" style="white-space:pre">        </span>device->migrate_threshold = x;</div><div><span class="" style="white-space:pre">  </span>return r;</div>
<div>}</div><div><br></div><div>static struct device_sysfs_entry migrate_threshold_entry = {</div><div><span class="" style="white-space:pre">        </span>.attr = { .name = "migrate_threshold", .mode = S_IRUGO | S_IWUSR },</div>
<div><span class="" style="white-space:pre">    </span>.show = migrate_threshold_show,</div><div><span class="" style="white-space:pre">    </span>.store = migrate_threshold_store,</div><div>};</div><div><br></div><div>static ssize_t nr_dirty_caches_show(struct lc_device *device, char *page)</div>
<div>{</div><div><span class="" style="white-space:pre">    </span>unsigned long val = atomic64_read(&device->nr_dirty_caches);</div><div><span class="" style="white-space:pre">        </span>return var_show(val, page);</div>
<div>}</div><div><br></div><div>static struct device_sysfs_entry nr_dirty_caches_entry = {</div><div><span class="" style="white-space:pre">  </span>.attr = { .name = "nr_dirty_caches", .mode = S_IRUGO },</div><div>
<span class="" style="white-space:pre">       </span>.show = nr_dirty_caches_show,</div><div>};</div><div><br></div><div>static struct attribute *device_default_attrs[] = {</div><div><span class="" style="white-space:pre">  </span>&cache_id_entry.attr,</div>
<div><span class="" style="white-space:pre">    </span>&dev_entry.attr,</div><div><span class="" style="white-space:pre">       </span>&migrate_threshold_entry.attr,</div><div><span class="" style="white-space:pre"> </span>&nr_dirty_caches_entry.attr,</div>
<div><span class="" style="white-space:pre">    </span>NULL,</div><div>};</div><div><br></div><div>static const struct sysfs_ops device_sysfs_ops = {</div><div><span class="" style="white-space:pre">   </span>.show = device_attr_show,</div>
<div><span class="" style="white-space:pre">    </span>.store = device_attr_store,</div><div>};</div><div><br></div><div>static void device_release(struct kobject *kobj)</div><div>{</div><div><span class="" style="white-space:pre">       </span>return;</div>
<div>}</div><div><br></div><div>static struct kobj_type device_ktype = {</div><div><span class="" style="white-space:pre">    </span>.sysfs_ops = &device_sysfs_ops,</div><div><span class="" style="white-space:pre">        </span>.default_attrs = device_default_attrs,</div>
<div><span class="" style="white-space:pre">    </span>.release = device_release,</div><div>};</div><div><br></div><div>/*</div><div> * <device-id> <path> <cache-id></div><div> */</div><div>static int lc_ctr(struct dm_target *ti, unsigned int argc, char **argv)</div>
<div>{</div><div>#if LINUX_VERSION_CODE >= KERNEL_VERSION(3, 6, 0)</div><div><span class="" style="white-space:pre"> </span>int r;</div><div><span class="" style="white-space:pre">     </span>r = dm_set_target_max_io_len(ti, (1 << 3));</div>
<div><span class="" style="white-space:pre">    </span>if (r)</div><div><span class="" style="white-space:pre">             </span>return r;</div><div><br></div><div>#else</div><div><span class="" style="white-space:pre">     </span>ti->split_io = (1 << 3);</div>
<div>#endif</div><div><br></div><div><span class="" style="white-space:pre">      </span>struct lc_device *lc = kzalloc(sizeof(*lc), GFP_KERNEL);</div><div><br></div><div><span class="" style="white-space:pre">  </span>/*</div>
<div><span class="" style="white-space:pre">    </span> * EMC's book says</div><div><span class="" style="white-space:pre">     </span> * storage should keep its disk util less than 70%.</div><div><span class="" style="white-space:pre">        </span> */</div>
<div><span class="" style="white-space:pre">    </span>lc->migrate_threshold = 70;</div><div><br></div><div><span class="" style="white-space:pre">    </span>atomic64_set(&lc->nr_dirty_caches, 0);</div><div><span class="" style="white-space:pre">      </span>atomic64_inc(&lc->nr_dirty_caches);</div>
<div><span class="" style="white-space:pre">    </span>atomic64_dec(&lc->nr_dirty_caches);</div><div><br></div><div><span class="" style="white-space:pre">        </span>unsigned device_id;</div><div><span class="" style="white-space:pre">        </span>if (sscanf(argv[0], "%u", &device_id) != 1)</div>
<div><span class="" style="white-space:pre">            </span>return -EINVAL;</div><div><br></div><div><span class="" style="white-space:pre">   </span>lc->id = device_id;</div><div><br></div><div><span class="" style="white-space:pre">    </span>struct dm_dev *dev;</div>
<div><span class="" style="white-space:pre">    </span>if (dm_get_device(ti, argv[1], dm_table_get_mode(ti->table), &dev))</div><div><span class="" style="white-space:pre">         </span>return -EINVAL;</div><div><br></div>
<div><span class="" style="white-space:pre">    </span>lc->device = dev;</div><div><br></div><div><span class="" style="white-space:pre">      </span>lc->cache = NULL;</div><div><span class="" style="white-space:pre">       </span>unsigned cache_id;</div>
<div><span class="" style="white-space:pre">    </span>if (sscanf(argv[2], "%u", &cache_id) != 1)</div><div><span class="" style="white-space:pre">           </span>return -EINVAL;</div><div><br></div><div><span class="" style="white-space:pre">   </span>if (cache_id)</div>
<div><span class="" style="white-space:pre">            </span>lc->cache = lc_caches[cache_id];</div><div><br></div><div><span class="" style="white-space:pre">       </span>lc_devices[lc->id] = lc;</div><div><span class="" style="white-space:pre">        </span>ti->private = lc;</div>
<div><br></div><div>#if LINUX_VERSION_CODE >= PER_BIO_VERSION</div><div><span class="" style="white-space:pre">        </span>ti->per_bio_data_size = sizeof(struct per_bio_data);</div><div>#endif</div><div><br></div><div>#if LINUX_VERSION_CODE >= KERNEL_VERSION(3, 9, 0)</div>
<div><span class="" style="white-space:pre">    </span>ti->num_flush_bios = 1;</div><div><span class="" style="white-space:pre"> </span>ti->num_discard_bios = 1;</div><div>#else</div><div><span class="" style="white-space:pre">   </span>ti->num_flush_requests = 1;</div>
<div><span class="" style="white-space:pre">    </span>ti->num_discard_requests = 1;</div><div>#endif</div><div><br></div><div><span class="" style="white-space:pre">     </span>ti->discard_zeroes_data_unsupported = true;</div>
<div><br></div><div><span class="" style="white-space:pre">   </span>/*</div><div><span class="" style="white-space:pre"> </span> * /sys/module/dm_lc/devices/$id/$atribute</div><div><span class="" style="white-space:pre"> </span> *                              /dev # -> Note</div>
<div><span class="" style="white-space:pre">    </span> *                              /device</div><div><span class="" style="white-space:pre">    </span> */</div><div><br></div><div><span class="" style="white-space:pre">       </span>/*</div>
<div><span class="" style="white-space:pre">    </span> * Note:</div><div><span class="" style="white-space:pre">   </span> * Reference to the mapped_device</div><div><span class="" style="white-space:pre">  </span> * is used to show device name (major:minor).</div>
<div><span class="" style="white-space:pre">    </span> * major:minor is used in admin scripts</div><div><span class="" style="white-space:pre">    </span> * to get the sysfs node of a lc_device.</div><div><span class="" style="white-space:pre">   </span> */</div>
<div><span class="" style="white-space:pre">    </span>lc->md = dm_table_get_md(ti->table);</div><div><br></div><div><span class="" style="white-space:pre">        </span>return 0;</div><div>}</div><div><br></div><div>static void lc_dtr(struct dm_target *ti)</div>
<div>{</div><div><span class="" style="white-space:pre">    </span>struct lc_device *lc = ti->private;</div><div><br></div><div><span class="" style="white-space:pre">    </span>dm_put_device(ti, lc->device);</div><div><br>
</div><div><span class="" style="white-space:pre">        </span>ti->private = NULL;</div><div><span class="" style="white-space:pre">     </span>kfree(lc);</div><div>}</div><div><br></div><div>struct kobject *get_bdev_kobject(struct block_device *bdev)</div>
<div>{</div><div><span class="" style="white-space:pre">    </span>return &disk_to_dev(bdev->bd_disk)->kobj;</div><div>}</div><div><br></div><div>static int lc_message(struct dm_target *ti, unsigned argc, char **argv)</div>
<div>{</div><div><span class="" style="white-space:pre">    </span>int r;</div><div><br></div><div><span class="" style="white-space:pre">    </span>struct lc_device *lc = ti->private;</div><div><br></div><div><span class="" style="white-space:pre">    </span>char *cmd = argv[0];</div>
<div><br></div><div><span class="" style="white-space:pre">   </span>if (!strcasecmp(cmd, "add_sysfs")) {</div><div><span class="" style="white-space:pre">             </span>r = kobject_init_and_add(&lc->kobj, &device_ktype,</div>
<div><span class="" style="white-space:pre">                                    </span> devices_kobj, "%u", lc->id);</div><div><span class="" style="white-space:pre">         </span>struct kobject *dev_kobj = get_bdev_kobject(lc->device->bdev);</div>
<div><span class="" style="white-space:pre">            </span>r = sysfs_create_link(&lc->kobj, dev_kobj, "device");</div><div><br></div><div><span class="" style="white-space:pre">            </span>/* kobject_uevent(&lc->kobj, KOBJ_ADD); */</div>
<div><span class="" style="white-space:pre">            </span>return 0;</div><div><span class="" style="white-space:pre">  </span>}</div><div><br></div><div><span class="" style="white-space:pre"> </span>if (!strcasecmp(cmd, "remove_sysfs")) {</div>
<div><span class="" style="white-space:pre">            </span>sysfs_remove_link(&lc->kobj, "device");</div><div><span class="" style="white-space:pre">           </span>kobject_del(&lc->kobj);</div><div><span class="" style="white-space:pre">             </span>kobject_put(&lc->kobj);</div>
<div><br></div><div><span class="" style="white-space:pre">           </span>lc_devices[lc->id] = NULL;</div><div><br></div><div><span class="" style="white-space:pre">             </span>/* kobject_uevent(&lc->kobj, KOBJ_REMOVE); */</div>
<div><span class="" style="white-space:pre">            </span>return 0;</div><div><span class="" style="white-space:pre">  </span>}</div><div><br></div><div><span class="" style="white-space:pre"> </span>return -EINVAL;</div><div>}</div>
<div><br></div><div>static int lc_merge(struct dm_target *ti, struct bvec_merge_data *bvm,</div><div><span class="" style="white-space:pre">              </span>    struct bio_vec *biovec, int max_size)</div><div>{</div><div><span class="" style="white-space:pre">  </span>struct lc_device *lc = ti->private;</div>
<div><span class="" style="white-space:pre">    </span>struct dm_dev *device = lc->device;</div><div><span class="" style="white-space:pre">     </span>struct request_queue *q = bdev_get_queue(device->bdev);</div><div><br>
</div><div><span class="" style="white-space:pre">        </span>if (!q->merge_bvec_fn)</div><div><span class="" style="white-space:pre">          </span>return max_size;</div><div><br></div><div><span class="" style="white-space:pre">  </span>bvm->bi_bdev = device->bdev;</div>
<div><span class="" style="white-space:pre">    </span>return min(max_size, q->merge_bvec_fn(q, bvm, biovec));</div><div>}</div><div><br></div><div>static int lc_iterate_devices(struct dm_target *ti,</div><div><span class="" style="white-space:pre">                      </span>      iterate_devices_callout_fn fn, void *data)</div>
<div>{</div><div><span class="" style="white-space:pre">    </span>struct lc_device *lc = ti->private;</div><div><span class="" style="white-space:pre">     </span>struct dm_dev *orig = lc->device;</div><div><span class="" style="white-space:pre">       </span>sector_t start = 0;</div>
<div><span class="" style="white-space:pre">    </span>sector_t len = dm_devsize(orig);</div><div><span class="" style="white-space:pre">   </span>return fn(ti, orig, start, len, data);</div><div>}</div><div><br></div><div>static void lc_io_hints(struct dm_target *ti, struct queue_limits *limits)</div>
<div>{</div><div><span class="" style="white-space:pre">    </span>blk_limits_io_min(limits, 512);</div><div><span class="" style="white-space:pre">    </span>blk_limits_io_opt(limits, 4096);</div><div>}</div><div><br></div><div>
static</div><div>#if LINUX_VERSION_CODE >= KERNEL_VERSION(3, 8, 0)</div><div>void</div><div>#else</div><div>int</div><div>#endif</div><div>lc_status(</div><div><span class="" style="white-space:pre">                </span>struct dm_target *ti, status_type_t type,</div>
<div>#if LINUX_VERSION_CODE >= KERNEL_VERSION(3, 6, 0)</div><div><span class="" style="white-space:pre">         </span>unsigned flags,</div><div>#endif</div><div><span class="" style="white-space:pre">               </span>char *result,</div>
<div><span class="" style="white-space:pre">            </span>unsigned maxlen)</div><div>{</div><div><span class="" style="white-space:pre">   </span>unsigned int sz = 0;</div><div><br></div><div><span class="" style="white-space:pre">      </span>struct lc_device *lc = ti->private;</div>
<div><span class="" style="white-space:pre">    </span>switch (type) {</div><div><span class="" style="white-space:pre">    </span>case STATUSTYPE_INFO:</div><div><span class="" style="white-space:pre">              </span>result[0] = '\0';</div>
<div><span class="" style="white-space:pre">            </span>break;</div><div><br></div><div><span class="" style="white-space:pre">    </span>case STATUSTYPE_TABLE:</div><div><span class="" style="white-space:pre">             </span>DMEMIT("%d %s %d", lc->id, lc->device->name, cache_id_of(lc));</div>
<div><span class="" style="white-space:pre">            </span>break;</div><div><span class="" style="white-space:pre">     </span>}</div><div>#if LINUX_VERSION_CODE < KERNEL_VERSION(3, 8, 0)</div><div><span class="" style="white-space:pre">        </span>return 0;</div>
<div>#endif</div><div>}</div><div><br></div><div>static struct target_type lc_target = {</div><div><span class="" style="white-space:pre">        </span>.name = "lc",</div><div><span class="" style="white-space:pre">    </span>.version = {1, 0, 0},</div>
<div><span class="" style="white-space:pre">    </span>.module = THIS_MODULE,</div><div><span class="" style="white-space:pre">     </span>.map = lc_map,</div><div><span class="" style="white-space:pre">     </span>.ctr = lc_ctr,</div>
<div><span class="" style="white-space:pre">    </span>.dtr = lc_dtr,</div><div><span class="" style="white-space:pre">     </span>.end_io = lc_end_io,</div><div><span class="" style="white-space:pre">       </span>.merge = lc_merge,</div>
<div><span class="" style="white-space:pre">    </span>.message = lc_message,</div><div><span class="" style="white-space:pre">     </span>.status = lc_status,</div><div><span class="" style="white-space:pre">       </span>.io_hints = lc_io_hints,</div>
<div><span class="" style="white-space:pre">    </span>.iterate_devices = lc_iterate_devices,</div><div>};</div><div><br></div><div>static int lc_mgr_map(struct dm_target *ti, struct bio *bio</div><div>#if LINUX_VERSION_CODE < KERNEL_VERSION(3, 8, 0)</div>
<div><span class="" style="white-space:pre">            </span>, union map_info *map_context</div><div>#endif</div><div><span class="" style="white-space:pre">         </span>)</div><div>{</div><div><span class="" style="white-space:pre">  </span>bio_endio(bio, 0);</div>
<div><span class="" style="white-space:pre">    </span>return DM_MAPIO_SUBMITTED;</div><div>}</div><div><br></div><div>static int lc_mgr_ctr(struct dm_target *ti, unsigned int argc, char **argv)</div><div>{</div><div><span class="" style="white-space:pre">      </span>return 0;</div>
<div>}</div><div><br></div><div>static void lc_mgr_dtr(struct dm_target *ti)</div><div>{</div><div><span class="" style="white-space:pre">        </span>return;</div><div>}</div><div><br></div><div>static struct kobject *caches_kobj;</div>
<div><br></div><div>struct cache_sysfs_entry {</div><div><span class="" style="white-space:pre">  </span>struct attribute attr;</div><div><span class="" style="white-space:pre">     </span>ssize_t (*show)(struct lc_cache *, char *);</div>
<div><span class="" style="white-space:pre">    </span>ssize_t (*store)(struct lc_cache *, const char *, size_t);</div><div>};</div><div><br></div><div>#define to_cache(attr) container_of((attr), struct cache_sysfs_entry, attr)</div>
<div>static ssize_t cache_attr_show(struct kobject *kobj,</div><div><span class="" style="white-space:pre">                 </span>       struct attribute *attr, char *page)</div><div>{</div><div><span class="" style="white-space:pre"> </span>struct cache_sysfs_entry *entry = to_cache(attr);</div>
<div><span class="" style="white-space:pre">    </span>struct lc_cache *cache =</div><div><span class="" style="white-space:pre">           </span>container_of(kobj, struct lc_cache, kobj);</div><div><br></div><div><span class="" style="white-space:pre">        </span>return entry->show(cache, page);</div>
<div>}</div><div><br></div><div>static ssize_t cache_attr_store(struct kobject *kobj, struct attribute *attr,</div><div><span class="" style="white-space:pre">               </span>const char *page, size_t len)</div><div>{</div><div><span class="" style="white-space:pre">      </span>struct cache_sysfs_entry *entry = to_cache(attr);</div>
<div><span class="" style="white-space:pre">    </span>if (!entry->store)</div><div><span class="" style="white-space:pre">              </span>return -EIO;</div><div><br></div><div><span class="" style="white-space:pre">      </span>struct lc_cache *cache = container_of(kobj, struct lc_cache, kobj);</div>
<div><span class="" style="white-space:pre">    </span>return entry->store(cache, page, len);</div><div>}</div><div><br></div><div>static ssize_t commit_super_block_interval_show(struct lc_cache *cache,</div><div><span class="" style="white-space:pre">                                           </span>char *page)</div>
<div>{</div><div><span class="" style="white-space:pre">    </span>return var_show(cache->commit_super_block_interval, (page));</div><div>}</div><div><br></div><div>static ssize_t commit_super_block_interval_store(struct lc_cache *cache,</div>
<div><span class="" style="white-space:pre">                                            </span> const char *page, size_t count)</div><div>{</div><div><span class="" style="white-space:pre">   </span>unsigned long x;</div><div><span class="" style="white-space:pre">   </span>ssize_t r = var_store(&x, page, count);</div>
<div><span class="" style="white-space:pre">    </span>cache->commit_super_block_interval = x;</div><div><span class="" style="white-space:pre"> </span>return r;</div><div>}</div><div><br></div><div>static struct cache_sysfs_entry commit_super_block_interval_entry = {</div>
<div><span class="" style="white-space:pre">    </span>.attr = { .name = "commit_super_block_interval",</div><div><span class="" style="white-space:pre">         </span>  .mode = S_IRUGO | S_IWUSR },</div><div><span class="" style="white-space:pre">     </span>.show = commit_super_block_interval_show,</div>
<div><span class="" style="white-space:pre">    </span>.store = commit_super_block_interval_store,</div><div>};</div><div><br></div><div>static ssize_t allow_migrate_show(struct lc_cache *cache, char *page)</div><div>{</div>
<div><span class="" style="white-space:pre">    </span>return var_show(cache->allow_migrate, (page));</div><div>}</div><div><br></div><div>static ssize_t allow_migrate_store(struct lc_cache *cache,</div><div><span class="" style="white-space:pre">                                </span>   const char *page, size_t count)</div>
<div>{</div><div><span class="" style="white-space:pre">    </span>unsigned long x;</div><div><span class="" style="white-space:pre">   </span>ssize_t r = var_store(&x, page, count);</div><div><span class="" style="white-space:pre">        </span>cache->allow_migrate = x;</div>
<div><span class="" style="white-space:pre">    </span>return r;</div><div>}</div><div><br></div><div>static struct cache_sysfs_entry allow_migrate_entry = {</div><div><span class="" style="white-space:pre">   </span>.attr = { .name = "allow_migrate", .mode = S_IRUGO | S_IWUSR },</div>
<div><span class="" style="white-space:pre">    </span>.show = allow_migrate_show,</div><div><span class="" style="white-space:pre">        </span>.store = allow_migrate_store,</div><div>};</div><div><br></div><div>static ssize_t force_migrate_show(struct lc_cache *cache, char *page)</div>
<div>{</div><div><span class="" style="white-space:pre">    </span>return var_show(cache->force_migrate, page);</div><div>}</div><div><br></div><div>static ssize_t force_migrate_store(struct lc_cache *cache,</div><div><span class="" style="white-space:pre">                          </span>   const char *page, size_t count)</div>
<div>{</div><div><span class="" style="white-space:pre">    </span>unsigned long x;</div><div><span class="" style="white-space:pre">   </span>ssize_t r = var_store(&x, page, count);</div><div><span class="" style="white-space:pre">        </span>cache->force_migrate = x;</div>
<div><span class="" style="white-space:pre">    </span>return r;</div><div>}</div><div><br></div><div>static struct cache_sysfs_entry force_migrate_entry = {</div><div><span class="" style="white-space:pre">   </span>.attr = { .name = "force_migrate", .mode = S_IRUGO | S_IWUSR },</div>
<div><span class="" style="white-space:pre">    </span>.show = force_migrate_show,</div><div><span class="" style="white-space:pre">        </span>.store = force_migrate_store,</div><div>};</div><div><br></div><div>static ssize_t update_interval_show(struct lc_cache *cache, char *page)</div>
<div>{</div><div><span class="" style="white-space:pre">    </span>return var_show(cache->update_interval, page);</div><div>}</div><div><br></div><div>static ssize_t update_interval_store(struct lc_cache *cache,</div><div>
<span class="" style="white-space:pre">                               </span>     const char *page, size_t count)</div><div>{</div><div><span class="" style="white-space:pre">       </span>unsigned long x;</div><div><span class="" style="white-space:pre">   </span>ssize_t r = var_store(&x, page, count);</div>
<div><span class="" style="white-space:pre">    </span>cache->update_interval = x;</div><div><span class="" style="white-space:pre">     </span>return r;</div><div>}</div><div><br></div><div>static struct cache_sysfs_entry update_interval_entry = {</div>
<div><span class="" style="white-space:pre">    </span>.attr = { .name = "update_interval", .mode = S_IRUGO | S_IWUSR },</div><div><span class="" style="white-space:pre">        </span>.show = update_interval_show,</div><div>
<span class="" style="white-space:pre">       </span>.store = update_interval_store,</div><div>};</div><div><br></div><div>static ssize_t flush_current_buffer_interval_show(struct lc_cache *cache,</div><div><span class="" style="white-space:pre">                                          </span>  char *page)</div>
<div>{</div><div><span class="" style="white-space:pre">    </span>return var_show(cache->flush_current_buffer_interval, page);</div><div>}</div><div><br></div><div>static ssize_t flush_current_buffer_interval_store(</div>
<div><span class="" style="white-space:pre">            </span>struct lc_cache *cache, const char *page, size_t count)</div><div>{</div><div><span class="" style="white-space:pre">    </span>unsigned long x;</div><div><span class="" style="white-space:pre">   </span>ssize_t r = var_store(&x, page, count);</div>
<div><span class="" style="white-space:pre">    </span>cache->flush_current_buffer_interval = x;</div><div><span class="" style="white-space:pre">       </span>return r;</div><div>}</div><div><br></div><div>static struct cache_sysfs_entry flush_current_buffer_interval_entry = {</div>
<div><span class="" style="white-space:pre">    </span>.attr = { .name = "flush_current_buffer_interval",</div><div><span class="" style="white-space:pre">               </span>  .mode = S_IRUGO | S_IWUSR },</div><div><span class="" style="white-space:pre">     </span>.show = flush_current_buffer_interval_show,</div>
<div><span class="" style="white-space:pre">    </span>.store = flush_current_buffer_interval_store,</div><div>};</div><div><br></div><div>static ssize_t commit_super_block_show(struct lc_cache *cache, char *page)</div><div>
{</div><div><span class="" style="white-space:pre">       </span>return var_show(0, (page));</div><div>}</div><div><br></div><div>static ssize_t commit_super_block_store(struct lc_cache *cache,</div><div><span class="" style="white-space:pre">                                 </span>const char *page, size_t count)</div>
<div>{</div><div><span class="" style="white-space:pre">    </span>unsigned long x;</div><div><span class="" style="white-space:pre">   </span>ssize_t r = var_store(&x, page, count);</div><div><br></div><div><span class="" style="white-space:pre">       </span>if (x < 1)</div>
<div><span class="" style="white-space:pre">            </span>return -EIO;</div><div><br></div><div><span class="" style="white-space:pre">      </span>mutex_lock(&cache->io_lock);</div><div><span class="" style="white-space:pre">        </span>commit_super_block(cache);</div>
<div><span class="" style="white-space:pre">    </span>mutex_unlock(&cache->io_lock);</div><div><br></div><div><span class="" style="white-space:pre">     </span>return r;</div><div>}</div><div><br></div><div>static struct cache_sysfs_entry commit_super_block_entry = {</div>
<div><span class="" style="white-space:pre">    </span>.attr = { .name = "commit_super_block", .mode = S_IRUGO | S_IWUSR },</div><div><span class="" style="white-space:pre">     </span>.show = commit_super_block_show,</div>
<div><span class="" style="white-space:pre">    </span>.store = commit_super_block_store,</div><div>};</div><div><br></div><div>static ssize_t flush_current_buffer_show(struct lc_cache *cache, char *page)</div><div>{</div><div>
<span class="" style="white-space:pre">       </span>return var_show(0, (page));</div><div>}</div><div><br></div><div>static ssize_t flush_current_buffer_store(struct lc_cache *cache,</div><div><span class="" style="white-space:pre">                                       </span>  const char *page, size_t count)</div>
<div>{</div><div><span class="" style="white-space:pre">    </span>unsigned long x;</div><div><br></div><div><span class="" style="white-space:pre">  </span>ssize_t r = var_store(&x, page, count);</div><div><span class="" style="white-space:pre">        </span>if (x < 1)</div>
<div><span class="" style="white-space:pre">            </span>return -EIO;</div><div><br></div><div><span class="" style="white-space:pre">      </span>flush_current_buffer_sync(cache);</div><div><span class="" style="white-space:pre">  </span>return r;</div>
<div>}</div><div><br></div><div>static struct cache_sysfs_entry flush_current_buffer_entry = {</div><div><span class="" style="white-space:pre">      </span>.attr = { .name = "flush_current_buffer", .mode = S_IRUGO | S_IWUSR },</div>
<div><span class="" style="white-space:pre">    </span>.show = flush_current_buffer_show,</div><div><span class="" style="white-space:pre"> </span>.store = flush_current_buffer_store,</div><div>};</div><div><br></div><div>static ssize_t last_flushed_segment_id_show(struct lc_cache *cache, char *page)</div>
<div>{</div><div><span class="" style="white-space:pre">    </span>return var_show(cache->last_flushed_segment_id, (page));</div><div>}</div><div><br></div><div>static struct cache_sysfs_entry last_flushed_segment_id_entry = {</div>
<div><span class="" style="white-space:pre">    </span>.attr = { .name = "last_flushed_segment_id", .mode = S_IRUGO },</div><div><span class="" style="white-space:pre">  </span>.show = last_flushed_segment_id_show,</div>
<div>};</div><div><br></div><div>static ssize_t last_migrated_segment_id_show(struct lc_cache *cache, char *page)</div><div>{</div><div><span class="" style="white-space:pre">   </span>return var_show(cache->last_migrated_segment_id, (page));</div>
<div>}</div><div><br></div><div>static struct cache_sysfs_entry last_migrated_segment_id_entry = {</div><div><span class="" style="white-space:pre">  </span>.attr = { .name = "last_migrated_segment_id", .mode = S_IRUGO },</div>
<div><span class="" style="white-space:pre">    </span>.show = last_migrated_segment_id_show,</div><div>};</div><div><br></div><div>static ssize_t barrier_deadline_ms_show(struct lc_cache *cache, char *page)</div><div>{</div>
<div><span class="" style="white-space:pre">    </span>return var_show(cache->barrier_deadline_ms, (page));</div><div>}</div><div><br></div><div>static ssize_t barrier_deadline_ms_store(struct lc_cache *cache,</div><div><span class="" style="white-space:pre">                                    </span> const char *page, size_t count)</div>
<div>{</div><div><span class="" style="white-space:pre">    </span>unsigned long x;</div><div><span class="" style="white-space:pre">   </span>ssize_t r = var_store(&x, page, count);</div><div><br></div><div><span class="" style="white-space:pre">       </span>cache->barrier_deadline_ms = x;</div>
<div><span class="" style="white-space:pre">    </span>return r;</div><div>}</div><div><br></div><div>static struct cache_sysfs_entry barrier_deadline_ms_entry = {</div><div><span class="" style="white-space:pre">     </span>.attr = { .name = "barrier_deadline_ms", .mode = S_IRUGO | S_IWUSR },</div>
<div><span class="" style="white-space:pre">    </span>.show = barrier_deadline_ms_show,</div><div><span class="" style="white-space:pre">  </span>.store = barrier_deadline_ms_store,</div><div>};</div><div><br></div><div>static struct attribute *cache_default_attrs[] = {</div>
<div><span class="" style="white-space:pre">    </span>&commit_super_block_interval_entry.attr,</div><div><span class="" style="white-space:pre">       </span>&allow_migrate_entry.attr,</div><div><span class="" style="white-space:pre">     </span>&commit_super_block_entry.attr,</div>
<div><span class="" style="white-space:pre">    </span>&flush_current_buffer_entry.attr,</div><div><span class="" style="white-space:pre">      </span>&flush_current_buffer_interval_entry.attr,</div><div><span class="" style="white-space:pre">     </span>&force_migrate_entry.attr,</div>
<div><span class="" style="white-space:pre">    </span>&update_interval_entry.attr,</div><div><span class="" style="white-space:pre">   </span>&last_flushed_segment_id_entry.attr,</div><div><span class="" style="white-space:pre">   </span>&last_migrated_segment_id_entry.attr,</div>
<div><span class="" style="white-space:pre">    </span>&barrier_deadline_ms_entry.attr,</div><div><span class="" style="white-space:pre">       </span>NULL,</div><div>};</div><div><br></div><div>static const struct sysfs_ops cache_sysfs_ops = {</div>
<div><span class="" style="white-space:pre">    </span>.show = cache_attr_show,</div><div><span class="" style="white-space:pre">   </span>.store = cache_attr_store,</div><div>};</div><div><br></div><div>static void cache_release(struct kobject *kobj)</div>
<div>{</div><div><span class="" style="white-space:pre">    </span>return;</div><div>}</div><div><br></div><div>static struct kobj_type cache_ktype = {</div><div><span class="" style="white-space:pre">     </span>.sysfs_ops = &cache_sysfs_ops,</div>
<div><span class="" style="white-space:pre">    </span>.default_attrs = cache_default_attrs,</div><div><span class="" style="white-space:pre">      </span>.release = cache_release,</div><div>};</div><div><br></div><div>static int lc_mgr_message(struct dm_target *ti, unsigned int argc, char **argv)</div>
<div>{</div><div><span class="" style="white-space:pre">    </span>char *cmd = argv[0];</div><div><br></div><div><span class="" style="white-space:pre">      </span>/*</div><div><span class="" style="white-space:pre"> </span> * <path></div>
<div><span class="" style="white-space:pre">    </span> * @path path to the cache device</div><div><span class="" style="white-space:pre">  </span> */</div><div><span class="" style="white-space:pre">        </span>if (!strcasecmp(cmd, "format_cache_device")) {</div>
<div><span class="" style="white-space:pre">            </span>struct dm_dev *dev;</div><div><span class="" style="white-space:pre">                </span>if (dm_get_device(ti, argv[1],</div><div><span class="" style="white-space:pre">                             </span>  dm_table_get_mode(ti->table), &dev))</div>
<div><span class="" style="white-space:pre">                    </span>return -EINVAL;</div><div><br></div><div><span class="" style="white-space:pre">           </span>format_cache_device(dev);</div><div><br></div><div><span class="" style="white-space:pre">         </span>dm_put_device(ti, dev);</div>
<div><span class="" style="white-space:pre">            </span>return 0;</div><div><span class="" style="white-space:pre">  </span>}</div><div><br></div><div><span class="" style="white-space:pre"> </span>/*</div><div><span class="" style="white-space:pre"> </span> * <id></div>
<div><span class="" style="white-space:pre">    </span> *</div><div><span class="" style="white-space:pre"> </span> * lc-mgr has cursor to point the</div><div><span class="" style="white-space:pre">  </span> * cache device to operate.</div>
<div><span class="" style="white-space:pre">    </span> */</div><div><span class="" style="white-space:pre">        </span>if (!strcasecmp(cmd, "switch_to")) {</div><div><span class="" style="white-space:pre">             </span>unsigned id;</div>
<div><span class="" style="white-space:pre">            </span>if (sscanf(argv[1], "%u", &id) != 1)</div><div><span class="" style="white-space:pre">                 </span>return -EINVAL;</div><div><br></div><div><span class="" style="white-space:pre">           </span>cache_id_ptr = id;</div>
<div><span class="" style="white-space:pre">            </span>return 0;</div><div><span class="" style="white-space:pre">  </span>}</div><div><br></div><div><span class="" style="white-space:pre"> </span>if (!strcasecmp(cmd, "clear_stat")) {</div>
<div><span class="" style="white-space:pre">            </span>struct lc_cache *cache = lc_caches[cache_id_ptr];</div><div><span class="" style="white-space:pre">          </span>if (!cache)</div><div><span class="" style="white-space:pre">                        </span>return -EINVAL;</div>
<div><br></div><div><span class="" style="white-space:pre">           </span>clear_stat(cache);</div><div><span class="" style="white-space:pre">         </span>return 0;</div><div><span class="" style="white-space:pre">  </span>}</div><div>
<br></div><div><span class="" style="white-space:pre">      </span>/*</div><div><span class="" style="white-space:pre"> </span> * <path></div><div><span class="" style="white-space:pre">    </span> */</div><div><span class="" style="white-space:pre">        </span>if (!strcasecmp(cmd, "resume_cache")) {</div>
<div><span class="" style="white-space:pre">            </span>struct lc_cache *cache = kzalloc(sizeof(*cache), GFP_KERNEL);</div><div><br></div><div><span class="" style="white-space:pre">             </span>struct dm_dev *dev;</div><div><span class="" style="white-space:pre">                </span>if (dm_get_device(ti, argv[1],</div>
<div><span class="" style="white-space:pre">                            </span>  dm_table_get_mode(ti->table), &dev))</div><div><span class="" style="white-space:pre">                      </span>return -EINVAL;</div><div><br></div><div><span class="" style="white-space:pre">           </span>cache->id = cache_id_ptr;</div>
<div><span class="" style="white-space:pre">            </span>cache->device = dev;</div><div><span class="" style="white-space:pre">            </span>cache->nr_segments = calc_nr_segments(cache->device);</div><div><span class="" style="white-space:pre">                </span>cache->nr_caches = cache->nr_segments * NR_CACHES_INSEG;</div>
<div><br></div><div><span class="" style="white-space:pre">           </span>mutex_init(&cache->io_lock);</div><div><br></div><div><span class="" style="white-space:pre">               </span>cache->wb_pool = kmalloc(</div><div><span class="" style="white-space:pre">                       </span>sizeof(struct writebuffer) * NR_WB_POOL, GFP_KERNEL);</div>
<div><span class="" style="white-space:pre">            </span>struct writebuffer *wb;</div><div><span class="" style="white-space:pre">            </span>int i;</div><div><span class="" style="white-space:pre">             </span>for (i = 0; i < NR_WB_POOL; i++) {</div>
<div><span class="" style="white-space:pre">                    </span>wb = cache->wb_pool + i;</div><div><span class="" style="white-space:pre">                        </span>init_completion(&wb->done);</div><div><span class="" style="white-space:pre">                 </span>complete_all(&wb->done);</div>
<div><br></div><div><span class="" style="white-space:pre">                   </span>wb->data = kmalloc(</div><div><span class="" style="white-space:pre">                             </span>1 << (LC_SEGMENTSIZE_ORDER + SECTOR_SHIFT),</div><div><span class="" style="white-space:pre">                          </span>GFP_KERNEL);</div>
<div><span class="" style="white-space:pre">            </span>}</div><div><br></div><div><span class="" style="white-space:pre">         </span>/*</div><div><span class="" style="white-space:pre">         </span> * Select arbitrary one</div><div>
<span class="" style="white-space:pre">               </span> * as the initial writebuffer.</div><div><span class="" style="white-space:pre">             </span> */</div><div><span class="" style="white-space:pre">                </span>cache->current_wb = cache->wb_pool + 0;</div>
<div><br></div><div><span class="" style="white-space:pre">           </span>init_segment_header_array(cache);</div><div><span class="" style="white-space:pre">          </span>mb_array_empty_init(cache);</div><div><span class="" style="white-space:pre">                </span>ht_empty_init(cache);</div>
<div><br></div><div><span class="" style="white-space:pre">           </span>cache->on_terminate = false;</div><div><span class="" style="white-space:pre">            </span>cache->allow_migrate = false;</div><div><span class="" style="white-space:pre">           </span>cache->force_migrate = false;</div>
<div><span class="" style="white-space:pre">            </span>cache->reserving_segment_id = 0;</div><div><br></div><div><span class="" style="white-space:pre">               </span>cache->flush_wq = create_singlethread_workqueue("flushwq");</div>
<div><span class="" style="white-space:pre">            </span>spin_lock_init(&cache->flush_queue_lock);</div><div><span class="" style="white-space:pre">           </span>INIT_WORK(&cache->flush_work, flush_proc);</div><div><span class="" style="white-space:pre">          </span>INIT_LIST_HEAD(&cache->flush_queue);</div>
<div><span class="" style="white-space:pre">            </span>init_waitqueue_head(&cache->flush_wait_queue);</div><div><span class="" style="white-space:pre">              </span>queue_work(cache->flush_wq, &cache->flush_work);</div>
<div><br></div><div><span class="" style="white-space:pre">           </span>cache->migrate_wq = create_singlethread_workqueue("migratewq");</div><div><span class="" style="white-space:pre">               </span>INIT_WORK(&cache->migrate_work, migrate_proc);</div>
<div><span class="" style="white-space:pre">            </span>queue_work(cache->migrate_wq, &cache->migrate_work);</div><div><br></div><div><span class="" style="white-space:pre">            </span>init_waitqueue_head(&cache->migrate_wait_queue);</div>
<div><span class="" style="white-space:pre">            </span>atomic_set(&cache->migrate_fail_count, 0);</div><div><span class="" style="white-space:pre">          </span>atomic_set(&cache->migrate_io_count, 0);</div><div><span class="" style="white-space:pre">            </span>cache->migrate_buffer = kmalloc(</div>
<div><span class="" style="white-space:pre">                            </span>1 << (LC_SEGMENTSIZE_ORDER + SECTOR_SHIFT),</div><div><span class="" style="white-space:pre">                          </span>GFP_KERNEL);</div><div><br></div><div><span class="" style="white-space:pre">              </span>setup_timer(&cache->barrier_deadline_timer,</div>
<div><span class="" style="white-space:pre">                    </span>    barrier_deadline_proc, (unsigned long) cache);</div><div><span class="" style="white-space:pre">         </span>bio_list_init(&cache->barrier_ios);</div><div><br></div>
<div><span class="" style="white-space:pre">            </span>/*</div><div><span class="" style="white-space:pre">         </span> * deadline is 3 ms by default.</div><div><span class="" style="white-space:pre">            </span> * 2.5 us to process on bio</div>
<div><span class="" style="white-space:pre">            </span> * and 3 ms is enough long to process 255 bios.</div><div><span class="" style="white-space:pre">            </span> * If the buffer doesn't get full within 3 ms,</div><div><span class="" style="white-space:pre">         </span> * we can doubt write starves</div>
<div><span class="" style="white-space:pre">            </span> * by waiting formerly submitted barrier to be complete.</div><div><span class="" style="white-space:pre">           </span> */</div><div><span class="" style="white-space:pre">                </span>cache->barrier_deadline_ms = 3;</div>
<div><span class="" style="white-space:pre">            </span>INIT_WORK(&cache->barrier_deadline_work, flush_barrier_ios);</div><div><br></div><div><span class="" style="white-space:pre">               </span>recover_cache(cache);</div>
<div><span class="" style="white-space:pre">            </span>lc_caches[cache->id] = cache;</div><div><br></div><div><span class="" style="white-space:pre">          </span>clear_stat(cache);</div><div><br></div><div><span class="" style="white-space:pre">                </span>/*</div>
<div><span class="" style="white-space:pre">            </span> * /sys/module/dm_lc/caches/$id/$attribute</div><div><span class="" style="white-space:pre">         </span> *                             /device -> /sys/block/$name</div>
<div><span class="" style="white-space:pre">            </span> */</div><div><br></div><div><span class="" style="white-space:pre">               </span>int r;</div><div><br></div><div><span class="" style="white-space:pre">            </span>cache->update_interval = 1;</div>
<div><span class="" style="white-space:pre">            </span>cache->commit_super_block_interval = 0;</div><div><span class="" style="white-space:pre">         </span>cache->flush_current_buffer_interval = 0;</div><div><span class="" style="white-space:pre">               </span>r = kobject_init_and_add(&cache->kobj, &cache_ktype,</div>
<div><span class="" style="white-space:pre">                                    </span> caches_kobj, "%u", cache->id);</div><div><br></div><div><span class="" style="white-space:pre">              </span>struct kobject *dev_kobj =</div><div><span class="" style="white-space:pre">                 </span>get_bdev_kobject(cache->device->bdev);</div>
<div><span class="" style="white-space:pre">            </span>r = sysfs_create_link(&cache->kobj, dev_kobj, "device");</div><div><br></div><div><span class="" style="white-space:pre">         </span>return 0;</div><div><span class="" style="white-space:pre">  </span>}</div>
<div><br></div><div><span class="" style="white-space:pre">   </span>if (!strcasecmp(cmd, "free_cache")) {</div><div><span class="" style="white-space:pre">            </span>cache_id id = cache_id_ptr;</div><div><span class="" style="white-space:pre">                </span>struct lc_cache *cache = lc_caches[id];</div>
<div><br></div><div><span class="" style="white-space:pre">           </span>cache->on_terminate = true;</div><div><br></div><div><span class="" style="white-space:pre">            </span>cancel_work_sync(&cache->flush_work);</div>
<div><span class="" style="white-space:pre">            </span>destroy_workqueue(cache->flush_wq);</div><div><br></div><div><span class="" style="white-space:pre">            </span>cancel_work_sync(&cache->barrier_deadline_work);</div>
<div><br></div><div><span class="" style="white-space:pre">           </span>kfree(cache->migrate_buffer);</div><div><span class="" style="white-space:pre">           </span>cancel_work_sync(&cache->migrate_work);</div><div><span class="" style="white-space:pre">             </span>destroy_workqueue(cache->migrate_wq);</div>
<div><br></div><div><span class="" style="white-space:pre">           </span>size_t i;</div><div><span class="" style="white-space:pre">          </span>struct writebuffer *wb;</div><div><span class="" style="white-space:pre">            </span>for (i = 0; i < NR_WB_POOL; i++) {</div>
<div><span class="" style="white-space:pre">                    </span>wb = cache->wb_pool + i;</div><div><span class="" style="white-space:pre">                        </span>kfree(wb->data);</div><div><span class="" style="white-space:pre">                </span>}</div>
<div><span class="" style="white-space:pre">            </span>kfree(cache->wb_pool);</div><div><br></div><div><span class="" style="white-space:pre">         </span>kill_arr(cache->htable);</div><div><span class="" style="white-space:pre">                </span>kill_arr(cache->segment_header_array);</div>
<div><br></div><div><span class="" style="white-space:pre">           </span>sysfs_remove_link(&cache->kobj, "device");</div><div><span class="" style="white-space:pre">                </span>kobject_del(&cache->kobj);</div>
<div><span class="" style="white-space:pre">            </span>kobject_put(&cache->kobj);</div><div><br></div><div><span class="" style="white-space:pre">         </span>dm_put_device(ti, cache->device);</div><div><span class="" style="white-space:pre">               </span>kfree(cache);</div>
<div><br></div><div><span class="" style="white-space:pre">           </span>lc_caches[id] = NULL;</div><div><br></div><div><span class="" style="white-space:pre">             </span>return 0;</div><div><span class="" style="white-space:pre">  </span>}</div>
<div><br></div><div><span class="" style="white-space:pre">   </span>return -EINVAL;</div><div>}</div><div><br></div><div>static size_t calc_static_memory_consumption(struct lc_cache *cache)</div><div>{</div><div><span class="" style="white-space:pre">        </span>size_t seg = sizeof(struct segment_header) * cache->nr_segments;</div>
<div><span class="" style="white-space:pre">    </span>size_t ht = sizeof(struct ht_head) * cache->htsize;</div><div><br></div><div><span class="" style="white-space:pre">    </span>return seg + ht;</div><div>};</div><div><br>
</div><div>static</div><div>#if LINUX_VERSION_CODE >= KERNEL_VERSION(3, 8, 0)</div><div>void</div><div>#else</div><div>int</div><div>#endif</div><div>lc_mgr_status(</div><div><span class="" style="white-space:pre">             </span>struct dm_target *ti, status_type_t type,</div>
<div>#if LINUX_VERSION_CODE >= KERNEL_VERSION(3, 6, 0)</div><div><span class="" style="white-space:pre">         </span>unsigned flags,</div><div>#endif</div><div><span class="" style="white-space:pre">               </span>char *result, unsigned int maxlen)</div>
<div>{</div><div><span class="" style="white-space:pre">    </span>unsigned int sz = 0;</div><div><br></div><div><span class="" style="white-space:pre">      </span>switch (type) {</div><div><span class="" style="white-space:pre">    </span>case STATUSTYPE_INFO:</div>
<div><span class="" style="white-space:pre">            </span>DMEMIT("\n");</div><div><span class="" style="white-space:pre">            </span>DMEMIT("current cache_id_ptr: %u\n", cache_id_ptr);</div><div><br></div><div>
<span class="" style="white-space:pre">               </span>if (cache_id_ptr == 0) {</div><div><span class="" style="white-space:pre">                   </span>DMEMIT(</div><div><span class="" style="white-space:pre">                    </span>"sizeof(struct metablock): %lu\n",</div>
<div><span class="" style="white-space:pre">                    </span>       sizeof(struct metablock));</div><div><span class="" style="white-space:pre">                  </span>DMEMIT(</div><div><span class="" style="white-space:pre">                    </span>"sizeof(struct metablock_device): %lu\n",</div>
<div><span class="" style="white-space:pre">                    </span>       sizeof(struct metablock_device));</div><div><span class="" style="white-space:pre">                   </span>DMEMIT(</div><div><span class="" style="white-space:pre">                    </span>"sizeof(struct segment_header): %lu\n",</div>
<div><span class="" style="white-space:pre">                    </span>       sizeof(struct segment_header));</div><div><span class="" style="white-space:pre">                     </span>DMEMIT(</div><div><span class="" style="white-space:pre">                    </span>"sizeof(struct segment_header_device): %lu (<= 4096)",</div>
<div><span class="" style="white-space:pre">                    </span>       sizeof(struct segment_header_device));</div><div><span class="" style="white-space:pre">                      </span>break;</div><div><span class="" style="white-space:pre">             </span>}</div>
<div><br></div><div><span class="" style="white-space:pre">           </span>struct lc_cache *cache = lc_caches[cache_id_ptr];</div><div><span class="" style="white-space:pre">          </span>if (!cache)</div><div>#if LINUX_VERSION_CODE < KERNEL_VERSION(3, 8, 0)</div>
<div><span class="" style="white-space:pre">                    </span>return -EINVAL;</div><div>#else</div><div><span class="" style="white-space:pre">                        </span>return;</div><div>#endif</div><div><br></div><div><span class="" style="white-space:pre">              </span>DMEMIT("static RAM(approx.): %lu (byte)\n",</div>
<div><span class="" style="white-space:pre">            </span>       calc_static_memory_consumption(cache));</div><div><span class="" style="white-space:pre">             </span>DMEMIT("allow_migrate: %d\n", cache->allow_migrate);</div>
<div><span class="" style="white-space:pre">            </span>DMEMIT("nr_segments: %lu\n", cache->nr_segments);</div><div><span class="" style="white-space:pre">             </span>DMEMIT("last_migrated_segment_id: %lu\n",</div>
<div><span class="" style="white-space:pre">            </span>       cache->last_migrated_segment_id);</div><div><span class="" style="white-space:pre">                </span>DMEMIT("last_flushed_segment_id: %lu\n",</div><div><span class="" style="white-space:pre">         </span>       cache->last_flushed_segment_id);</div>
<div><span class="" style="white-space:pre">            </span>DMEMIT("current segment id: %lu\n",</div><div><span class="" style="white-space:pre">              </span>       cache->current_seg->global_id);</div><div><span class="" style="white-space:pre">               </span>DMEMIT("cursor: %u\n", cache->cursor);</div>
<div><span class="" style="white-space:pre">            </span>DMEMIT("\n");</div><div><span class="" style="white-space:pre">            </span>DMEMIT("write? hit? on_buffer? fullsize?\n");</div><div><span class="" style="white-space:pre">            </span>int i;</div>
<div><span class="" style="white-space:pre">            </span>for (i = 0; i < STATLEN; i++) {</div><div><span class="" style="white-space:pre">                 </span>if (i == (STATLEN-1))</div><div><span class="" style="white-space:pre">                              </span>break;</div>
<div><br></div><div><span class="" style="white-space:pre">                   </span>atomic64_t *v = &cache->stat[i];</div><div><span class="" style="white-space:pre">                    </span>DMEMIT("%d %d %d %d %lu",</div><div><span class="" style="white-space:pre">                                </span>i & (1 << STAT_WRITE)      ? 1 : 0,</div>
<div><span class="" style="white-space:pre">                            </span>i & (1 << STAT_HIT)        ? 1 : 0,</div><div><span class="" style="white-space:pre">                              </span>i & (1 << STAT_ON_BUFFER)  ? 1 : 0,</div><div><span class="" style="white-space:pre">                              </span>i & (1 << STAT_FULLSIZE)   ? 1 : 0,</div>
<div><span class="" style="white-space:pre">                            </span>atomic64_read(v));</div><div><span class="" style="white-space:pre">                 </span>DMEMIT("\n");</div><div><span class="" style="white-space:pre">            </span>}</div>
<div><span class="" style="white-space:pre">            </span>break;</div><div><br></div><div><span class="" style="white-space:pre">    </span>case STATUSTYPE_TABLE:</div><div><span class="" style="white-space:pre">             </span>break;</div>
<div><span class="" style="white-space:pre">    </span>}</div><div><br></div><div>#if LINUX_VERSION_CODE < KERNEL_VERSION(3, 8, 0)</div><div><span class="" style="white-space:pre">       </span>return 0;</div><div>#endif</div><div>
}</div><div><br></div><div>static struct target_type lc_mgr_target = {</div><div><span class="" style="white-space:pre">    </span>.name = "lc-mgr",</div><div><span class="" style="white-space:pre">        </span>.version = {1, 0, 0},</div>
<div><span class="" style="white-space:pre">    </span>.module = THIS_MODULE,</div><div><span class="" style="white-space:pre">     </span>.map = lc_mgr_map,</div><div><span class="" style="white-space:pre"> </span>.ctr = lc_mgr_ctr,</div>
<div><span class="" style="white-space:pre">    </span>.dtr = lc_mgr_dtr,</div><div><span class="" style="white-space:pre"> </span>.message = lc_mgr_message,</div><div><span class="" style="white-space:pre"> </span>.status = lc_mgr_status,</div>
<div>};</div><div><br></div><div>static int __init lc_module_init(void)</div><div>{</div><div><span class="" style="white-space:pre">     </span>int r;</div><div><br></div><div><span class="" style="white-space:pre">    </span>safe_io_wq = alloc_workqueue("safeiowq",</div>
<div><span class="" style="white-space:pre">                            </span>     WQ_NON_REENTRANT | WQ_MEM_RECLAIM, 0);</div><div><br></div><div><span class="" style="white-space:pre">       </span>lc_io_client = dm_io_client_create();</div><div><br>
</div><div><span class="" style="white-space:pre">        </span>r = dm_register_target(&lc_target);</div><div><span class="" style="white-space:pre">    </span>if (r < 0) {</div><div><span class="" style="white-space:pre">            </span>DMERR("register lc failed %d", r);</div>
<div><span class="" style="white-space:pre">            </span>return r;</div><div><span class="" style="white-space:pre">  </span>}</div><div><br></div><div><span class="" style="white-space:pre"> </span>r = dm_register_target(&lc_mgr_target);</div>
<div><span class="" style="white-space:pre">    </span>if (r < 0) {</div><div><span class="" style="white-space:pre">            </span>DMERR("register lc-mgr failed %d", r);</div><div><span class="" style="white-space:pre">           </span>return r;</div>
<div><span class="" style="white-space:pre">    </span>}</div><div><br></div><div><span class="" style="white-space:pre"> </span>cache_id_ptr = 0;</div><div><br></div><div><span class="" style="white-space:pre"> </span>size_t i;</div>
<div><span class="" style="white-space:pre">    </span>for (i = 0; i < LC_NR_SLOTS; i++)</div><div><span class="" style="white-space:pre">               </span>lc_devices[i] = NULL;</div><div><br></div><div><span class="" style="white-space:pre">     </span>for (i = 0; i < LC_NR_SLOTS; i++)</div>
<div><span class="" style="white-space:pre">            </span>lc_caches[i] = NULL;</div><div><br></div><div><span class="" style="white-space:pre">      </span>/*</div><div><span class="" style="white-space:pre"> </span> * /sys/module/dm_lc/devices</div>
<div><span class="" style="white-space:pre">    </span> *                  /caches</div><div><span class="" style="white-space:pre">        </span> */</div><div><br></div><div><span class="" style="white-space:pre">       </span>struct module *mod = THIS_MODULE;</div>
<div><span class="" style="white-space:pre">    </span>struct kobject *lc_kobj = &(mod->mkobj.kobj);</div><div><span class="" style="white-space:pre">       </span>devices_kobj = kobject_create_and_add("devices", lc_kobj);</div>
<div><span class="" style="white-space:pre">    </span>caches_kobj = kobject_create_and_add("caches", lc_kobj);</div><div><br></div><div><span class="" style="white-space:pre">        </span>return 0;</div><div>}</div><div>
<br></div><div>static void __exit lc_module_exit(void)</div><div>{</div><div><span class="" style="white-space:pre">        </span>destroy_workqueue(safe_io_wq);</div><div><span class="" style="white-space:pre">     </span>dm_io_client_destroy(lc_io_client);</div>
<div><br></div><div><span class="" style="white-space:pre">   </span>kobject_put(devices_kobj);</div><div><span class="" style="white-space:pre"> </span>kobject_put(caches_kobj);</div><div><br></div><div><span class="" style="white-space:pre"> </span>dm_unregister_target(&lc_mgr_target);</div>
<div><span class="" style="white-space:pre">    </span>dm_unregister_target(&lc_target);</div><div>}</div><div><br></div><div>module_init(lc_module_init);</div><div>module_exit(lc_module_exit);</div><div><br></div><div>MODULE_AUTHOR("Akira Hayakawa <<a href="mailto:ruby.wktk@gmail.com">ruby.wktk@gmail.com</a>>");</div>
<div>MODULE_DESCRIPTION(DM_NAME " lc target");</div><div>MODULE_LICENSE("GPL");</div><div><br></div><div><br></div><div><br></div><div>(dm-lc.txt)</div><div><br></div><div>dm-lc</div><div>=====</div><div>
<br></div><div>dm-lc provides write-back log-structured caching.</div><div>It batches random writes into a big sequential write.</div><div><br></div><div>1. Setup</div><div>========</div><div>dm-lc is composed of two target_type instances named lc and lc-mgr.</div>
<div>lc is responsible for creating logical volumes and controlling ios and</div><div>lc-mgr is reponsible for managing</div><div>formatting/initializing/destructing cache devices on the other hand.</div><div>Operating dm-lc through these native interfaces are not recommended.</div>
<div><br></div><div>To easily get started with dm-lc, nice userland tools are provided in</div><div><span class="" style="white-space:pre">       </span><a href="https://github.com/akiradeveloper/dm-lc">https://github.com/akiradeveloper/dm-lc</a></div>
<div>where you are also accesible to portable dm-lc kernel code</div><div>that supports since 3.2 kernel until before dm-lc staged upstream, 3.x.</div><div>please git clone it.</div><div><br></div><div>To install the tools, move under Admin directory and run</div>
<div><span class="" style="white-space:pre">    </span>python setup.py install</div><div>and now you have a lisence for dm-lc admin.</div><div><br></div><div>2. Example scripts</div><div>==================</div><div>Let's create a logical volume named myLV</div>
<div>backed by /dev/myVg/myBacking</div><div>and use /dev/myCache as a cache device.</div><div><br></div><div>myLV -- (backing store) /dev/myVg/myBacking</div><div>     -- (cache device)  /dev/myCache</div><div><br></div>
<div>Note that backing store is restricted to</div><div>a logical device that is created by LVM</div><div>for technical reasons.</div><div><br></div><div>1. Format myCache</div><div>Format the medata blocks on a device.</div>
<div>Note that this erases all the existing data.</div><div><br></div><div><span class="" style="white-space:pre">        </span>lc-format-cache /dev/myCache</div><div><br></div><div>2. Create myLV</div><div>Create a logical volume just backed by a existing volume.</div>
<div>We give device id 5 to the volume in this example.</div><div><br></div><div>As of now, this operation create a logical volume</div><div>with different name from the backing store.</div><div>But some users don't want to change the name</div>
<div>because the backing store is in use</div><div>and want to apply dm-lc on the fly.</div><div>This can be technically realizable</div><div>but I haven't implemented it at this time</div><div>because it is too tricky to be portable.</div>
<div><br></div><div><span class="" style="white-space:pre">   </span>lc-create myLV 5 /dev/myVg/myBacking</div><div><br></div><div>3. Resume myCache</div><div>Resuming cache device builds in-memory structures</div><div>such as a hashtable scanned from the metadata on the device.</div>
<div>We give cache id 3 to the device in this example.</div><div><br></div><div>Note that you MUST create all the LVs as the destinations</div><div>of the dirty caches on the cache device for technical reasons.</div><div>
Otherwise, it leads to kernel crash. Be careful.</div><div><br></div><div><span class="" style="white-space:pre">       </span>lc-resume 3 /dev/myCache</div><div><br></div><div>4. Attach myCache to myLV</div><div>To start caching writes to the myLV, you must attach myCache to myLV.</div>
<div>This can be done on the fly.</div><div><br></div><div><span class="" style="white-space:pre">        </span>lc-attach 5 3</div><div><br></div><div>5. Start userland daemon</div><div>dm-lc provides daemon program</div><div>that autonomously control the module behavior</div>
<div>such as write-back from myCache to myBacking</div><div>which dm-lc calls "Migration".</div><div><br></div><div><span class="" style="white-space:pre"> </span>lc-daemon start</div><div><br></div><div>6. Terminate myLV</div>
<div>Safely terminating myLV already attached to myCache</div><div>is easy to mistake and that's why dm-lc provides these admin tools.</div><div>myLV can not detach from myCache</div><div>until all the dirty caches on myCache</div>
<div>are migrated to myBacking</div><div><br></div><div><span class="" style="white-space:pre">   </span>lc-detach 5</div><div><span class="" style="white-space:pre">        </span>lc-remove 5</div><div><br></div><div>7. Terminate myCache</div>
<div>After terminate all the LVs that is attached</div><div>to myCache. myCache can be terminated.</div><div><br></div><div><span class="" style="white-space:pre">   </span>lc-daemon stop</div><div><span class="" style="white-space:pre">     </span>lc-free-cache 3</div>
<div><br></div><div>3. Sysfs</div><div>========</div><div>dm-lc provides some sysfs interfaces to control the module behavior.</div><div>The sysfs tree is located under /sys/module/dm_lc.</div><div><br></div><div>/sys/module/dm_lc</div>
<div>|</div><div>|-- devices</div><div>|   `-- 5</div><div>|       |-- cache_id</div><div>|       |-- dev</div><div>|       |-- device -> ../../../../devices/virtual/block/dm-0</div><div>|       |-- migrate_threshold</div>
<div>|       |-- nr_dirty_caches</div><div>|</div><div>|-- caches</div><div>|   `-- 3</div><div>|       |-- allow_migrate</div><div>|       |-- barrier_deadline_ms</div><div>|       |-- commit_super_block</div><div>|       |-- commit_super_block_interval</div>
<div>|       |-- device -> ../../../../devices/virtual/block/dm-1</div><div>|       |-- flush_current_buffer</div><div>|       |-- flush_current_buffer_interval</div><div>|       |-- force_migrate</div><div>|       |-- last_flushed_segment_id</div>
<div>|       |-- last_migrated_segment_id</div><div>|       `-- update_interval</div><div><br></div><div>4. Technical Issues</div><div>===================</div><div>There are not a few technical issues</div><div>that distinguishes dm-lc from other cache softwares.</div>
<div><br></div><div>4.1 RAM buffer and immediate completion</div><div>dm-lc allocated RAM buffers of 64MB in total by default.</div><div>All of the writes are first stored in one of these RAM buffers</div><div>and immediate completion is notified to the upper layer</div>
<div>that is usually in few microseconds that is unimaginably fast.</div><div><br></div><div>4.2 Metadata durability</div><div>After RAM buffer gets full or some deadline comes</div><div>dm-lc creates segment log that gathers RAM buffer and its metadata.</div>
<div>Metadata have information such as connection between</div><div>address in cache device and the counterpart in backing store.</div><div>As the segment log finally is written to persistent cache device,</div><div>any data will not be lost after machine failure.</div>
<div><br></div><div>4.3 Asynchronous log flushing</div><div>dm-lc has a background worker called flush daemon.</div><div>Flushing segment log starts from simply queueing the flush task.</div><div>Flush daemon in background asynchronously checks if the queue has some tasks</div>
<div>and actually executes the tasks if exists.</div><div>The fact that the upper layer doesn't block in queueing the task</div><div>maximizes the write throughput</div><div>that is 259MB/s random writes with cache device of 266MB/s sequential write</div>
<div>which is only 3% loss</div><div>and 1.5GB/s theoritically with a fast enough cache like PCI-e SSDs.</div><div><br></div><div>4.4 Asynchronous and automated migration</div><div>Some time after a log segment is flushed to cache device</div>
<div>it will be migrated to backing store.</div><div>Migrate daemon is also a background worker</div><div>that periodically check if log segments to migrate exist.</div><div><br></div><div>Restlessly migrating highly loads backing store</div>
<div>so migration is better to execute when the backing store is in lazy time.</div><div>lc-daemon in userland surveils the load of backing store</div><div>and turns migration on and off according to the load.</div><div><br>
</div><div>4.5 Lazy handling of REQ_FUA and REQ_FLUSH bios</div><div>Some applications such as NFS, journal filesystems</div><div>and databases often submit SYNC write that</div><div>leads to bios flagged with REQ_FUA or REQ_FLUSH.</div>
<div>Handling these irregular bios immediately and thus synchronously</div><div>desparately deteriorates the whole throughput.</div><div>To address this issue, dm-lc handles these bios lazily or in deferred manner.</div><div>
Completion related to these bios will not be done until</div><div>they are written persistently to the cache device</div><div>so this storategy doesn't break the semantics.</div><div>In the worst case scenario, a bio with some of these flags</div>
<div>is completed in deadline period that is described</div><div>in barrier_deadline_ms in sysfs.</div><div><br></div><div><br></div><div>Thanks for reading,</div><div>Akira</div></div>