[lvm-devel] LVM2/doc lvm_fault_handling.txt
jbrassow at sourceware.org
jbrassow at sourceware.org
Mon Jul 26 20:31:54 UTC 2010
CVSROOT: /cvs/lvm2
Module name: LVM2
Changes by: jbrassow at sourceware.org 2010-07-26 20:31:54
Added files:
doc : lvm_fault_handling.txt
Log message:
Initial import of document describing LVM's policies
surrounding device faults/failures.
Patches:
http://sourceware.org/cgi-bin/cvsweb.cgi/LVM2/doc/lvm_fault_handling.txt.diff?cvsroot=lvm2&r1=NONE&r2=1.1
/cvs/lvm2/LVM2/doc/lvm_fault_handling.txt,v --> standard output
revision 1.1
--- LVM2/doc/lvm_fault_handling.txt
+++ - 2010-07-26 20:31:54.280508000 +0000
@@ -0,0 +1,221 @@
+LVM device fault handling
+=========================
+
+Introduction
+------------
+This document is to serve as the definitive source for information
+regarding the policies and procedures surrounding device failures
+in LVM. It codifies LVM's responses to device failures as well as
+the responsibilities of administrators.
+
+Device failures can be permanent or transient. A permanent failure
+is one where a device becomes inaccessible and will never be
+revived. A transient failure is a failure that can be recovered
+from (e.g. a power failure, intermittent network outage, block
+relocation, etc). The policies for handling both types of failures
+is described herein.
+
+Available Operations During a Device Failure
+--------------------------------------------
+When there is a device failure, LVM behaves somewhat differently because
+only a subset of the available devices will be found for the particular
+volume group. The number of operations available to the administrator
+is diminished. It is not possible to create new logical volumes while
+PVs cannot be accessed, for example. Operations that create, convert, or
+resize logical volumes are disallowed, such as:
+- lvcreate
+- lvresize
+- lvreduce
+- lvextend
+- lvconvert (unless '--repair' is used)
+Operations that activate, deactivate, remove, report, or repair logical
+volumes are allowed, such as:
+- lvremove
+- vgremove (will remove all LVs, but not the VG until consistent)
+- pvs
+- vgs
+- lvs
+- lvchange -a [yn]
+- vgchange -a [yn]
+Operations specific to the handling of failed devices are allowed and
+are as follows:
+
+- 'vgreduce --removemissing <VG>': This action is designed to remove
+ the reference of a failed device from the LVM metadata stored on the
+ remaining devices. If there are (portions of) logical volumes on the
+ failed devices, the ability of the operation to proceed will depend
+ on the type of logical volumes found. If an image (i.e leg or side)
+ of a mirror is located on the device, that image/leg of the mirror
+ is eliminated along with the failed device. The result of such a
+ mirror reduction could be a no-longer-redundant linear device. If
+ a linear, stripe, or snapshot device is located on the failed device
+ the command will not proceed without a '--force' option. The result
+ of using the '--force' option is the entire removal and complete
+ loss of the non-redundant logical volume. Once this operation is
+ complete, the volume group will again have a complete and consistent
+ view of the devices it contains. Thus, all operations will be
+ permitted - including creation, conversion, and resizing operations.
+
+- 'lvconvert --repair <VG/LV>': This action is designed specifically
+ to operate on mirrored logical volumes. It is used on logical volumes
+ individually and does not remove the faulty device from the volume
+ group. If, for example, a failed device happened to contain the
+ images of four distinct mirrors, it would be necessary to run
+ 'lvconvert --repair' on each of them. The ultimate result is to leave
+ the faulty device in the volume group, but have no logical volumes
+ referencing it. In addition to removing mirror images that reside
+ on failed devices, 'lvconvert --repair' can also replace the failed
+ device if there are spare devices available in the volume group. The
+ user is prompted whether to simply remove the failed portions of the
+ mirror or to also allocate a replacement, if run from the command-line.
+ Optionally, the '--use-policies' flag can be specified which will
+ cause the operation not to prompt the user, but instead respect
+ the policies outlined in the LVM configuration file - usually,
+ /etc/lvm/lvm.conf. Once this operation is complete, mirrored logical
+ volumes will be consistent and I/O will be allowed to continue.
+ However, the volume group will still be inconsistent - due to the
+ refernced-but-missing device/PV - and operations will still be
+ restricted to the aformentioned actions until either the device is
+ restored or 'vgreduce --removemissing' is run.
+
+Device Revival (transient failures):
+------------------------------------
+During a device failure, the above section describes what limitations
+a user can expect. However, if the device returns after a period of
+time, what to expect will depend on what has happened during the time
+period when the device was failed. If no automated actions (described
+below) or user actions were necessary or performed, then no change in
+operations or logical volume layout will occur. However, if an
+automated action or one of the aforementioned repair commands was
+manually run, the returning device will be perceived as having stale
+LVM metadata. In this case, the user can expect to see a warning
+concerning inconsistent metadata. The metadata on the returning
+device will be automatically replaced with the latest copy of the
+LVM metadata - restoring consistency. Note, while most LVM commands
+will automatically update the metadata on a restored devices, the
+following possible exceptions exist:
+- pvs (when it does not read/update VG metadata)
+
+Automated Target Response to Failures:
+--------------------------------------
+The only LVM target type (i.e. "personality") that has an automated
+response to failures is a mirrored logical volume. The other target
+types (linear, stripe, snapshot, etc) will simply propagate the failure.
+[A snapshot becomes invalid if its underlying device fails, but the
+origin will remain valid - presuming the origin device has not failed.]
+There are three types of errors that a mirror can suffer - read, write,
+and resynchronization errors. Each is described in depth below.
+
+Mirror read failures:
+If a mirror is 'in-sync' (i.e. all images have been initialized and
+are identical), a read failure will only produce a warning. Data is
+simply pulled from one of the other images and the fault is recorded.
+Sometimes - like in the case of bad block relocation - read errors can
+be recovered from by the storage hardware. Therefore, it is up to the
+user to decide whether to reconfigure the mirror and remove the device
+that caused the error. Managing the composition of a mirror is done with
+'lvconvert' and removing a device from a volume group can be done with
+'vgreduce'.
+
+If a mirror is not 'in-sync', a read failure will produce an I/O error.
+This error will propagate all the way up to the applications above the
+logical volume (e.g. the file system). No automatic intervention will
+take place in this case either. It is up to the user to decide what
+can be done/salvaged in this senario. If the user is confident that the
+images of the mirror are the same (or they are willing to simply attempt
+to retreive whatever data they can), 'lvconvert' can be used to eliminate
+the failed image and proceed.
+
+Mirror resynchronization errors:
+A resynchronization error is one that occurs when trying to initialize
+all mirror images to be the same. It can happen due to a failure to
+read the primary image (the image considered to have the 'good' data), or
+due to a failure to write the secondary images. This type of failure
+only produces a warning, and it is up to the user to take action in this
+case. If the error is transient, the user can simply reactivate the
+mirrored logical volume to make another attempt at resynchronization.
+If attempts to finish resynchronization fail, 'lvconvert' can be used to
+remove the faulty device from the mirror.
+
+TODO...
+Some sort of response to this type of error could be automated.
+Since this document is the definitive source for how to handle device
+failures, the process should be defined here. If the process is defined
+but not implemented, it should be noted as such. One idea might be to
+make a single attempt to suspend/resume the mirror in an attempt to
+redo the sync operation that failed. On the other hand, if there is
+a permanent failure, it may simply be best to wait for the user or the
+automated response that is sure to follow from a write failure.
+...TODO
+
+Mirror write failures:
+When a write error occurs on a mirror constituent device, an attempt
+to handle the failure is automatically made. This is done by calling
+'lvconvert --repair --use-policies'. The policies implied by this
+command are set in the LVM configuration file. They are:
+- mirror_log_fault_policy: This defines what action should be taken
+ if the device containing the log fails. The available options are
+ "remove" and "allocate". Either of these options will cause the
+ faulty log device to be removed from the mirror. The "allocate"
+ policy will attempt the further action of trying to replace the
+ failed disk log by using space that might be available in the
+ volume group. If the allocation fails (or the "remove" policy
+ is specified), the mirror log will be maintained in memory. Should
+ the machine be rebooted or the logical volume deactivated, a
+ complete resynchronization of the mirror will be necessary upon
+ the follow activation - such is the nature of a mirror with a 'core'
+ log. The default policy for handling log failures is "allocate".
+ The service disruption incurred by replacing the failed log is
+ negligible, while the benefits of having persistent log is
+ pronounced.
+- mirror_image_fault_policy: This defines what action should be taken
+ if a device containing an image fails. Again, the available options
+ are "remove" and "allocate". Both of these options will cause the
+ faulty image device to be removed - adjusting the logical volume
+ accordingly. For example, if one image of a 2-way mirror fails, the
+ mirror will be converted to a linear device. If one image of a
+ 3-way mirror fails, the mirror will be converted to a 2-way mirror.
+ The "allocate" policy takes the further action of trying to replace
+ the failed image using space that is available in the volume group.
+ Replacing a failed mirror image will incure the cost of
+ resynchronizing - degrading the performance of the mirror. The
+ default policy for handling an image failure is "remove". This
+ allows the mirror to still function, but gives the administrator the
+ choice of when to incure the extra performance costs of replacing
+ the failed image.
+
+TODO...
+The appropriate time to take permanent corrective action on a mirror
+should be driven by policy. There should be a directive that takes
+a time or percentage argument. Something like the following:
+- mirror_fault_policy_WHEN = "10sec"/"10%"
+A time value would signal the amount of time to wait for transient
+failures to resolve themselves. The percentage value would signal the
+amount a mirror could become out-of-sync before the faulty device is
+removed.
+
+A mirror cannot be used unless /some/ corrective action is taken,
+however. One option is to replace the failed mirror image with an
+error target, forgo the use of 'handle_errors', and simply let the
+out-of-sync regions accumulate and be tracked by the log. Mirrors
+that have more than 2 images would have to "stack" to perform the
+tracking, as each failed image would have to be associated with a
+log. If the failure is transient, the device would replace the
+error target that was holding its spot and the log that was tracking
+the deltas would be used to quickly restore the portions that changed.
+
+One unresolved issue with the above scheme is how to know which
+regions of the mirror are out-of-sync when a problem occurs. When
+a write failure occurs in the kernel, the log will contain those
+regions that are not in-sync. If the log is a disk log, that log
+could continue to be used to track differences. However, if the
+log was a core log - or if the log device failed at the same time
+as an image device - there would be no way to determine which
+regions are out-of-sync to begin with as we start to track the
+deltas for the failed image. I don't have a solution for this
+problem other than to only be able to handle errors in this way
+if conditions are right. These issues will have to be ironed out
+before proceeding. This could be another case, where it is better
+to handle failures in the kernel by allowing the kernel to store
+updates in various metadata areas.
+...TODO
More information about the lvm-devel
mailing list