[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

[Libguestfs] [PATCH 14/16] docs: Move architecture and internals documentation to guestfs-internals(1).



---
 .gitignore                   |   3 +
 docs/Makefile.am             |  15 ++
 docs/guestfs-faq.pod         |   4 +-
 docs/guestfs-hacking.pod     |   1 +
 docs/guestfs-internals.pod   | 415 +++++++++++++++++++++++++++++++++++++++++++
 docs/guestfs-performance.pod |   1 +
 po-docs/language.mk          |   1 +
 po-docs/podfiles             |   1 +
 src/guestfs.pod              | 402 +----------------------------------------
 9 files changed, 444 insertions(+), 399 deletions(-)
 create mode 100644 docs/guestfs-internals.pod

diff --git a/.gitignore b/.gitignore
index 3338d27..a33a286 100644
--- a/.gitignore
+++ b/.gitignore
@@ -132,12 +132,14 @@ Makefile.in
 /diff/virt-diff.1
 /docs/guestfs-faq.1
 /docs/guestfs-hacking.1
+/docs/guestfs-internals.1
 /docs/guestfs-performance.1
 /docs/guestfs-recipes.1
 /docs/guestfs-release-notes.1
 /docs/guestfs-testing.1
 /docs/stamp-guestfs-faq.pod
 /docs/stamp-guestfs-hacking.pod
+/docs/stamp-guestfs-internals.pod
 /docs/stamp-guestfs-performance.pod
 /docs/stamp-guestfs-recipes.pod
 /docs/stamp-guestfs-release-notes.pod
@@ -234,6 +236,7 @@ Makefile.in
 /html/guestfs-examples.3.html
 /html/guestfs-faq.1.html
 /html/guestfs-hacking.1.html
+/html/guestfs-internals.1.html
 /html/guestfs-golang.3.html
 /html/guestfs-java.3.html
 /html/guestfs-lua.3.html
diff --git a/docs/Makefile.am b/docs/Makefile.am
index 4e6a0b5..9ead4c8 100644
--- a/docs/Makefile.am
+++ b/docs/Makefile.am
@@ -20,6 +20,7 @@ include $(top_srcdir)/subdir-rules.mk
 EXTRA_DIST = \
 	guestfs-faq.pod \
 	guestfs-hacking.pod \
+	guestfs-internals.pod \
 	guestfs-performance.pod \
 	guestfs-recipes.pod \
 	guestfs-release-notes.pod \
@@ -29,6 +30,7 @@ EXTRA_DIST = \
 CLEANFILES = \
 	stamp-guestfs-faq.pod \
 	stamp-guestfs-hacking.pod \
+	stamp-guestfs-internals.pod \
 	stamp-guestfs-performance.pod \
 	stamp-guestfs-recipes.pod \
 	stamp-guestfs-release-notes.pod \
@@ -37,6 +39,7 @@ CLEANFILES = \
 man_MANS = \
 	guestfs-faq.1 \
 	guestfs-hacking.1 \
+	guestfs-internals.1 \
 	guestfs-performance.1 \
 	guestfs-recipes.1 \
 	guestfs-release-notes.1 \
@@ -44,6 +47,7 @@ man_MANS = \
 noinst_DATA = \
 	$(top_builddir)/html/guestfs-faq.1.html \
 	$(top_builddir)/html/guestfs-hacking.1.html \
+	$(top_builddir)/html/guestfs-internals.1.html \
 	$(top_builddir)/html/guestfs-performance.1.html \
 	$(top_builddir)/html/guestfs-recipes.1.html \
 	$(top_builddir)/html/guestfs-release-notes.1.html \
@@ -71,6 +75,17 @@ stamp-guestfs-hacking.pod: guestfs-hacking.pod
 	  $<
 	touch $@
 
+guestfs-internals.1 $(top_builddir)/html/guestfs-internals.1.html: stamp-guestfs-internals.pod
+
+stamp-guestfs-internals.pod: guestfs-internals.pod
+	$(PODWRAPPER) \
+	  --section 1 \
+	  --man guestfs-internals.1 \
+	  --html $(top_builddir)/html/guestfs-internals.1.html \
+	  --license LGPLv2+ \
+	  $<
+	touch $@
+
 guestfs-performance.1 $(top_builddir)/html/guestfs-performance.1.html: stamp-guestfs-performance.pod
 
 stamp-guestfs-performance.pod: guestfs-performance.pod
diff --git a/docs/guestfs-faq.pod b/docs/guestfs-faq.pod
index 55dfed2..5215e92 100644
--- a/docs/guestfs-faq.pod
+++ b/docs/guestfs-faq.pod
@@ -990,7 +990,7 @@ F<examples/debug-logging.c> program in the libguestfs sources.
 =head2 Digging deeper into the appliance boot process.
 
 Enable debugging and then read this documentation on the appliance
-boot process: L<guestfs(3)/INTERNALS>.
+boot process: L<guestfs-internals(1)>.
 
 =head2 libguestfs hangs or fails during run/launch.
 
@@ -1015,6 +1015,8 @@ useful debugging information from libvirtd in F</tmp/libvirtd.log>
 
 =head1 DESIGN/INTERNALS OF LIBGUESTFS
 
+See also L<guestfs-internals(1)>.
+
 =head2 Why don't you do everything through the FUSE / filesystem
 interface?
 
diff --git a/docs/guestfs-hacking.pod b/docs/guestfs-hacking.pod
index 7935b56..76b1b8d 100644
--- a/docs/guestfs-hacking.pod
+++ b/docs/guestfs-hacking.pod
@@ -735,6 +735,7 @@ Create the branch in git:
 
 L<guestfs(3)>,
 L<guestfs-examples(3)>,
+L<guestfs-internals(3)>,
 L<guestfs-performance(1)>,
 L<guestfs-release-notes(1)>,
 L<guestfs-testing(1)>,
diff --git a/docs/guestfs-internals.pod b/docs/guestfs-internals.pod
new file mode 100644
index 0000000..a2cc17f
--- /dev/null
+++ b/docs/guestfs-internals.pod
@@ -0,0 +1,415 @@
+=head1 NAME
+
+guestfs-internals - architecture and internals of libguestfs
+
+=head1 DESCRIPTION
+
+This manual page is for hackers who want to understand how libguestfs
+works internally.  This is just a description of how libguestfs works
+now, and it may change at any time in the future.
+
+=head1 ARCHITECTURE
+
+Internally, libguestfs is implemented by running an appliance (a
+special type of small virtual machine) using L<qemu(1)>.  Qemu runs as
+a child process of the main program.
+
+ ┌───────────────────┐
+ │ main program      │
+ │                   │
+ │                   │           child process / appliance
+ │                   │          ┌──────────────────────────┐
+ │                   │          │ qemu                     │
+ ├───────────────────┤   RPC    │      ┌─────────────────┐ │
+ │ libguestfs  ◀╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍▶ guestfsd        │ │
+ │                   │          │      ├─────────────────┤ │
+ └───────────────────┘          │      │ Linux kernel    │ │
+                                │      └────────┬────────┘ │
+                                └───────────────│──────────┘
+                                                │
+                                                │ virtio-scsi
+                                         ┌──────┴──────┐
+                                         │  Device or  │
+                                         │  disk image │
+                                         └─────────────┘
+
+The library, linked to the main program, creates the child process and
+hence the appliance in the L</guestfs_launch> function.
+
+Inside the appliance is a Linux kernel and a complete stack of
+userspace tools (such as LVM and ext2 programs) and a small
+controlling daemon called L</guestfsd>.  The library talks to
+L</guestfsd> using remote procedure calls (RPC).  There is a mostly
+one-to-one correspondence between libguestfs API calls and RPC calls
+to the daemon.  Lastly the disk image(s) are attached to the qemu
+process which translates device access by the appliance's Linux kernel
+into accesses to the image.
+
+A common misunderstanding is that the appliance "is" the virtual
+machine.  Although the disk image you are attached to might also be
+used by some virtual machine, libguestfs doesn't know or care about
+this.  (But you will care if both libguestfs's qemu process and your
+virtual machine are trying to update the disk image at the same time,
+since these usually results in massive disk corruption).
+
+=head1 STATE MACHINE
+
+libguestfs uses a state machine to model the child process:
+
+                         |
+          guestfs_create / guestfs_create_flags
+                         |
+                         |
+                     ____V_____
+                    /          \
+                    |  CONFIG  |
+                    \__________/
+                       ^   ^  \
+                       |    \  \ guestfs_launch
+                       |    _\__V______
+                       |   /           \
+                       |   | LAUNCHING |
+                       |   \___________/
+                       |       /
+                       |  guestfs_launch
+                       |     /
+                     __|____V
+                    /        \
+                    | READY  |
+                    \________/
+
+The normal transitions are (1) CONFIG (when the handle is created, but
+there is no child process), (2) LAUNCHING (when the child process is
+booting up), (3) READY meaning the appliance is up, actions can be
+issued to, and carried out by, the child process.
+
+The guest may be killed by L</guestfs_kill_subprocess>, or may die
+asynchronously at any time (eg. due to some internal error), and that
+causes the state to transition back to CONFIG.
+
+Configuration commands for qemu such as L</guestfs_set_path> can only
+be issued when in the CONFIG state.
+
+The API offers one call that goes from CONFIG through LAUNCHING to
+READY.  L</guestfs_launch> blocks until the child process is READY to
+accept commands (or until some failure or timeout).
+L</guestfs_launch> internally moves the state from CONFIG to LAUNCHING
+while it is running.
+
+API actions such as L</guestfs_mount> can only be issued when in the
+READY state.  These API calls block waiting for the command to be
+carried out.  There are no non-blocking versions, and no way to issue
+more than one command per handle at the same time.
+
+Finally, the child process sends asynchronous messages back to the
+main program, such as kernel log messages.  You can register a
+callback to receive these messages.
+
+=head1 INTERNALS
+
+=head2 APPLIANCE BOOT PROCESS
+
+This process has evolved and continues to evolve.  The description
+here corresponds only to the current version of libguestfs and is
+provided for information only.
+
+In order to follow the stages involved below, enable libguestfs
+debugging (set the environment variable C<LIBGUESTFS_DEBUG=1>).
+
+=over 4
+
+=item Create the appliance
+
+C<supermin --build> is invoked to create the kernel, a small initrd
+and the appliance.
+
+The appliance is cached in F</var/tmp/.guestfs-E<lt>UIDE<gt>> (or in
+another directory if C<LIBGUESTFS_CACHEDIR> or C<TMPDIR> are set).
+
+For a complete description of how the appliance is created and cached,
+read the L<supermin(1)> man page.
+
+=item Start qemu and boot the kernel
+
+qemu is invoked to boot the kernel.
+
+=item Run the initrd
+
+C<supermin --build> builds a small initrd.  The initrd is not the
+appliance.  The purpose of the initrd is to load enough kernel modules
+in order that the appliance itself can be mounted and started.
+
+The initrd is a cpio archive called
+F</var/tmp/.guestfs-E<lt>UIDE<gt>/appliance.d/initrd>.
+
+When the initrd has started you will see messages showing that kernel
+modules are being loaded, similar to this:
+
+ supermin: ext2 mini initrd starting up
+ supermin: mounting /sys
+ supermin: internal insmod libcrc32c.ko
+ supermin: internal insmod crc32c-intel.ko
+
+=item Find and mount the appliance device
+
+The appliance is a sparse file containing an ext2 filesystem which
+contains a familiar (although reduced in size) Linux operating system.
+It would normally be called
+F</var/tmp/.guestfs-E<lt>UIDE<gt>/appliance.d/root>.
+
+The regular disks being inspected by libguestfs are the first
+devices exposed by qemu (eg. as F</dev/vda>).
+
+The last disk added to qemu is the appliance itself (eg. F</dev/vdb>
+if there was only one regular disk).
+
+Thus the final job of the initrd is to locate the appliance disk,
+mount it, and switch root into the appliance, and run F</init> from
+the appliance.
+
+If this works successfully you will see messages such as:
+
+ supermin: picked /sys/block/vdb/dev as root device
+ supermin: creating /dev/root as block special 252:16
+ supermin: mounting new root on /root
+ supermin: chroot
+ Starting /init script ...
+
+Note that C<Starting /init script ...> indicates that the appliance's
+init script is now running.
+
+=item Initialize the appliance
+
+The appliance itself now initializes itself.  This involves starting
+certain processes like C<udev>, possibly printing some debug
+information, and finally running the daemon (C<guestfsd>).
+
+=item The daemon
+
+Finally the daemon (C<guestfsd>) runs inside the appliance.  If it
+runs you should see:
+
+ verbose daemon enabled
+
+The daemon expects to see a named virtio-serial port exposed by qemu
+and connected on the other end to the library.
+
+The daemon connects to this port (and hence to the library) and sends
+a four byte message C<GUESTFS_LAUNCH_FLAG>, which initiates the
+communication protocol (see below).
+
+=back
+
+=head2 COMMUNICATION PROTOCOL
+
+Don't rely on using this protocol directly.  This section documents
+how it currently works, but it may change at any time.
+
+The protocol used to talk between the library and the daemon running
+inside the qemu virtual machine is a simple RPC mechanism built on top
+of XDR (RFC 1014, RFC 1832, RFC 4506).
+
+The detailed format of structures is in F<src/guestfs_protocol.x>
+(note: this file is automatically generated).
+
+There are two broad cases, ordinary functions that don't have any
+C<FileIn> and C<FileOut> parameters, which are handled with very
+simple request/reply messages.  Then there are functions that have any
+C<FileIn> or C<FileOut> parameters, which use the same request and
+reply messages, but they may also be followed by files sent using a
+chunked encoding.
+
+=head3 ORDINARY FUNCTIONS (NO FILEIN/FILEOUT PARAMS)
+
+For ordinary functions, the request message is:
+
+ total length (header + arguments,
+      but not including the length word itself)
+ struct guestfs_message_header (encoded as XDR)
+ struct guestfs_<foo>_args (encoded as XDR)
+
+The total length field allows the daemon to allocate a fixed size
+buffer into which it slurps the rest of the message.  As a result, the
+total length is limited to C<GUESTFS_MESSAGE_MAX> bytes (currently
+4MB), which means the effective size of any request is limited to
+somewhere under this size.
+
+Note also that many functions don't take any arguments, in which case
+the C<guestfs_I<foo>_args> is completely omitted.
+
+The header contains the procedure number (C<guestfs_proc>) which is
+how the receiver knows what type of args structure to expect, or none
+at all.
+
+For functions that take optional arguments, the optional arguments are
+encoded in the C<guestfs_I<foo>_args> structure in the same way as
+ordinary arguments.  A bitmask in the header indicates which optional
+arguments are meaningful.  The bitmask is also checked to see if it
+contains bits set which the daemon does not know about (eg. if more
+optional arguments were added in a later version of the library), and
+this causes the call to be rejected.
+
+The reply message for ordinary functions is:
+
+ total length (header + ret,
+      but not including the length word itself)
+ struct guestfs_message_header (encoded as XDR)
+ struct guestfs_<foo>_ret (encoded as XDR)
+
+As above the C<guestfs_I<foo>_ret> structure may be completely omitted
+for functions that return no formal return values.
+
+As above the total length of the reply is limited to
+C<GUESTFS_MESSAGE_MAX>.
+
+In the case of an error, a flag is set in the header, and the reply
+message is slightly changed:
+
+ total length (header + error,
+      but not including the length word itself)
+ struct guestfs_message_header (encoded as XDR)
+ struct guestfs_message_error (encoded as XDR)
+
+The C<guestfs_message_error> structure contains the error message as a
+string.
+
+=head3 FUNCTIONS THAT HAVE FILEIN PARAMETERS
+
+A C<FileIn> parameter indicates that we transfer a file I<into> the
+guest.  The normal request message is sent (see above).  However this
+is followed by a sequence of file chunks.
+
+ total length (header + arguments,
+      but not including the length word itself,
+      and not including the chunks)
+ struct guestfs_message_header (encoded as XDR)
+ struct guestfs_<foo>_args (encoded as XDR)
+ sequence of chunks for FileIn param #0
+ sequence of chunks for FileIn param #1 etc.
+
+The "sequence of chunks" is:
+
+ length of chunk (not including length word itself)
+ struct guestfs_chunk (encoded as XDR)
+ length of chunk
+ struct guestfs_chunk (encoded as XDR)
+   ...
+ length of chunk
+ struct guestfs_chunk (with data.data_len == 0)
+
+The final chunk has the C<data_len> field set to zero.  Additionally a
+flag is set in the final chunk to indicate either successful
+completion or early cancellation.
+
+At time of writing there are no functions that have more than one
+FileIn parameter.  However this is (theoretically) supported, by
+sending the sequence of chunks for each FileIn parameter one after
+another (from left to right).
+
+Both the library (sender) I<and> the daemon (receiver) may cancel the
+transfer.  The library does this by sending a chunk with a special
+flag set to indicate cancellation.  When the daemon sees this, it
+cancels the whole RPC, does I<not> send any reply, and goes back to
+reading the next request.
+
+The daemon may also cancel.  It does this by writing a special word
+C<GUESTFS_CANCEL_FLAG> to the socket.  The library listens for this
+during the transfer, and if it gets it, it will cancel the transfer
+(it sends a cancel chunk).  The special word is chosen so that even if
+cancellation happens right at the end of the transfer (after the
+library has finished writing and has started listening for the reply),
+the "spurious" cancel flag will not be confused with the reply
+message.
+
+This protocol allows the transfer of arbitrary sized files (no 32 bit
+limit), and also files where the size is not known in advance
+(eg. from pipes or sockets).  However the chunks are rather small
+(C<GUESTFS_MAX_CHUNK_SIZE>), so that neither the library nor the
+daemon need to keep much in memory.
+
+=head3 FUNCTIONS THAT HAVE FILEOUT PARAMETERS
+
+The protocol for FileOut parameters is exactly the same as for FileIn
+parameters, but with the roles of daemon and library reversed.
+
+ total length (header + ret,
+      but not including the length word itself,
+      and not including the chunks)
+ struct guestfs_message_header (encoded as XDR)
+ struct guestfs_<foo>_ret (encoded as XDR)
+ sequence of chunks for FileOut param #0
+ sequence of chunks for FileOut param #1 etc.
+
+=head3 INITIAL MESSAGE
+
+When the daemon launches it sends an initial word
+(C<GUESTFS_LAUNCH_FLAG>) which indicates that the guest and daemon is
+alive.  This is what L</guestfs_launch> waits for.
+
+=head3 PROGRESS NOTIFICATION MESSAGES
+
+The daemon may send progress notification messages at any time.  These
+are distinguished by the normal length word being replaced by
+C<GUESTFS_PROGRESS_FLAG>, followed by a fixed size progress message.
+
+The library turns them into progress callbacks (see
+L</GUESTFS_EVENT_PROGRESS>) if there is a callback registered, or
+discards them if not.
+
+The daemon self-limits the frequency of progress messages it sends
+(see C<daemon/proto.c:notify_progress>).  Not all calls generate
+progress messages.
+
+=head2 FIXED APPLIANCE
+
+When libguestfs (or libguestfs tools) are run, they search a path
+looking for an appliance.  The path is built into libguestfs, or can
+be set using the C<LIBGUESTFS_PATH> environment variable.
+
+Normally a supermin appliance is located on this path (see
+L<supermin(1)/SUPERMIN APPLIANCE>).  libguestfs reconstructs this
+into a full appliance by running C<supermin --build>.
+
+However, a simpler "fixed appliance" can also be used.  libguestfs
+detects this by looking for a directory on the path containing all
+the following files:
+
+=over 4
+
+=item * F<kernel>
+
+=item * F<initrd>
+
+=item * F<root>
+
+=item * F<README.fixed> (note that it B<must> be present as well)
+
+=back
+
+If the fixed appliance is found, libguestfs skips supermin entirely
+and just runs the virtual machine (using qemu or the current backend,
+see L</BACKEND>) with the kernel, initrd and root disk from the fixed
+appliance.
+
+Thus the fixed appliance can be used when a platform or a Linux
+distribution does not support supermin.  You build the fixed appliance
+on a platform that does support supermin using
+L<libguestfs-make-fixed-appliance(1)>, copy it over, and use that
+to run libguestfs.
+
+=head1 SEE ALSO
+
+L<guestfs(3)>,
+L<guestfs-hacking(3)>,
+L<guestfs-examples(3)>,
+L<libguestfs-test-tool(1)>,
+L<libguestfs-make-fixed-appliance(1)>,
+L<http://libguestfs.org/>.
+
+=head1 AUTHORS
+
+Richard W.M. Jones (C<rjones at redhat dot com>)
+
+=head1 COPYRIGHT
+
+Copyright (C) 2009-2015 Red Hat Inc.
diff --git a/docs/guestfs-performance.pod b/docs/guestfs-performance.pod
index 27f24b4..5a7c01b 100644
--- a/docs/guestfs-performance.pod
+++ b/docs/guestfs-performance.pod
@@ -570,6 +570,7 @@ L<supermin(1)>,
 L<guestfish(1)>,
 L<guestfs(3)>,
 L<guestfs-examples(3)>,
+L<guestfs-internals(1)>,
 L<libguestfs-make-fixed-appliance(1)>,
 L<stap(1)>,
 L<qemu(1)>,
diff --git a/po-docs/language.mk b/po-docs/language.mk
index 8b70c55..73fcacd 100644
--- a/po-docs/language.mk
+++ b/po-docs/language.mk
@@ -33,6 +33,7 @@ MANPAGES = \
 	guestfs-examples.3 \
 	guestfs-faq.1 \
 	guestfs-hacking.1 \
+	guestfs-internals.1 \
 	guestfs-golang.3 \
 	guestfs-java.3 \
 	guestfs-lua.3 \
diff --git a/po-docs/podfiles b/po-docs/podfiles
index 2f932fe..611b549 100644
--- a/po-docs/podfiles
+++ b/po-docs/podfiles
@@ -15,6 +15,7 @@
 ../diff/virt-diff.pod
 ../docs/guestfs-faq.pod
 ../docs/guestfs-hacking.pod
+../docs/guestfs-internals.pod
 ../docs/guestfs-performance.pod
 ../docs/guestfs-recipes.pod
 ../docs/guestfs-release-notes.pod
diff --git a/src/guestfs.pod b/src/guestfs.pod
index b904ce4..399396e 100644
--- a/src/guestfs.pod
+++ b/src/guestfs.pod
@@ -41,7 +41,8 @@ For tips and recipes, see L<guestfs-recipes(1)>.
 If you are having performance problems, read
 L<guestfs-performance(1)>.  To help test libguestfs, read
 L<libguestfs-test-tool(1)> and L<guestfs-testing(1)>.  To contribute
-code to libguestfs, see L<guestfs-hacking(1)>.
+code to libguestfs, see L<guestfs-hacking(1)>.  To find out how
+libguestfs works, see L<guestfs-internals(1)>.
 
 =head1 API OVERVIEW
 
@@ -3482,402 +3483,6 @@ In the first terminal, stap trace output similar to this is shown:
  1318248061071167 (+4233108):   launch_end
  1318248061280324 (+209157):    guestfs_mkfs g=0x1024ab0 fstype=0x46116f device=0x1024e60
 
-=begin html
-
-<!-- old anchor for the next section -->
-<a name="state_machine_and_low_level_event_api"/>
-
-=end html
-
-=head1 ARCHITECTURE
-
-Internally, libguestfs is implemented by running an appliance (a
-special type of small virtual machine) using L<qemu(1)>.  Qemu runs as
-a child process of the main program.
-
- ┌───────────────────┐
- │ main program      │
- │                   │
- │                   │           child process / appliance
- │                   │          ┌──────────────────────────┐
- │                   │          │ qemu                     │
- ├───────────────────┤   RPC    │      ┌─────────────────┐ │
- │ libguestfs  ◀╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍▶ guestfsd        │ │
- │                   │          │      ├─────────────────┤ │
- └───────────────────┘          │      │ Linux kernel    │ │
-                                │      └────────┬────────┘ │
-                                └───────────────│──────────┘
-                                                │
-                                                │ virtio-scsi
-                                         ┌──────┴──────┐
-                                         │  Device or  │
-                                         │  disk image │
-                                         └─────────────┘
-
-The library, linked to the main program, creates the child process and
-hence the appliance in the L</guestfs_launch> function.
-
-Inside the appliance is a Linux kernel and a complete stack of
-userspace tools (such as LVM and ext2 programs) and a small
-controlling daemon called L</guestfsd>.  The library talks to
-L</guestfsd> using remote procedure calls (RPC).  There is a mostly
-one-to-one correspondence between libguestfs API calls and RPC calls
-to the daemon.  Lastly the disk image(s) are attached to the qemu
-process which translates device access by the appliance's Linux kernel
-into accesses to the image.
-
-A common misunderstanding is that the appliance "is" the virtual
-machine.  Although the disk image you are attached to might also be
-used by some virtual machine, libguestfs doesn't know or care about
-this.  (But you will care if both libguestfs's qemu process and your
-virtual machine are trying to update the disk image at the same time,
-since these usually results in massive disk corruption).
-
-=head1 STATE MACHINE
-
-libguestfs uses a state machine to model the child process:
-
-                         |
-          guestfs_create / guestfs_create_flags
-                         |
-                         |
-                     ____V_____
-                    /          \
-                    |  CONFIG  |
-                    \__________/
-                       ^   ^  \
-                       |    \  \ guestfs_launch
-                       |    _\__V______
-                       |   /           \
-                       |   | LAUNCHING |
-                       |   \___________/
-                       |       /
-                       |  guestfs_launch
-                       |     /
-                     __|____V
-                    /        \
-                    | READY  |
-                    \________/
-
-The normal transitions are (1) CONFIG (when the handle is created, but
-there is no child process), (2) LAUNCHING (when the child process is
-booting up), (3) READY meaning the appliance is up, actions can be
-issued to, and carried out by, the child process.
-
-The guest may be killed by L</guestfs_kill_subprocess>, or may die
-asynchronously at any time (eg. due to some internal error), and that
-causes the state to transition back to CONFIG.
-
-Configuration commands for qemu such as L</guestfs_set_path> can only
-be issued when in the CONFIG state.
-
-The API offers one call that goes from CONFIG through LAUNCHING to
-READY.  L</guestfs_launch> blocks until the child process is READY to
-accept commands (or until some failure or timeout).
-L</guestfs_launch> internally moves the state from CONFIG to LAUNCHING
-while it is running.
-
-API actions such as L</guestfs_mount> can only be issued when in the
-READY state.  These API calls block waiting for the command to be
-carried out.  There are no non-blocking versions, and no way to issue
-more than one command per handle at the same time.
-
-Finally, the child process sends asynchronous messages back to the
-main program, such as kernel log messages.  You can register a
-callback to receive these messages.
-
-=head1 INTERNALS
-
-=head2 APPLIANCE BOOT PROCESS
-
-This process has evolved and continues to evolve.  The description
-here corresponds only to the current version of libguestfs and is
-provided for information only.
-
-In order to follow the stages involved below, enable libguestfs
-debugging (set the environment variable C<LIBGUESTFS_DEBUG=1>).
-
-=over 4
-
-=item Create the appliance
-
-C<supermin --build> is invoked to create the kernel, a small initrd
-and the appliance.
-
-The appliance is cached in F</var/tmp/.guestfs-E<lt>UIDE<gt>> (or in
-another directory if C<LIBGUESTFS_CACHEDIR> or C<TMPDIR> are set).
-
-For a complete description of how the appliance is created and cached,
-read the L<supermin(1)> man page.
-
-=item Start qemu and boot the kernel
-
-qemu is invoked to boot the kernel.
-
-=item Run the initrd
-
-C<supermin --build> builds a small initrd.  The initrd is not the
-appliance.  The purpose of the initrd is to load enough kernel modules
-in order that the appliance itself can be mounted and started.
-
-The initrd is a cpio archive called
-F</var/tmp/.guestfs-E<lt>UIDE<gt>/appliance.d/initrd>.
-
-When the initrd has started you will see messages showing that kernel
-modules are being loaded, similar to this:
-
- supermin: ext2 mini initrd starting up
- supermin: mounting /sys
- supermin: internal insmod libcrc32c.ko
- supermin: internal insmod crc32c-intel.ko
-
-=item Find and mount the appliance device
-
-The appliance is a sparse file containing an ext2 filesystem which
-contains a familiar (although reduced in size) Linux operating system.
-It would normally be called
-F</var/tmp/.guestfs-E<lt>UIDE<gt>/appliance.d/root>.
-
-The regular disks being inspected by libguestfs are the first
-devices exposed by qemu (eg. as F</dev/vda>).
-
-The last disk added to qemu is the appliance itself (eg. F</dev/vdb>
-if there was only one regular disk).
-
-Thus the final job of the initrd is to locate the appliance disk,
-mount it, and switch root into the appliance, and run F</init> from
-the appliance.
-
-If this works successfully you will see messages such as:
-
- supermin: picked /sys/block/vdb/dev as root device
- supermin: creating /dev/root as block special 252:16
- supermin: mounting new root on /root
- supermin: chroot
- Starting /init script ...
-
-Note that C<Starting /init script ...> indicates that the appliance's
-init script is now running.
-
-=item Initialize the appliance
-
-The appliance itself now initializes itself.  This involves starting
-certain processes like C<udev>, possibly printing some debug
-information, and finally running the daemon (C<guestfsd>).
-
-=item The daemon
-
-Finally the daemon (C<guestfsd>) runs inside the appliance.  If it
-runs you should see:
-
- verbose daemon enabled
-
-The daemon expects to see a named virtio-serial port exposed by qemu
-and connected on the other end to the library.
-
-The daemon connects to this port (and hence to the library) and sends
-a four byte message C<GUESTFS_LAUNCH_FLAG>, which initiates the
-communication protocol (see below).
-
-=back
-
-=head2 COMMUNICATION PROTOCOL
-
-Don't rely on using this protocol directly.  This section documents
-how it currently works, but it may change at any time.
-
-The protocol used to talk between the library and the daemon running
-inside the qemu virtual machine is a simple RPC mechanism built on top
-of XDR (RFC 1014, RFC 1832, RFC 4506).
-
-The detailed format of structures is in F<src/guestfs_protocol.x>
-(note: this file is automatically generated).
-
-There are two broad cases, ordinary functions that don't have any
-C<FileIn> and C<FileOut> parameters, which are handled with very
-simple request/reply messages.  Then there are functions that have any
-C<FileIn> or C<FileOut> parameters, which use the same request and
-reply messages, but they may also be followed by files sent using a
-chunked encoding.
-
-=head3 ORDINARY FUNCTIONS (NO FILEIN/FILEOUT PARAMS)
-
-For ordinary functions, the request message is:
-
- total length (header + arguments,
-      but not including the length word itself)
- struct guestfs_message_header (encoded as XDR)
- struct guestfs_<foo>_args (encoded as XDR)
-
-The total length field allows the daemon to allocate a fixed size
-buffer into which it slurps the rest of the message.  As a result, the
-total length is limited to C<GUESTFS_MESSAGE_MAX> bytes (currently
-4MB), which means the effective size of any request is limited to
-somewhere under this size.
-
-Note also that many functions don't take any arguments, in which case
-the C<guestfs_I<foo>_args> is completely omitted.
-
-The header contains the procedure number (C<guestfs_proc>) which is
-how the receiver knows what type of args structure to expect, or none
-at all.
-
-For functions that take optional arguments, the optional arguments are
-encoded in the C<guestfs_I<foo>_args> structure in the same way as
-ordinary arguments.  A bitmask in the header indicates which optional
-arguments are meaningful.  The bitmask is also checked to see if it
-contains bits set which the daemon does not know about (eg. if more
-optional arguments were added in a later version of the library), and
-this causes the call to be rejected.
-
-The reply message for ordinary functions is:
-
- total length (header + ret,
-      but not including the length word itself)
- struct guestfs_message_header (encoded as XDR)
- struct guestfs_<foo>_ret (encoded as XDR)
-
-As above the C<guestfs_I<foo>_ret> structure may be completely omitted
-for functions that return no formal return values.
-
-As above the total length of the reply is limited to
-C<GUESTFS_MESSAGE_MAX>.
-
-In the case of an error, a flag is set in the header, and the reply
-message is slightly changed:
-
- total length (header + error,
-      but not including the length word itself)
- struct guestfs_message_header (encoded as XDR)
- struct guestfs_message_error (encoded as XDR)
-
-The C<guestfs_message_error> structure contains the error message as a
-string.
-
-=head3 FUNCTIONS THAT HAVE FILEIN PARAMETERS
-
-A C<FileIn> parameter indicates that we transfer a file I<into> the
-guest.  The normal request message is sent (see above).  However this
-is followed by a sequence of file chunks.
-
- total length (header + arguments,
-      but not including the length word itself,
-      and not including the chunks)
- struct guestfs_message_header (encoded as XDR)
- struct guestfs_<foo>_args (encoded as XDR)
- sequence of chunks for FileIn param #0
- sequence of chunks for FileIn param #1 etc.
-
-The "sequence of chunks" is:
-
- length of chunk (not including length word itself)
- struct guestfs_chunk (encoded as XDR)
- length of chunk
- struct guestfs_chunk (encoded as XDR)
-   ...
- length of chunk
- struct guestfs_chunk (with data.data_len == 0)
-
-The final chunk has the C<data_len> field set to zero.  Additionally a
-flag is set in the final chunk to indicate either successful
-completion or early cancellation.
-
-At time of writing there are no functions that have more than one
-FileIn parameter.  However this is (theoretically) supported, by
-sending the sequence of chunks for each FileIn parameter one after
-another (from left to right).
-
-Both the library (sender) I<and> the daemon (receiver) may cancel the
-transfer.  The library does this by sending a chunk with a special
-flag set to indicate cancellation.  When the daemon sees this, it
-cancels the whole RPC, does I<not> send any reply, and goes back to
-reading the next request.
-
-The daemon may also cancel.  It does this by writing a special word
-C<GUESTFS_CANCEL_FLAG> to the socket.  The library listens for this
-during the transfer, and if it gets it, it will cancel the transfer
-(it sends a cancel chunk).  The special word is chosen so that even if
-cancellation happens right at the end of the transfer (after the
-library has finished writing and has started listening for the reply),
-the "spurious" cancel flag will not be confused with the reply
-message.
-
-This protocol allows the transfer of arbitrary sized files (no 32 bit
-limit), and also files where the size is not known in advance
-(eg. from pipes or sockets).  However the chunks are rather small
-(C<GUESTFS_MAX_CHUNK_SIZE>), so that neither the library nor the
-daemon need to keep much in memory.
-
-=head3 FUNCTIONS THAT HAVE FILEOUT PARAMETERS
-
-The protocol for FileOut parameters is exactly the same as for FileIn
-parameters, but with the roles of daemon and library reversed.
-
- total length (header + ret,
-      but not including the length word itself,
-      and not including the chunks)
- struct guestfs_message_header (encoded as XDR)
- struct guestfs_<foo>_ret (encoded as XDR)
- sequence of chunks for FileOut param #0
- sequence of chunks for FileOut param #1 etc.
-
-=head3 INITIAL MESSAGE
-
-When the daemon launches it sends an initial word
-(C<GUESTFS_LAUNCH_FLAG>) which indicates that the guest and daemon is
-alive.  This is what L</guestfs_launch> waits for.
-
-=head3 PROGRESS NOTIFICATION MESSAGES
-
-The daemon may send progress notification messages at any time.  These
-are distinguished by the normal length word being replaced by
-C<GUESTFS_PROGRESS_FLAG>, followed by a fixed size progress message.
-
-The library turns them into progress callbacks (see
-L</GUESTFS_EVENT_PROGRESS>) if there is a callback registered, or
-discards them if not.
-
-The daemon self-limits the frequency of progress messages it sends
-(see C<daemon/proto.c:notify_progress>).  Not all calls generate
-progress messages.
-
-=head2 FIXED APPLIANCE
-
-When libguestfs (or libguestfs tools) are run, they search a path
-looking for an appliance.  The path is built into libguestfs, or can
-be set using the C<LIBGUESTFS_PATH> environment variable.
-
-Normally a supermin appliance is located on this path (see
-L<supermin(1)/SUPERMIN APPLIANCE>).  libguestfs reconstructs this
-into a full appliance by running C<supermin --build>.
-
-However, a simpler "fixed appliance" can also be used.  libguestfs
-detects this by looking for a directory on the path containing all
-the following files:
-
-=over 4
-
-=item * F<kernel>
-
-=item * F<initrd>
-
-=item * F<root>
-
-=item * F<README.fixed> (note that it B<must> be present as well)
-
-=back
-
-If the fixed appliance is found, libguestfs skips supermin entirely
-and just runs the virtual machine (using qemu or the current backend,
-see L</BACKEND>) with the kernel, initrd and root disk from the fixed
-appliance.
-
-Thus the fixed appliance can be used when a platform or a Linux
-distribution does not support supermin.  You build the fixed appliance
-on a platform that does support supermin using
-L<libguestfs-make-fixed-appliance(1)>, copy it over, and use that
-to run libguestfs.
-
 =head1 LIBGUESTFS VERSION NUMBERS
 
 Since April 2010, libguestfs has started to make separate development
@@ -3946,7 +3551,7 @@ time.
 =head2 PROTOCOL LIMITS
 
 Internally libguestfs uses a message-based protocol to pass API calls
-and their responses to and from a small "appliance" (see L</INTERNALS>
+and their responses to and from a small "appliance" (see L<guestfs-internals(1)>
 for plenty more detail about this).  The maximum message size used by
 the protocol is slightly less than 4 MB.  For some API calls you may
 need to be aware of this limit.  The API calls which may be affected
@@ -4207,6 +3812,7 @@ L<virt-v2v(1)>,
 L<virt-win-reg(1)>,
 L<guestfs-faq(1)>,
 L<guestfs-hacking(1)>,
+L<guestfs-internals(1)>,
 L<guestfs-performance(1)>,
 L<guestfs-release-notes(1)>,
 L<guestfs-testing(1)>,
-- 
2.5.0


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]