<div dir="ltr"><div><div>Gentle reminder.  Humble request for another round of review. <br><br></div>thanks<br></div>imran<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jul 1, 2015 at 11:30 PM, Imran Khan <span dir="ltr"><<a href="mailto:ik.nitk@gmail.com" target="_blank">ik.nitk@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div><div><div><div>Dear Michal,<br><br></div>Thanks a lot for reviewing the changes.  I really appreciate your ability to spot very minute errors which have not been spotted by me or my local reviews. <br><br></div>I have reworked almost all of the review comments you made. and Have sent a separate email with subject "Inherit namespace feature".  But for convenience i would like to paste the changes here too. Please forgive for big mail.   To answer some of your questions below:<br></div>   1.  Why this feature :<br></div><div>         (imran):  As the latest container technology (lxc-tools  and docker) already provides sharing of network namespace. We think that this feature is important to be added in libvirt lxc too.   <br></div><div>          please check this link for lxc-start option :[ <b>--share-[net|ipc|uts] </b><i>name|pid</i> ]   <br>           <a href="http://man7.org/linux/man-pages/man1/lxc-start.1.html" target="_blank">http://man7.org/linux/man-pages/man1/lxc-start.1.html</a><br></div><div>          please check this link for docker --net option : <br>              <a href="https://docs.docker.com/articles/networking/" target="_blank">https://docs.docker.com/articles/networking/</a>: <code>--net=container:NAME_or_ID</code></div>   2.  Moving open namespace in driver. <br></div><div>         (imran) I have moved the code to driver now.  But regarding the function i used the readymade functions instead of using internal data structure because i would end up writing almost same functions again which i felt was redundant. <br></div><div>   3. Regarding the security:<br></div><div>         (imran) This can always be taken care by adding checks in pre-install or post-install scripts for libvirt lxc:<br>          <a href="https://libvirt.org/drvlxc.html#security" target="_blank">https://libvirt.org/drvlxc.html#security</a><br><br></div><div><br></div>code snippet <br><br>
<br>
---<br>
 docs/<a href="http://drvlxc.html.in" rel="noreferrer" target="_blank">drvlxc.html.in</a>                   |  18 +++<br>
 docs/schemas/domaincommon.rng         |  42 ++++++<br>
 src/Makefile.am                       |   4 +-<br>
 src/lxc/lxc_conf.c                    |   2 +-<br>
 src/lxc/lxc_conf.h                    |  15 +++<br>
 src/lxc/lxc_container.c               | 236 +++++++++++++++++++++++++++++++++-<br>
 src/lxc/lxc_domain.c                  | 164 ++++++++++++++++++++++-<br>
 src/lxc/lxc_domain.h                  |   1 +<br>
 tests/lxcxml2xmldata/lxc-sharenet.xml |  33 +++++<br>
 tests/lxcxml2xmltest.c                |   1 +<br>
 10 files changed, 507 insertions(+), 9 deletions(-)<br>
 create mode 100644 tests/lxcxml2xmldata/lxc-sharenet.xml<br>
<br>
diff --git a/docs/<a href="http://drvlxc.html.in" rel="noreferrer" target="_blank">drvlxc.html.in</a> b/docs/<a href="http://drvlxc.html.in" rel="noreferrer" target="_blank">drvlxc.html.in</a><br>
index a094bd9..d14d4c7 100644<br>
--- a/docs/<a href="http://drvlxc.html.in" rel="noreferrer" target="_blank">drvlxc.html.in</a><br>
+++ b/docs/<a href="http://drvlxc.html.in" rel="noreferrer" target="_blank">drvlxc.html.in</a><br>
@@ -590,6 +590,24 @@ Note that allowing capabilities that are normally dropped by default can serious<br>
 affect the security of the container and the host.<br>
 </p><br>
<br>
+<h2><a name="share">Inherit namespaces</a></h2><br>
+<br>
+<p><br>
+Libvirt allows you to inherit the namespace from container/process just like lxc tools<br>
+or docker provides to share the network namespace. The following can be used to share<br>
+required namespaces. If we want to share only one then the other namespaces can be ignored.<br>
+</p><br>
+<pre><br>
+&lt;domain type='lxc' xmlns:lxc='<a href="http://libvirt.org/schemas/domain/lxc/1.0%27&gt" rel="noreferrer" target="_blank">http://libvirt.org/schemas/domain/lxc/1.0'&gt</a>;<br>
+...<br>
+&lt;lxc:namespace&gt;<br>
+  &lt;lxc:sharenet type='netns' value='red'/&gt;<br>
+  &lt;lxc:shareuts type='name' value='container1'/&gt;<br>
+  &lt;lxc:shareipc type='pid' value='12345'/&gt;<br>
+&lt;/lxc:namespace&gt;<br>
+&lt;/domain&gt;<br>
+</pre><br>
+<br>
 <h2><a name="usage">Container usage / management</a></h2><br>
<br>
 <p><br>
diff --git a/docs/schemas/domaincommon.rng b/docs/schemas/domaincommon.rng<br>
index 1120003..803b327 100644<br>
--- a/docs/schemas/domaincommon.rng<br>
+++ b/docs/schemas/domaincommon.rng<br>
@@ -68,6 +68,9 @@<br>
           <ref name='qemucmdline'/><br>
         </optional><br>
         <optional><br>
+          <ref name='lxcsharens'/><br>
+        </optional><br>
+        <optional><br>
           <ref name='keywrap'/><br>
         </optional><br>
       </interleave><br>
@@ -5012,6 +5015,45 @@<br>
     </element><br>
   </define><br>
<br>
+  <!--<br>
+       Optional hypervisor extensions in their own namespace:<br>
+       LXC<br>
+    --><br>
+  <define name="lxcsharens"><br>
+    <element name="namespace" ns="<a href="http://libvirt.org/schemas/domain/lxc/1.0" rel="noreferrer" target="_blank">http://libvirt.org/schemas/domain/lxc/1.0</a>"><br>
+      <zeroOrMore><br>
+        <element name="sharenet"><br>
+          <attribute name="type"><br>
+            <choice><br>
+              <value>netns</value><br>
+              <value>name</value><br>
+              <value>pid</value><br>
+            </choice><br>
+          </attribute><br>
+          <attribute name='value'/><br>
+        </element><br>
+        <element name="shareipc"><br>
+          <attribute name="type"><br>
+            <choice><br>
+              <value>name</value><br>
+              <value>pid</value><br>
+            </choice><br>
+          </attribute><br>
+          <attribute name='value'/><br>
+        </element><br>
+        <element name="shareuts"><br>
+          <attribute name="type"><br>
+            <choice><br>
+              <value>name</value><br>
+              <value>pid</value><br>
+            </choice><br>
+          </attribute><br>
+          <attribute name='value'/><br>
+        </element><br>
+      </zeroOrMore><br>
+    </element><br>
+  </define><br>
+<br>
   <define name="metadata"><br>
     <element name="metadata"><br>
       <zeroOrMore><br>
diff --git a/src/Makefile.am b/src/Makefile.am<br>
index be63e26..ef96a5a 100644<br>
--- a/src/Makefile.am<br>
+++ b/src/Makefile.am<br>
@@ -1319,7 +1319,7 @@ libvirt_driver_lxc_impl_la_CFLAGS = \<span class=""><br>
                -I$(srcdir)/access \<br>
                -I$(srcdir)/conf \<br>
                $(AM_CFLAGS)<br>
-libvirt_driver_lxc_impl_la_LIBADD = $(CAPNG_LIBS) $(LIBNL_LIBS) $(FUSE_LIBS)<br></span>
+libvirt_driver_lxc_impl_la_LIBADD = $(CAPNG_LIBS) $(LIBNL_LIBS) $(LIBXML_LIBS) <a href="http://libvirt-lxc.la" rel="noreferrer" target="_blank">libvirt-lxc.la</a> $(FUSE_LIBS)<span class=""><br>
 if WITH_BLKID<br>
 libvirt_driver_lxc_impl_la_CFLAGS += $(BLKID_CFLAGS)<br>
 libvirt_driver_lxc_impl_la_LIBADD += $(BLKID_LIBS)<br></span>
@@ -2709,6 +2709,8 @@ libvirt_lxc_LDADD =                       \<br>
                <a href="http://libvirt-net-rpc.la" rel="noreferrer" target="_blank">libvirt-net-rpc.la</a> \<br>
                <a href="http://libvirt_security_manager.la" rel="noreferrer" target="_blank">libvirt_security_manager.la</a> \<br>
                <a href="http://libvirt_conf.la" rel="noreferrer" target="_blank">libvirt_conf.la</a> \<br>
+               <a href="http://libvirt.la" rel="noreferrer" target="_blank">libvirt.la</a> \<br>
+               <a href="http://libvirt-lxc.la" rel="noreferrer" target="_blank">libvirt-lxc.la</a> \<br>
                <a href="http://libvirt_util.la" rel="noreferrer" target="_blank">libvirt_util.la</a> \<br>
                ../gnulib/lib/<a href="http://libgnu.la" rel="noreferrer" target="_blank">libgnu.la</a><span class=""><br>
 if WITH_DTRACE_PROBES<br>
diff --git a/src/lxc/lxc_conf.c b/src/lxc/lxc_conf.c<br>
index c393cb5..96a0f47 100644<br>
--- a/src/lxc/lxc_conf.c<br>
+++ b/src/lxc/lxc_conf.c<br>
@@ -213,7 +213,7 @@ lxcDomainXMLConfInit(void)<br>
 {<br>
     return virDomainXMLOptionNew(&virLXCDriverDomainDefParserConfig,<br>
                                  &virLXCDriverPrivateDataCallbacks,<br>
-                                 NULL);<br>
+                                 &virLXCDriverDomainXMLNamespace);<br>
 }<br>
<br>
<br>
diff --git a/src/lxc/lxc_conf.h b/src/lxc/lxc_conf.h<br></span>
index 8340b1f..72b1d44 100644<br>
--- a/src/lxc/lxc_conf.h<br>
+++ b/src/lxc/lxc_conf.h<br>
@@ -67,6 +67,21 @@ struct _virLXCDriverConfig {<span class=""><br>
     bool securityRequireConfined;<br>
 };<br>
<br>
+<br>
+typedef enum {<br>
+    VIR_DOMAIN_NAMESPACE_SHARENET = 0,<br>
+    VIR_DOMAIN_NAMESPACE_SHAREIPC,<br>
+    VIR_DOMAIN_NAMESPACE_SHAREUTS,<br>
+    VIR_DOMAIN_NAMESPACE_LAST,<br>
+} virDomainNamespace;<br>
+<br></span><span class="">
+typedef struct _lxcDomainDef lxcDomainDef;<br>
+typedef lxcDomainDef *lxcDomainDefPtr;<br>
+struct _lxcDomainDef {<br></span>
+    char *ns_type[VIR_DOMAIN_NAMESPACE_LAST];<span class=""><br>
+    char *ns_val[VIR_DOMAIN_NAMESPACE_LAST];<br>
+};<br>
+<br>
 struct _virLXCDriver {<br>
     virMutex lock;<br>
<br>
diff --git a/src/lxc/lxc_container.c b/src/lxc/lxc_container.c<br></span>
index 11e9514..d8362ab 100644<br>
--- a/src/lxc/lxc_container.c<br>
+++ b/src/lxc/lxc_container.c<br>
@@ -27,6 +27,7 @@<br>
 #include <config.h><span class=""><br>
<br>
 #include <fcntl.h><br>
+#include <sched.h><br>
 #include <limits.h><br>
 #include <stdlib.h><br>
 #include <stdio.h><br></span>
@@ -38,7 +39,6 @@<span class=""><br>
 #include <mntent.h><br>
 #include <sys/reboot.h><br>
 #include <linux/reboot.h><br>
-<br>
 /* Yes, we want linux private one, for _syscall2() macro */<br>
 #include <linux/unistd.h><br>
<br></span>
@@ -2321,6 +2321,181 @@ virArch lxcContainerGetAlt32bitArch(virArch arch)<br>
     return VIR_ARCH_NONE;<br>
 }<br><span class="">
<br>
+struct ns_info {<br>
+        const char *proc_name;<br>
+        int clone_flag;<br></span>
+}ns_info_local[VIR_DOMAIN_NAMESPACE_LAST] = {<span class=""><br>
+    [VIR_DOMAIN_NAMESPACE_SHARENET] = {"net", CLONE_NEWNET},<br>
+    [VIR_DOMAIN_NAMESPACE_SHAREIPC] = {"ipc", CLONE_NEWIPC},<br>
+    [VIR_DOMAIN_NAMESPACE_SHAREUTS] = {"uts", CLONE_NEWUTS}<br>
+};<br>
+<br></span>
+static int lxcOpen_ns(lxcDomainDefPtr lxcDef, int ns_fd[VIR_DOMAIN_NAMESPACE_LAST])<br>
+{<br>
+    int i, n, rc = 0;<span class=""><br>
+    virDomainPtr dom = NULL;<br>
+    virConnectPtr conn = NULL;<br>
+    pid_t pid;<br>
+    int nfdlist;<br>
+    int *fdlist;<br>
+    char *path = NULL;<br>
+    char *eptr;<br></span>
+    for (i = 0; i < VIR_DOMAIN_NAMESPACE_LAST; i++)<span class=""><br>
+        ns_fd[i] = -1;<br>
+<br></span>
+    if (STREQ_NULLABLE("netns", lxcDef->ns_type[VIR_DOMAIN_NAMESPACE_SHARENET])) {<br>
+        if (virAsprintf(&path, "/var/run/netns/%s", lxcDef->ns_val[VIR_DOMAIN_NAMESPACE_SHARENET]) < 0)<br>
+            return  -1;<br>
+        ns_fd[VIR_DOMAIN_NAMESPACE_SHARENET] = open(path, O_RDONLY);<br>
+        VIR_FREE(path);<br>
+        if (ns_fd[VIR_DOMAIN_NAMESPACE_SHARENET] < 0) {<br>
+            virReportSystemError(errno,<br>
+                      _("failed to open netns %s"), lxcDef->ns_val[VIR_DOMAIN_NAMESPACE_SHARENET]);<span class=""><br>
+            return -1;<br>
+        }<br>
+    }<br></span>
+    for (i = 0; i < VIR_DOMAIN_NAMESPACE_LAST; i++) {<br>
+        /* If not yet intialized by above: netns*/<br>
+        if (lxcDef->ns_type[i] && ns_fd[i] == -1) {<br>
+            pid = strtol(lxcDef->ns_val[i], &eptr, 10);<span class=""><br>
+            if (*eptr != '\0' || pid < 1) {<br>
+                /* check if the domain is running, then set the namespaces<br>
+                 * to that container<br>
+                 */<br></span>
+                const char *ns[] = { "user", "ipc", "uts", "net", "pid", "mnt" };<span class=""><br>
+                conn = virConnectOpen("lxc:///");<br>
+                if (!conn) {<br></span>
+                    virReportError(virGetLastError()->code,<br>
+                      _("unable to get connect to lxc %s"), lxcDef->ns_val[i]);<br>
+                    rc = -1;<br>
+                    goto cleanup;<br>
+                }<br>
+                dom = virDomainLookupByName(conn, lxcDef->ns_val[i]);<br>
+                if (!dom) {<br>
+                    virReportError(virGetLastError()->code,<br>
+                                   _("Unable to lookup peer containeri %s"),<br>
+                                   lxcDef->ns_val[i]);<br>
+                    rc = -1;<br>
+                    goto cleanup;<br>
+                }<span class=""><br>
+                if ((nfdlist = virDomainLxcOpenNamespace(dom, &fdlist, 0)) < 0) {<br></span>
+                    virReportError(virGetLastError()->code,<br>
+                                   _("Unable to open %s"), lxcDef->ns_val[i]);<br>
+                    rc = -1;<br>
+                    goto cleanup;<br>
+                }<br>
+                for (n = 0; n < ARRAY_CARDINALITY(ns); n++) {<br>
+                    if (STREQ(ns[n], ns_info_local[i].proc_name)) {<br>
+                        ns_fd[i] = fdlist[n];<br>
+                    } else {<br>
+                        if (VIR_CLOSE(fdlist[n]) < 0)<br>
+                           VIR_ERROR(_("failed to close fd. ignoring.."));<span class=""><br>
+                    }<br>
+                }<br>
+                if (nfdlist > 0)<br></span>
+                    VIR_FREE(fdlist);<br>
+            } else {<br>
+                if (virAsprintf(&path, "/proc/%d/ns/%s", pid, ns_info_local[i].proc_name) < 0)<br>
+                    return -1;<br>
+                ns_fd[i] = open(path, O_RDONLY);<br>
+                VIR_FREE(path);<span class=""><br>
+                if (ns_fd[i] < 0) {<br></span>
+                    virReportSystemError(errno,<br>
+                      _("failed to open ns %s"), lxcDef->ns_val[i]);<span class=""><br>
+                    return -1;<br>
+                }<br>
+            }<br>
+        }<br></span>
+    }<br>
+ cleanup:<span class=""><br>
+    if (dom)<br>
+        virDomainFree(dom);<br></span>
+    if (conn)<br>
+        virConnectClose(conn);<br>
+    return rc;<br>
+}<br>
+<br>
+<br>
+static void lxcClose_ns(int ns_fd[VIR_DOMAIN_NAMESPACE_LAST])<div><div class="h5"><br>
+{<br>
+    int i;<br>
+    for (i = 0; i < VIR_DOMAIN_NAMESPACE_LAST; i++) {<br>
+        if (ns_fd[i] > -1) {<br>
+            if (VIR_CLOSE(ns_fd[i]) < 0)<br>
+                virReportSystemError(errno, "%s", _("failed to close file"));<br>
+            ns_fd[i] = -1;<br>
+        }<br>
+    }<br>
+}<br>
+<br>
+<br>
+/**<br>
+ * lxcPreserve_ns:<br>
+ * @ns_fd: array to store current namespace<br>
+ * @clone_flags: namespaces that need to be preserved<br>
+ */<br>
+static int lxcPreserve_ns(int ns_fd[VIR_DOMAIN_NAMESPACE_LAST], int clone_flags)<br>
+{<br>
+    int i, saved_errno;<br>
+    char *path = NULL;<br>
+<br>
+    for (i = 0; i < VIR_DOMAIN_NAMESPACE_LAST; i++)<br>
+        ns_fd[i] = -1;<br>
+<br></div></div>
+    if (!virFileExists("/proc/self/ns")) {<span class=""><br>
+        virReportSystemError(errno, "%s",<br>
+                             _("Kernel does not support attach; preserve_ns ignored"));<br></span>
+        return -1;<span class=""><br>
+    }<br>
+<br>
+    for (i = 0; i < VIR_DOMAIN_NAMESPACE_LAST; i++) {<br>
+        if ((clone_flags & ns_info_local[i].clone_flag) == 0)<br>
+            continue;<br>
+        if (virAsprintf(&path, "/proc/self/ns/%s",<br>
+                         ns_info_local[i].proc_name) < 0)<br>
+              goto error;<br></span><span class="">
+        ns_fd[i] = open(path, O_RDONLY | O_CLOEXEC);<br>
+        if (ns_fd[i] < 0)<br>
+            goto error;<br></span>
+        VIR_FREE(path);<br>
+    }<br>
+    return 0;<span class=""><br>
+ error:<br>
+    saved_errno = errno;<br></span>
+    lxcClose_ns(ns_fd);<br>
+    errno = saved_errno;<br>
+    virReportSystemError(errno, _("lxcPreserve_ns failed for '%s'"), path);<br>
+    VIR_FREE(path);<span class=""><br>
+    return -1;<br>
+}<br>
+<br>
+/**<br>
+ * lxcAttach_ns:<br>
+ * @ns_fd: array of namespaces to attach<br>
+ */<br>
+static int lxcAttach_ns(const int ns_fd[VIR_DOMAIN_NAMESPACE_LAST])<br>
+{<br>
+    int i;<br>
+<br>
+    for (i = 0; i < VIR_DOMAIN_NAMESPACE_LAST; i++) {<br>
+        if (ns_fd[i] < 0)<br>
+            continue;<br>
+         VIR_DEBUG("Setting into namespace\n");<br></span>
+        /* We get EINVAL if new NS is same as the current<span class=""><br>
+         * NS, or if the fd namespace doesn't match the<br>
+         * type passed to setns()'s second param. Since we<br>
+         * pass 0, we know the EINVAL is harmless<br>
+         */<br>
+        if (setns(ns_fd[i], 0) < 0 &&<br>
+            errno != EINVAL) {<br></span>
+            virReportSystemError(errno, _("failed to set namespace '%s'")<span class=""><br>
+                                 , ns_info_local[i].proc_name);<br>
+            return -1;<br>
+        }<br>
+    }<br></span><span class="">
+    return 0;<br>
+}<br>
+<br>
<br></span>
 /**<br>
  * lxcContainerStart:<br>
@@ -2346,9 +2521,13 @@ int lxcContainerStart(virDomainDefPtr def,<span class=""><br>
                       char **ttyPaths)<br>
 {<br>
     pid_t pid;<br>
-    int cflags;<br>
+    int cflags, i;<br>
     int stacksize = getpagesize() * 4;<br>
     char *stack, *stacktop;<br>
+    int saved_ns_fd[VIR_DOMAIN_NAMESPACE_LAST];<br></span>
+    int ns_inherit_fd[VIR_DOMAIN_NAMESPACE_LAST];<span class=""><br>
+    int preserve_mask = 0;<br>
+    lxcDomainDefPtr lxcDef;<br>
     lxc_child_argv_t args = {<br>
         .config = def,<br>
         .securityDriver = securityDriver,<br></span>
@@ -2368,7 +2547,12 @@ int lxcContainerStart(virDomainDefPtr def,<span class=""><br>
<br>
     stacktop = stack + stacksize;<br>
<br>
-    cflags = CLONE_NEWPID|CLONE_NEWNS|CLONE_NEWUTS|CLONE_NEWIPC|SIGCHLD;<br>
+    lxcDef = def->namespaceData;<br>
+    for (i = 0; i < VIR_DOMAIN_NAMESPACE_LAST; i++)<br></span>
+       if (lxcDef && lxcDef->ns_type[i])<span class=""><br>
+           preserve_mask |= ns_info_local[i].clone_flag;<br>
+<br></span>
+    cflags = CLONE_NEWPID|CLONE_NEWNS|SIGCHLD;<br>
<br>
     if (userns_required(def)) {<br>
         if (userns_supported()) {<br>
@@ -2381,10 +2565,43 @@ int lxcContainerStart(virDomainDefPtr def,<br>
             return -1;<br>
         }<br>
     }<br>
+    if (!lxcDef || !lxcDef->ns_type[VIR_DOMAIN_NAMESPACE_SHARENET]) {<span class=""><br>
+        if (lxcNeedNetworkNamespace(def)) {<br>
+            VIR_DEBUG("Enable network namespaces");<br>
+            cflags |= CLONE_NEWNET;<br>
+        }<br>
+    } else {<br>
+        VIR_DEBUG("Inheriting a net namespace");<br>
+    }<br></span>
+<br>
+    if (!lxcDef || !lxcDef->ns_type[VIR_DOMAIN_NAMESPACE_SHAREIPC]) {<span class=""><br>
+        cflags |= CLONE_NEWIPC;<br>
+    } else {<br>
+        VIR_DEBUG("Inheriting an IPC namespace");<br>
+    }<br>
+<br></span>
+    if (!lxcDef || !lxcDef->ns_type[VIR_DOMAIN_NAMESPACE_SHAREUTS]) {<span class=""><br>
+        cflags |= CLONE_NEWUTS;<br>
+    } else {<br>
+        VIR_DEBUG("Inheriting a UTS namespace");<br>
+    }<br>
+<br>
+    if (lxcDef && lxcPreserve_ns(saved_ns_fd, preserve_mask) < 0) {<br>
+        virReportError(VIR_ERR_SYSTEM_ERROR, "%s",<br>
+                       _("failed to preserve the namespace"));<br>
+        return -1;<br>
+    }<br>
<br></span><span class="">
-    if (lxcNeedNetworkNamespace(def)) {<br>
-        VIR_DEBUG("Enable network namespaces");<br>
-        cflags |= CLONE_NEWNET;<br></span>
+    if (lxcDef && lxcOpen_ns(lxcDef, ns_inherit_fd)) {<br>
+        virReportError(VIR_ERR_SYSTEM_ERROR, "%s",<br>
+                       _("failed to open the namespace"));<span class=""><br>
+        return -1;<br>
+    }<br>
+<br></span>
+    if (lxcDef && lxcAttach_ns(ns_inherit_fd) < 0) {<span class=""><br>
+        virReportError(VIR_ERR_SYSTEM_ERROR, "%s",<br>
+                       _("failed to attach the namespace"));<br>
+        return -1;<br>
     }<br>
<br>
     VIR_DEBUG("Cloning container init process");<br></span>
@@ -2397,7 +2614,14 @@ int lxcContainerStart(virDomainDefPtr def,<span class=""><br>
                              _("Failed to run clone container"));<br>
         return -1;<br>
     }<br>
+    if (lxcDef && lxcAttach_ns(saved_ns_fd)) {<br>
+        virReportError(VIR_ERR_SYSTEM_ERROR, "%s",<br>
+                             _("failed to restore saved namespaces"));<br>
+    }<br>
<br></span>
+    /* clean up */<br>
+    if (lxcDef)<br>
+        lxcClose_ns(ns_inherit_fd);<span class=""><br>
     return pid;<br>
 }<br>
<br>
diff --git a/src/lxc/lxc_domain.c b/src/lxc/lxc_domain.c<br></span>
index 70606f3..5e63969 100644<br>
--- a/src/lxc/lxc_domain.c<br>
+++ b/src/lxc/lxc_domain.c<br>
@@ -26,8 +26,14 @@<span class=""><br>
 #include "viralloc.h"<br>
 #include "virlog.h"<br>
 #include "virerror.h"<br>
+#include <fcntl.h><br>
+#include <libxml/xpathInternals.h><br>
+#include "virstring.h"<br>
+#include "virutil.h"<br>
+#include "virfile.h"<br>
<br>
 #define VIR_FROM_THIS VIR_FROM_LXC<br></span>
+#define LXC_NAMESPACE_HREF "<a href="http://libvirt.org/schemas/domain/lxc/1.0" rel="noreferrer" target="_blank">http://libvirt.org/schemas/domain/lxc/1.0</a>"<br>
<br>
 VIR_LOG_INIT("lxc.lxc_domain");<br>
<br>
@@ -41,6 +47,163 @@ static void *virLXCDomainObjPrivateAlloc(void)<br>
     return priv;<br>
 }<br><span class="">
<br>
+VIR_ENUM_DECL(virDomainNamespace)<br>
+VIR_ENUM_IMPL(virDomainNamespace, VIR_DOMAIN_NAMESPACE_LAST,<br>
+    N_("sharenet"),<br>
+    N_("shareipc"),<br>
+    N_("shareuts"))<br>
+<br>
+static void<br>
+lxcDomainDefNamespaceFree(void *nsdata)<br>
+{<br>
+    int j;<br>
+    lxcDomainDefPtr lxcDef = nsdata;<br>
+    for (j = 0; j < VIR_DOMAIN_NAMESPACE_LAST; j++) {<br></span>
+       VIR_FREE(lxcDef->ns_type[j]);<br>
+       VIR_FREE(lxcDef->ns_val[j]);<br>
+    }<br>
+    VIR_FREE(nsdata);<span class=""><br>
+}<br>
+<br>
+static int<br>
+lxcDomainDefNamespaceParse(xmlDocPtr xml ATTRIBUTE_UNUSED,<br>
+                           xmlNodePtr root ATTRIBUTE_UNUSED,<br>
+                           xmlXPathContextPtr ctxt,<br>
+                           void **data)<br>
+{<br>
+    lxcDomainDefPtr lxcDef = NULL;<br>
+    xmlNodePtr *nodes = NULL;<br>
+    bool uses_lxc_ns = false;<br>
+    xmlNodePtr node;<br>
+    int feature;<br>
+    int n;<br>
+    char *tmp = NULL;<br>
+    size_t i;<br>
+<br></span><span class="">
+    if (xmlXPathRegisterNs(ctxt, BAD_CAST "lxc", BAD_CAST LXC_NAMESPACE_HREF) < 0) {<br>
+        virReportError(VIR_ERR_INTERNAL_ERROR,<br>
+                       _("Failed to register xml namespace '%s'"),<br>
+                       LXC_NAMESPACE_HREF);<br>
+        return -1;<br>
+    }<br>
+<br>
+    if (VIR_ALLOC(lxcDef) < 0)<br>
+        return -1;<br></span>
+    /* Init ns_herit_fd for namespaces */<span class=""><br>
+    for (i = 0; i < VIR_DOMAIN_NAMESPACE_LAST; i++) {<br></span>
+        lxcDef->ns_type[i] = NULL;<div><div class="h5"><br>
+        lxcDef->ns_val[i] = NULL;<br>
+    }<br>
+<br>
+    node = ctxt->node;<br>
+    if ((n = virXPathNodeSet("./lxc:namespace/*", ctxt, &nodes)) < 0)<br>
+        goto error;<br>
+    uses_lxc_ns |= n > 0;<br>
+<br>
+    for (i = 0; i < n; i++) {<br>
+        feature =<br>
+            virDomainNamespaceTypeFromString((const char *) nodes[i]->name);<br>
+        if (feature < 0) {<br>
+            virReportError(VIR_ERR_CONFIG_UNSUPPORTED,<br>
+                            _("unsupported Namespace feature: %s"),<br>
+                            nodes[i]->name);<br>
+            goto error;<br>
+        }<br>
+<br>
+        ctxt->node = nodes[i];<br>
+<br>
+        switch ((virDomainNamespace) feature) {<br>
+        case VIR_DOMAIN_NAMESPACE_SHARENET:<br>
+        case VIR_DOMAIN_NAMESPACE_SHAREIPC:<br>
+        case VIR_DOMAIN_NAMESPACE_SHAREUTS:<br>
+            {<br>
+                tmp = virXMLPropString(nodes[i], "type");<br>
+                if (tmp == NULL) {<br>
+                    virReportError(VIR_ERR_INTERNAL_ERROR,<br>
+                                   "%s", _("No lxc environment type specified"));<br>
+                    goto error;<br>
+                }<br>
+                /* save the tmp so that its needed while writing to xml */<br>
+                lxcDef->ns_type[feature] = tmp;<br>
+                tmp = virXMLPropString(nodes[i], "value");<br>
+                if (tmp == NULL) {<br>
+                    virReportError(VIR_ERR_INTERNAL_ERROR,<br>
+                                   "%s", _("No lxc environment type specified"));<br>
+                    goto error;<br>
+                }<br>
+                lxcDef->ns_val[feature] = tmp;<br>
+            }<br></div></div><div><div class="h5">
+            break;<br>
+        case VIR_DOMAIN_NAMESPACE_LAST:<br>
+            break;<br>
+        }<br>
+    }<br>
+    VIR_FREE(nodes);<br>
+    ctxt->node = node;<br>
+    if (uses_lxc_ns)<br>
+        *data = lxcDef;<br>
+    else<br>
+        VIR_FREE(lxcDef);<br>
+    return 0;<br>
+ error:<br>
+    VIR_FREE(nodes);<br>
+    lxcDomainDefNamespaceFree(lxcDef);<br>
+    return -1;<br>
+}<br>
+<br>
+<br>
+static int<br>
+lxcDomainDefNamespaceFormatXML(virBufferPtr buf,<br>
+                               void *nsdata)<br>
+{<br>
+    lxcDomainDefPtr lxcDef = nsdata;<br>
+    size_t j;<br>
+<br>
+    if (!lxcDef)<br>
+       return 0;<br>
+<br>
+    virBufferAddLit(buf, "<lxc:namespace>\n");<br>
+    virBufferAdjustIndent(buf, 2);<br>
+<br>
+    for (j = 0; j < VIR_DOMAIN_NAMESPACE_LAST; j++) {<br>
+        switch ((virDomainNamespace) j) {<br>
+        case VIR_DOMAIN_NAMESPACE_SHAREIPC:<br>
+        case VIR_DOMAIN_NAMESPACE_SHAREUTS:<br>
+        case VIR_DOMAIN_NAMESPACE_SHARENET:<br>
+            {<br></div></div>
+                if (lxcDef->ns_type[j]) {<br>
+                    virBufferAsprintf(buf, "<lxc:%s type='%s' value='%s'/>\n",<div><div class="h5"><br>
+                                      virDomainNamespaceTypeToString(j),<br>
+                                      lxcDef->ns_type[j],<br>
+                                      lxcDef->ns_val[j]);<br>
+                }<br>
+            }<br>
+            break;<br>
+        case VIR_DOMAIN_NAMESPACE_LAST:<br>
+            break;<br>
+        }<br>
+    }<br>
+<br>
+    virBufferAdjustIndent(buf, -2);<br>
+    virBufferAddLit(buf, "</lxc:namespace>\n");<br>
+    return 0;<br>
+}<br>
+<br>
+static const char *<br>
+lxcDomainDefNamespaceHref(void)<br>
+{<br>
+    return "xmlns:lxc='" LXC_NAMESPACE_HREF "'";<br>
+}<br>
+<br>
+<br>
+virDomainXMLNamespace virLXCDriverDomainXMLNamespace = {<br>
+    .parse = lxcDomainDefNamespaceParse,<br>
+    .free = lxcDomainDefNamespaceFree,<br>
+    .format = lxcDomainDefNamespaceFormatXML,<br>
+    .href = lxcDomainDefNamespaceHref,<br>
+};<br>
+<br>
+<br>
 static void virLXCDomainObjPrivateFree(void *data)<br>
 {<br>
     virLXCDomainObjPrivatePtr priv = data;<br></div></div>
@@ -77,7 +240,6 @@ virLXCDomainObjPrivateXMLParse(xmlXPathContextPtr ctxt,<span class=""><br>
     } else {<br>
         priv->initpid = thepid;<br>
     }<br>
-<br>
     return 0;<br>
 }<br>
<br></span><span class="">
diff --git a/src/lxc/lxc_domain.h b/src/lxc/lxc_domain.h<br>
index 751aece..25df999 100644<br>
--- a/src/lxc/lxc_domain.h<br>
+++ b/src/lxc/lxc_domain.h<br>
@@ -41,6 +41,7 @@ struct _virLXCDomainObjPrivate {<br>
     virCgroupPtr cgroup;<br>
 };<br>
<br>
+extern virDomainXMLNamespace virLXCDriverDomainXMLNamespace;<br>
 extern virDomainXMLPrivateDataCallbacks virLXCDriverPrivateDataCallbacks;<br>
 extern virDomainDefParserConfig virLXCDriverDomainDefParserConfig;<br>
<br></span>
diff --git a/tests/lxcxml2xmldata/lxc-sharenet.xml b/tests/lxcxml2xmldata/lxc-sharenet.xml<br>
new file mode 100644<br>
index 0000000..a2b8d1b<br>
--- /dev/null<br>
+++ b/tests/lxcxml2xmldata/lxc-sharenet.xml<br>
@@ -0,0 +1,33 @@<br>
+<domain type='lxc' xmlns:lxc='<a href="http://libvirt.org/schemas/domain/lxc/1.0" rel="noreferrer" target="_blank">http://libvirt.org/schemas/domain/lxc/1.0</a>'><br>
+  <name>jessie</name><br>
+  <uuid>e21987a5-e98e-9c99-0e35-803e4d9ad1fe</uuid><br>
+  <memory unit='KiB'>1048576</memory><br>
+  <currentMemory unit='KiB'>1048576</currentMemory><br>
+  <vcpu placement='static'>1</vcpu><br>
+  <resource><br>
+    <partition>/machine</partition><br>
+  </resource><br>
+  <os><br>
+    <type arch='x86_64'>exe</type><br>
+    <init>/sbin/init</init><br>
+  </os><br>
+  <clock offset='utc'/><br>
+  <on_poweroff>destroy</on_poweroff><br>
+  <on_reboot>restart</on_reboot><br>
+  <on_crash>restart</on_crash><br>
+  <devices><br>
+    <emulator>/usr/libexec/libvirt_lxc</emulator><br>
+    <filesystem type='mount' accessmode='passthrough'><br>
+      <source dir='/mach/jessie'/><br>
+      <target dir='/'/><br>
+    </filesystem><br>
+    <console type='pty'><br>
+      <target type='lxc' port='0'/><br>
+    </console><br>
+  </devices><br>
+  <lxc:namespace><br>
+    <lxc:sharenet type='netns' value='red'/><br>
+    <lxc:shareipc type='pid' value='12345'/><br>
+    <lxc:shareuts type='name' value='container1'/><br>
+  </lxc:namespace><br>
+</domain><br>
diff --git a/tests/lxcxml2xmltest.c b/tests/lxcxml2xmltest.c<br>
index 3e00347..8d824b9 100644<br>
--- a/tests/lxcxml2xmltest.c<br>
+++ b/tests/lxcxml2xmltest.c<br>
@@ -133,6 +133,7 @@ mymain(void)<br>
     DO_TEST("filesystem-root");<br>
     DO_TEST("idmap");<br>
     DO_TEST("capabilities");<br>
+    DO_TEST("sharenet");<br>
<br>
     virObjectUnref(caps);<br>
     virObjectUnref(xmlopt);<br><br>-----------------------<br></div>-imran  <div><div><div><div><br><br></div></div></div></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="h5">On Thu, Jun 11, 2015 at 5:47 PM, Michal Privoznik <span dir="ltr"><<a href="mailto:mprivozn@redhat.com" target="_blank">mprivozn@redhat.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div class="h5"><span>On 21.05.2015 19:43, ik.nitk wrote:<br>
</span><span>>  This patch tries to add the similar option to libvirt lxc. So to inherit namespace from name<br>
>  container c2.<br>
>  add this into xml.<br>
>      <lxc:namespace><br>
>          <sharenet type='name' value='c2'/><br>
>      </lxc:namespace><br>
><br>
>  And to inherit namespace from a pid.<br>
>  add this into xml.<br>
>      <lxc:namespace><br>
>          <sharenet type='pid' value='10245'/><br>
>      </lxc:namespace><br>
><br>
>  And to inherit namespace from a netns.<br>
>  add this into xml.<br>
>      <lxc:namespace><br>
>          <sharenet type='netns' value='red'/><br>
>      </lxc:namespace><br>
><br>
>  Similar options for ipc/uts.<br>
>      <shareipc    /> , <shareuts /><br>
><br>
>  The reasong lxc xml namespace is added because this feature is very specific to lxc. Therfore wanted to<br>
>  keep it seperated from actual libvirt xml domain.<br>
><br>
>  -imran<br>
> ---<br>
<br>
</span>The subject line is just too long. Look at git log to see the style we<br>
use to write commit messages.<br>
<span><br>
>  src/Makefile.am         |   5 +-<br>
>  src/lxc/lxc_conf.c      |   2 +-<br>
>  src/lxc/lxc_conf.h      |  23 +++++<br>
>  src/lxc/lxc_container.c | 191 ++++++++++++++++++++++++++++++++++--<br>
>  src/lxc/lxc_domain.c    | 254 +++++++++++++++++++++++++++++++++++++++++++++++-<br>
>  src/lxc/lxc_domain.h    |   1 +<br>
>  6 files changed, 463 insertions(+), 13 deletions(-)<br>
<br>
<br>
</span>You are introducing new elements and namespace to the XML. This must<br>
always go hand in hand with RNG schema adjustment and a test case or two<br>
under tests/. I NACK every patch that does not comply with this rule.<br>
But let me review the rest.<br>
</div></div><span><div><div class="h5"><br>
><br>
> diff --git a/src/Makefile.am b/src/Makefile.am<br>
> index 579421d..1a78fde 100644<br>
> --- a/src/Makefile.am<br>
> +++ b/src/Makefile.am<br>
> @@ -1293,7 +1293,8 @@ libvirt_driver_lxc_impl_la_CFLAGS = \<br>
>               -I$(srcdir)/access \<br>
>               -I$(srcdir)/conf \<br>
>               $(AM_CFLAGS)<br>
> -libvirt_driver_lxc_impl_la_LIBADD = $(CAPNG_LIBS) $(LIBNL_LIBS) $(FUSE_LIBS)<br>
> +libvirt_driver_lxc_impl_la_LIBADD = $(CAPNG_LIBS) $(LIBNL_LIBS) $(LIBXML_LIBS) $(FUSE_LIBS)<br></div></div>
> +libvirt_driver_lxc_impl_la_LDFLAGS = <a href="http://libvirt-lxc.la" rel="noreferrer" target="_blank">libvirt-lxc.la</a><br>
<br>
</span>This won't fly. If you need <a href="http://libvirt-lxc.la" rel="noreferrer" target="_blank">libvirt-lxc.la</a> to be added, you must put it<span class=""><br>
into LIBADD. Otherwise automake will fail to see the dependency tree. It<br>
happened to me when I was building this with -j5. Although, this won't<br>
be needed at all IMO, but more on that later.<br>
</span><div><div><span class=""><br>
>  if WITH_BLKID<br>
>  libvirt_driver_lxc_impl_la_CFLAGS += $(BLKID_CFLAGS)<br>
>  libvirt_driver_lxc_impl_la_LIBADD += $(BLKID_LIBS)<br>
> @@ -2652,6 +2653,8 @@ libvirt_lxc_LDADD =                     \<br></span>
>               <a href="http://libvirt-net-rpc.la" rel="noreferrer" target="_blank">libvirt-net-rpc.la</a> \<br>
>               <a href="http://libvirt_security_manager.la" rel="noreferrer" target="_blank">libvirt_security_manager.la</a> \<br>
>               <a href="http://libvirt_conf.la" rel="noreferrer" target="_blank">libvirt_conf.la</a> \<br>
> +             <a href="http://libvirt.la" rel="noreferrer" target="_blank">libvirt.la</a> \<br>
> +             <a href="http://libvirt-lxc.la" rel="noreferrer" target="_blank">libvirt-lxc.la</a> \<br>
>               <a href="http://libvirt_util.la" rel="noreferrer" target="_blank">libvirt_util.la</a> \<br>
>               ../gnulib/lib/<a href="http://libgnu.la" rel="noreferrer" target="_blank">libgnu.la</a><div><div class="h5"><br>
>  if WITH_DTRACE_PROBES<br>
> diff --git a/src/lxc/lxc_conf.c b/src/lxc/lxc_conf.c<br>
> index c393cb5..96a0f47 100644<br>
> --- a/src/lxc/lxc_conf.c<br>
> +++ b/src/lxc/lxc_conf.c<br>
> @@ -213,7 +213,7 @@ lxcDomainXMLConfInit(void)<br>
>  {<br>
>      return virDomainXMLOptionNew(&virLXCDriverDomainDefParserConfig,<br>
>                                   &virLXCDriverPrivateDataCallbacks,<br>
> -                                 NULL);<br>
> +                                 &virLXCDriverDomainXMLNamespace);<br>
>  }<br>
><br>
><br>
> diff --git a/src/lxc/lxc_conf.h b/src/lxc/lxc_conf.h<br>
> index 8340b1f..59002e5 100644<br>
> --- a/src/lxc/lxc_conf.h<br>
> +++ b/src/lxc/lxc_conf.h<br>
> @@ -67,6 +67,29 @@ struct _virLXCDriverConfig {<br>
>      bool securityRequireConfined;<br>
>  };<br>
><br>
> +<br>
> +typedef enum {<br>
> +    VIR_DOMAIN_NAMESPACE_SHARENET = 0,<br>
> +    VIR_DOMAIN_NAMESPACE_SHAREIPC,<br>
> +    VIR_DOMAIN_NAMESPACE_SHAREUTS,<br>
> +    VIR_DOMAIN_NAMESPACE_LAST,<br>
> +} virDomainNamespace;<br>
> +<br>
> +struct ns_info {<br>
> +    const char *proc_name;<br>
> +    int clone_flag;<br>
> +};<br>
> +<br>
> +extern const struct ns_info ns_info[VIR_DOMAIN_NAMESPACE_LAST];<br>
> +<br>
> +typedef struct _lxcDomainDef lxcDomainDef;<br>
> +typedef lxcDomainDef *lxcDomainDefPtr;<br>
> +struct _lxcDomainDef {<br>
> +    int ns_inherit_fd[VIR_DOMAIN_NAMESPACE_LAST];<br>
> +    char *ns_type[VIR_DOMAIN_NAMESPACE_LAST];<br>
> +    char *ns_val[VIR_DOMAIN_NAMESPACE_LAST];<br>
> +};<br>
> +<br>
>  struct _virLXCDriver {<br>
>      virMutex lock;<br>
><br>
> diff --git a/src/lxc/lxc_container.c b/src/lxc/lxc_container.c<br>
> index 9a9ae5c..a9a7ba0 100644<br>
> --- a/src/lxc/lxc_container.c<br>
> +++ b/src/lxc/lxc_container.c<br>
> @@ -25,8 +25,8 @@<br>
>   */<br>
><br>
>  #include <config.h><br>
> -<br>
<br>
</div></div></div></div><div><div class="h5">No, we like the extra space here. config.h has a special status.<br>
<div><div><br>
>  #include <fcntl.h><br>
> +#include <sched.h><br>
>  #include <limits.h><br>
>  #include <stdlib.h><br>
>  #include <stdio.h><br>
> @@ -38,7 +38,6 @@<br>
>  #include <mntent.h><br>
>  #include <sys/reboot.h><br>
>  #include <linux/reboot.h><br>
> -<br>
>  /* Yes, we want linux private one, for _syscall2() macro */<br>
>  #include <linux/unistd.h><br>
><br>
> @@ -99,6 +98,50 @@ VIR_LOG_INIT("lxc.lxc_container");<br>
>  typedef char lxc_message_t;<br>
>  #define LXC_CONTINUE_MSG 'c'<br>
><br>
<br>
<br>
> +#ifdef __linux__<br>
> +/*<br>
> + * Workaround older glibc. While kernel may support the setns<br>
> + * syscall, the glibc wrapper might not exist. If that's the<br>
> + * case, use our own.<br>
> + */<br>
> +# ifndef __NR_setns<br>
> +#  if defined(__x86_64__)<br>
> +#   define __NR_setns 308<br>
> +#  elif defined(__i386__)<br>
> +#   define __NR_setns 346<br>
> +#  elif defined(__arm__)<br>
> +#   define __NR_setns 375<br>
> +#  elif defined(__aarch64__)<br>
> +#   define __NR_setns 375<br>
> +#  elif defined(__powerpc__)<br>
> +#   define __NR_setns 350<br>
> +#  elif defined(__s390__)<br>
> +#   define __NR_setns 339<br>
> +#  endif<br>
> +# endif<br>
> +<br>
> +# ifndef HAVE_SETNS<br>
> +#  if defined(__NR_setns)<br>
> +#   include <sys/syscall.h><br>
> +<br>
> +static inline int setns(int fd, int nstype)<br>
> +{<br>
> +    return syscall(__NR_setns, fd, nstype);<br>
> +}<br>
> +#  else /* !__NR_setns */<br>
> +#   error Please determine the syscall number for setns on your architecture<br>
> +#  endif<br>
> +# endif<br>
> +#else /* !__linux__ */<br>
> +static inline int setns(int fd ATTRIBUTE_UNUSED, int nstype ATTRIBUTE_UNUSED)<br>
> +{<br>
> +    virReportSystemError(ENOSYS, "%s",<br>
> +                         _("Namespaces are not supported on this platform."));<br>
> +    return -1;<br>
> +}<br>
> +#endif<br>
> +<br>
> +<br>
<br>
</div></div>This seems like copied over from util/virprocess.c. I think you can use<br>
setns() function defined there instead of redefining your own.<br>
<span><br>
>  typedef struct __lxc_child_argv lxc_child_argv_t;<br>
>  struct __lxc_child_argv {<br>
>      virDomainDefPtr config;<br>
> @@ -2233,7 +2276,6 @@ static int lxcContainerChild(void *data)<br>
>                      vmDef->os.init);<br>
>          goto cleanup;<br>
>      }<br>
> -<br>
<br>
</span>I think this empty line is intentional here.<br>
<div><div><br>
>      /* rename and enable interfaces */<br>
>      if (lxcContainerRenameAndEnableInterfaces(vmDef,<br>
>                                                argv->nveths,<br>
> @@ -2321,6 +2363,99 @@ virArch lxcContainerGetAlt32bitArch(virArch arch)<br>
>      return VIR_ARCH_NONE;<br>
>  }<br>
><br>
> +/* Used only for containers,same as the one defined in<br>
> + * domain_conf.c. But used locally<br>
> + */<br>
> +static const struct ns_info ns_info_local[VIR_DOMAIN_NAMESPACE_LAST] = {<br>
> +    [VIR_DOMAIN_NAMESPACE_SHARENET] = {"net", CLONE_NEWNET},<br>
> +    [VIR_DOMAIN_NAMESPACE_SHAREIPC] = {"ipc", CLONE_NEWIPC},<br>
> +    [VIR_DOMAIN_NAMESPACE_SHAREUTS] = {"uts", CLONE_NEWUTS}<br>
> +};<br>
> +<br>
> +<br>
> +static void close_ns(int ns_fd[VIR_DOMAIN_NAMESPACE_LAST])<br>
> +{<br>
> +    int i;<br>
> +    for (i = 0; i < VIR_DOMAIN_NAMESPACE_LAST; i++) {<br>
> +        if (ns_fd[i] > -1) {<br>
> +            if (VIR_CLOSE(ns_fd[i]) < 0)<br>
> +                virReportSystemError(errno, "%s", _("failed to close file"));<br>
> +            ns_fd[i] = -1;<br>
> +        }<br>
> +    }<br>
> +}<br>
> +<br>
> +<br>
> +/**<br>
> + * lxcPreserve_ns:<br>
> + * @ns_fd: array to store current namespace<br>
> + * @clone_flags: namespaces that need to be preserved<br>
> + */<br>
> +static int lxcPreserve_ns(int ns_fd[VIR_DOMAIN_NAMESPACE_LAST], int clone_flags)<br>
> +{<br>
> +    int i, saved_errno;<br>
> +    char *path = NULL;<br>
> +<br>
> +    for (i = 0; i < VIR_DOMAIN_NAMESPACE_LAST; i++)<br>
> +        ns_fd[i] = -1;<br>
> +<br>
> +    if (access("/proc/self/ns", X_OK)) {<br>
<br>
</div></div>virFileIsExecutable. Although I guess you want tot check if the file<br>
exists, in which case virFileExists is just enough.<br>
<span><br>
> +        virReportSystemError(errno, "%s",<br>
> +                             _("Kernel does not support attach; preserve_ns ignored"));<br>
> +        return 0;<br>
<br>
</span>So an error is reported, but success is claimed?<br>
<span><br>
> +    }<br>
> +<br>
> +    for (i = 0; i < VIR_DOMAIN_NAMESPACE_LAST; i++) {<br>
> +        if ((clone_flags & ns_info_local[i].clone_flag) == 0)<br>
> +            continue;<br>
> +        if (virAsprintf(&path, "/proc/self/ns/%s",<br>
> +                         ns_info_local[i].proc_name) < 0)<br>
> +              goto error;<br>
<br>
</span>If virAsprintf fails, an error is already reported. But due to 'goto<br>
error' you overwrite it with (wrong) error message.<br>
<span><br>
> +        ns_fd[i] = open(path, O_RDONLY | O_CLOEXEC);<br>
> +        if (ns_fd[i] < 0)<br>
> +            goto error;<br>
> +    }<br>
> +    VIR_FREE(path);<br>
> +    return 0;<br>
> + error:<br>
> +    saved_errno = errno;<br>
> +    close_ns(ns_fd);<br>
> +    errno = saved_errno;<br>
> +    VIR_FREE(path);<br>
> +    virReportSystemError(errno, _("failed to open '%s'"), path);<br>
<br>
</span>Ouch. @path is NULL at this point.<br>
<span><br>
> +    return -1;<br>
> +}<br>
> +<br>
> +/**<br>
> + * lxcAttach_ns:<br>
> + * @ns_fd: array of namespaces to attach<br>
> + */<br>
> +static int lxcAttach_ns(const int ns_fd[VIR_DOMAIN_NAMESPACE_LAST])<br>
> +{<br>
> +    int i;<br>
> +<br>
> +    for (i = 0; i < VIR_DOMAIN_NAMESPACE_LAST; i++) {<br>
> +        if (ns_fd[i] < 0)<br>
> +            continue;<br>
> +        VIR_DEBUG("Setting into namespace\n");<br>
> +<br>
> +        /* We get EINVAL if new NS is same as the current<br>
> +         * NS, or if the fd namespace doesn't match the<br>
> +         * type passed to setns()'s second param. Since we<br>
> +         * pass 0, we know the EINVAL is harmless<br>
> +         */<br>
> +        if (setns(ns_fd[i], 0) < 0 &&<br>
> +                    errno != EINVAL)<br>
> +            goto error;<br>
<br>
</span>Any reason why the block under 'error' label is not here directly?<br>
<div><div><br>
> +    }<br>
> +    return 0;<br>
> +<br>
> + error:<br>
> +    virReportSystemError(errno, _("failed to set namespace '%s'")<br>
> +                         , ns_info_local[i].proc_name);<br>
> +    return -1;<br>
> +}<br>
> +<br>
><br>
>  /**<br>
>   * lxcContainerStart:<br>
> @@ -2346,9 +2481,12 @@ int lxcContainerStart(virDomainDefPtr def,<br>
>                        char **ttyPaths)<br>
>  {<br>
>      pid_t pid;<br>
> -    int cflags;<br>
> +    int cflags, i;<br>
>      int stacksize = getpagesize() * 4;<br>
>      char *stack, *stacktop;<br>
> +    int saved_ns_fd[VIR_DOMAIN_NAMESPACE_LAST];<br>
> +    int preserve_mask = 0;<br>
> +    lxcDomainDefPtr lxcDef;<br>
>      lxc_child_argv_t args = {<br>
>          .config = def,<br>
>          .securityDriver = securityDriver,<br>
> @@ -2368,7 +2506,14 @@ int lxcContainerStart(virDomainDefPtr def,<br>
><br>
>      stacktop = stack + stacksize;<br>
><br>
> -    cflags = CLONE_NEWPID|CLONE_NEWNS|CLONE_NEWUTS|CLONE_NEWIPC|SIGCHLD;<br>
> +    lxcDef = def->namespaceData;<br>
> +    for (i = 0; i < VIR_DOMAIN_NAMESPACE_LAST; i++)<br>
> +       if (lxcDef && lxcDef->ns_inherit_fd[i] != -1)<br>
> +           preserve_mask |= ns_info_local[i].clone_flag;<br>
> +<br>
> +<br>
> +<br>
> +    cflags = CLONE_NEWPID|CLONE_NEWNS|SIGCHLD;<br>
><br>
>      if (userns_required(def)) {<br>
>          if (userns_supported()) {<br>
> @@ -2381,10 +2526,36 @@ int lxcContainerStart(virDomainDefPtr def,<br>
>              return -1;<br>
>          }<br>
>      }<br>
> +    if (!lxcDef || (lxcDef && lxcDef->ns_inherit_fd[VIR_DOMAIN_NAMESPACE_SHARENET] == -1)) {<br>
<br>
</div></div>C language short-cuircuit conditions evaluation. Therefore, if lxcDef is<br>
NULL, the first part of the condition will direct control into the if()<br>
body. Therefore there's no need to check:<br>
<br>
if (!a || (a && a->something));<br>
<br>
It's equivallent to:<br>
<br>
if (!a || a->something);<br>
<div><div><br>
> +        if (lxcNeedNetworkNamespace(def)) {<br>
> +            VIR_DEBUG("Enable network namespaces");<br>
> +            cflags |= CLONE_NEWNET;<br>
> +        }<br>
> +    } else {<br>
> +        VIR_DEBUG("Inheriting a net namespace");<br>
> +    }<br>
><br>
> -    if (lxcNeedNetworkNamespace(def)) {<br>
> -        VIR_DEBUG("Enable network namespaces");<br>
> -        cflags |= CLONE_NEWNET;<br>
> +    if (!lxcDef || (lxcDef && lxcDef->ns_inherit_fd[VIR_DOMAIN_NAMESPACE_SHAREIPC] == -1)) {<br>
> +        cflags |= CLONE_NEWIPC;<br>
> +    } else {<br>
> +        VIR_DEBUG("Inheriting an IPC namespace");<br>
> +    }<br>
> +<br>
> +    if (!lxcDef || (lxcDef && lxcDef->ns_inherit_fd[VIR_DOMAIN_NAMESPACE_SHAREUTS] == -1)) {<br>
> +        cflags |= CLONE_NEWUTS;<br>
> +    } else {<br>
> +        VIR_DEBUG("Inheriting a UTS namespace");<br>
> +    }<br>
> +<br>
> +    if (lxcDef && lxcPreserve_ns(saved_ns_fd, preserve_mask) < 0) {<br>
> +        virReportError(VIR_ERR_SYSTEM_ERROR, "%s",<br>
> +                       _("failed to preserve the namespace"));<br>
> +        return -1;<br>
> +    }<br>
> +    if (lxcDef && lxcAttach_ns(lxcDef->ns_inherit_fd) < 0) {<br>
> +        virReportError(VIR_ERR_SYSTEM_ERROR, "%s",<br>
> +                       _("failed to attach the namespace"));<br>
> +        return -1;<br>
>      }<br>
><br>
>      VIR_DEBUG("Cloning container init process");<br>
> @@ -2397,6 +2568,10 @@ int lxcContainerStart(virDomainDefPtr def,<br>
>                               _("Failed to run clone container"));<br>
>          return -1;<br>
>      }<br>
> +    if (lxcDef && lxcAttach_ns(saved_ns_fd)) {<br>
> +        virReportError(VIR_ERR_SYSTEM_ERROR, "%s",<br>
> +                             _("failed to restore saved namespaces"));<br>
> +    }<br>
><br>
>      return pid;<br>
>  }<br>
> diff --git a/src/lxc/lxc_domain.c b/src/lxc/lxc_domain.c<br>
> index c2180cb..6e4a19a 100644<br>
> --- a/src/lxc/lxc_domain.c<br>
> +++ b/src/lxc/lxc_domain.c<br>
> @@ -20,14 +20,18 @@<br>
>   */<br>
><br>
>  #include <config.h><br>
> -<br>
>  #include "lxc_domain.h"<br>
> -<br>
<br>
</div></div>I've raised this already.<br>
</div></div><div><div><div><div class="h5"><br>
>  #include "viralloc.h"<br>
>  #include "virlog.h"<br>
>  #include "virerror.h"<br>
> +#include <fcntl.h><br>
> +#include <libxml/xpathInternals.h><br>
> +#include "virstring.h"<br>
> +#include "virutil.h"<br>
> +#include "virfile.h"<br>
><br>
>  #define VIR_FROM_THIS VIR_FROM_LXC<br></div></div>
> +#define LXC_NAMESPACE_HREF "<a href="http://libvirt.org/schemas/domain/lxc/1.0" rel="noreferrer" target="_blank">http://libvirt.org/schemas/domain/lxc/1.0</a>"<div><div class="h5"><br>
><br>
>  VIR_LOG_INIT("lxc.lxc_domain");<br>
><br>
> @@ -41,6 +45,251 @@ static void *virLXCDomainObjPrivateAlloc(void)<br>
>      return priv;<br>
>  }<br>
><br>
> +<br>
> +static int open_ns(const char *nnsname_pid, const char *ns_proc_name)<br>
> +{<br>
> +    int fd = -1;<br>
> +    virDomainPtr dom = NULL;<br>
> +    virConnectPtr conn = NULL;<br>
> +    pid_t pid;<br>
> +    int nfdlist;<br>
> +    int *fdlist;<br>
> +    char *path = NULL;<br>
> +    char *eptr;<br>
> +    pid = strtol(nnsname_pid, &eptr, 10);<br>
> +    if (*eptr != '\0' || pid < 1) {<br>
> +        /* check if the domain is running, then set the namespaces<br>
> +        * to that container<br>
> +        */<br>
> +        size_t i = 0;<br>
> +        const char *ns[] = { "user", "ipc", "uts", "net", "pid", "mnt" };<br>
> +        conn = virConnectOpen("lxc:///");<br>
> +        if (!conn)<br>
> +          goto cleanup;<br>
> +        dom = virDomainLookupByName(conn, nnsname_pid);<br>
> +        if (!dom)<br>
> +           goto cleanup;<br>
> +        if ((nfdlist = virDomainLxcOpenNamespace(dom, &fdlist, 0)) < 0)<br>
> +           goto cleanup;<br>
> +         /* Internally above function calls  virProcessGetNamespaces<br>
> +          *  function which opens ns<br>
> +          * in the order { "user", "ipc", "uts", "net", "pid", "mnt" }<br>
> +          */<br>
> +        for (i = 0; i < ARRAY_CARDINALITY(ns); i++) {<br>
> +            if (STREQ(ns[i], ns_proc_name)) {<br>
> +              fd = fdlist[i];<br>
> +              break;<br>
> +           }<br>
> +        }<br>
> +        if (nfdlist > 0)<br>
<br>
</div></div></div></div><div><div class="h5">No, please no. If you read the virDomainLxcOpenNamespace() description,<br>
you'll notice that not only we must free the array, but also close all<br>
the file descriptors. You'll leak all of them but one. That's not<br>
acceptable.<br>
<span><br>
> +            VIR_FREE(fdlist);<br>
> +     } else {<br>
> +        if (virAsprintf(&path, "/proc/%d/ns/%s", pid, ns_proc_name) < 0)<br>
> +           goto cleanup;<br>
> +        fd = open(path, O_RDONLY);<br>
> +     }<br>
> +cleanup:<br>
> +     if (dom)<br>
> +       virDomainFree(dom);<br>
> +     VIR_FREE(path);<br>
> +     (fd < 0)? VIR_ERROR(<br>
> +         _("failed to open ns %s"), nnsname_pid):<br>
> +     VIR_DEBUG("OPENED NAMESPACE : fd %d\n", fd);<br>
> +     return fd;<br>
> +}<br>
<br>
<br>
</span>This is the part I'm having trouble with. IIUC, this code is run at the<br>
time of domain XML parsing. The other domain I'm referring to may still<br>
not be running. How am I going to set the same namespace then?<br>
<br>
I think this should be done at domain startup (and fail, if the<br>
referenced PID or domain does not exist). Having said that, when doing<br>
this in driver, that's starting a namespace, you have access to all<br>
internal variables and functions. Then you don't need to open a dummy<br>
connection just to look up the referenced domain. You can use an<br>
internal function which does not require virConnection object. Moreover,<br>
you will not need the Makefile change.<br>
<div><div><br>
> +<br>
> +<br>
> +/* Used only for containers */<br>
> +const struct ns_info ns_info[VIR_DOMAIN_NAMESPACE_LAST] = {<br>
> +    [VIR_DOMAIN_NAMESPACE_SHARENET] = {"net", CLONE_NEWNET},<br>
> +    [VIR_DOMAIN_NAMESPACE_SHAREIPC] = {"ipc", CLONE_NEWIPC},<br>
> +    [VIR_DOMAIN_NAMESPACE_SHAREUTS] = {"uts", CLONE_NEWUTS}<br>
> +};<br>
> +<br>
> +VIR_ENUM_DECL(virDomainNamespace)<br>
> +VIR_ENUM_IMPL(virDomainNamespace, VIR_DOMAIN_NAMESPACE_LAST,<br>
> +    N_("sharenet"),<br>
> +    N_("shareipc"),<br>
> +    N_("shareuts"))<br>
> +<br>
> +static void<br>
> +lxcDomainDefNamespaceFree(void *nsdata)<br>
> +{<br>
> +    int j;<br>
> +    lxcDomainDefPtr lxcDef = nsdata;<br>
> +    for (j = 0; j < VIR_DOMAIN_NAMESPACE_LAST; j++) {<br>
> +       if (lxcDef->ns_inherit_fd[j] > 0) {<br>
> +          VIR_FREE(lxcDef->ns_type);<br>
> +          VIR_FREE(lxcDef->ns_val);<br>
> +#if 0<br>
> +          if (VIR_CLOSE(lxcDef->ns_inherit_fd[j]) < 0)<br>
> +              virReportSystemError(errno, "%s", _("failed to close file"));<br>
> +#endif<br>
> +       }<br>
> +    }<br>
> +    VIR_FREE(nsdata);<br>
> +}<br>
> +<br>
> +static int<br>
> +lxcDomainDefNamespaceParse(xmlDocPtr xml ATTRIBUTE_UNUSED,<br>
> +                           xmlNodePtr root ATTRIBUTE_UNUSED,<br>
> +                           xmlXPathContextPtr ctxt,<br>
> +                           void **data)<br>
> +{<br>
> +    lxcDomainDefPtr lxcDef = NULL;<br>
> +    xmlNodePtr *nodes = NULL;<br>
> +    bool uses_lxc_ns = false;<br>
> +    xmlNodePtr node;<br>
> +    int feature;<br>
> +    int n;<br>
> +    char *tmp = NULL;<br>
> +    size_t i;<br>
> +    pid_t fd = -1;<br>
> +<br>
> +    if (xmlXPathRegisterNs(ctxt, BAD_CAST "lxc", BAD_CAST LXC_NAMESPACE_HREF) < 0) {<br>
> +        virReportError(VIR_ERR_INTERNAL_ERROR,<br>
> +                       _("Failed to register xml namespace '%s'"),<br>
> +                       LXC_NAMESPACE_HREF);<br>
> +        return -1;<br>
> +    }<br>
> +<br>
> +    if (VIR_ALLOC(lxcDef) < 0)<br>
> +        return -1;<br>
> +<br>
> +    /* Init ns_herit_fd for namespaces */<br>
> +    for (i = 0; i < VIR_DOMAIN_NAMESPACE_LAST; i++) {<br>
> +        lxcDef->ns_inherit_fd[i] = -1;<br>
> +        lxcDef->ns_type[i] = NULL;<br>
> +        lxcDef->ns_val[i] = NULL;<br>
> +    }<br>
> +<br>
> +    node = ctxt->node;<br>
> +    if ((n = virXPathNodeSet("./lxc:namespace/*", ctxt, &nodes)) < 0)<br>
> +        goto error;<br>
> +    uses_lxc_ns |= n > 0;<br>
> +<br>
> +    for (i = 0; i < n; i++) {<br>
> +        feature =<br>
> +            virDomainNamespaceTypeFromString((const char *) nodes[i]->name);<br>
> +        if (feature < 0) {<br>
> +            virReportError(VIR_ERR_CONFIG_UNSUPPORTED,<br>
> +                            _("unsupported Namespace feature: %s"),<br>
> +                            nodes[i]->name);<br>
> +            goto error;<br>
> +        }<br>
> +<br>
> +        ctxt->node = nodes[i];<br>
> +<br>
> +        switch ((virDomainNamespace) feature) {<br>
> +        case VIR_DOMAIN_NAMESPACE_SHARENET:<br>
> +        case VIR_DOMAIN_NAMESPACE_SHAREIPC:<br>
> +        case VIR_DOMAIN_NAMESPACE_SHAREUTS:<br>
> +            {<br>
> +                tmp = virXMLPropString(nodes[i], "type");<br>
> +                if (tmp == NULL) {<br>
> +                    virReportError(VIR_ERR_INTERNAL_ERROR,<br>
> +                                   "%s", _("No lxc environment type specified"));<br>
> +                    goto error;<br>
> +                }<br>
> +                /* save the tmp so that its needed while writing to xml */<br>
> +                lxcDef->ns_type[feature] = tmp;<br>
> +                tmp = virXMLPropString(nodes[i], "value");<br>
> +                if (tmp == NULL) {<br>
> +                    virReportError(VIR_ERR_INTERNAL_ERROR,<br>
> +                                   "%s", _("No lxc environment type specified"));<br>
> +                    goto error;<br>
> +                }<br>
> +                lxcDef->ns_val[feature] = tmp;<br>
> +                /*netns option is only for VIR_DOMAIN_NAMESPACE_SHARENET*/<br>
> +                if (STREQ("netns", lxcDef->ns_type[VIR_DOMAIN_NAMESPACE_SHARENET])) {<br>
<br>
</div></div>I don't think this is safe. ns_type[SHARENET] can be NULL at this point.<br>
STREQ_NULLABLE is more apropriate.<br>
<span><br>
> +                    char *path = NULL;<br>
> +                    if (virAsprintf(&path, "/var/run/netns/%s", tmp) < 0)<br>
> +                        goto error;<br>
> +                    fd = open(path, O_RDONLY);<br>
> +                    VIR_FREE(path);<br>
<br>
</span>What if open() fails?<br>
<div><div><br>
> +                } else {<br>
> +                    fd = open_ns(tmp, ns_info[feature].proc_name);<br>
> +                    if (fd < 0) {<br>
> +                        virReportError(VIR_ERR_XML_ERROR,<br>
> +                                       _("unable to open %s namespace for "<br>
> +                                         "namespace feature '%s'"), tmp,<br>
> +                                       nodes[i]->name);<br>
> +                        goto error;<br>
> +                    }<br>
> +                }<br>
> +                lxcDef->ns_inherit_fd[feature] = fd;<br>
> +            }<br>
> +            break;<br>
> +        case VIR_DOMAIN_NAMESPACE_LAST:<br>
> +            break;<br>
> +        }<br>
> +    }<br>
> +    VIR_FREE(nodes);<br>
> +    ctxt->node = node;<br>
> +    if (uses_lxc_ns)<br>
> +        *data = lxcDef;<br>
> +    else<br>
> +        VIR_FREE(lxcDef);<br>
> +    return 0;<br>
> + error:<br>
> +    VIR_FREE(nodes);<br>
> +    lxcDomainDefNamespaceFree(lxcDef);<br>
> +    return -1;<br>
> +}<br>
> +<br>
> +<br>
> +static int<br>
> +lxcDomainDefNamespaceFormatXML(virBufferPtr buf,<br>
> +                               void *nsdata)<br>
> +{<br>
> +    lxcDomainDefPtr lxcDef = nsdata;<br>
> +    size_t j;<br>
> +<br>
> +    if (!lxcDef)<br>
> +       return 0;<br>
> +<br>
> +    virBufferAddLit(buf, "<lxc:namespace>\n");<br>
> +    virBufferAdjustIndent(buf, 2);<br>
> +<br>
> +    for (j = 0; j < VIR_DOMAIN_NAMESPACE_LAST; j++) {<br>
> +        switch ((virDomainNamespace) j) {<br>
> +        case VIR_DOMAIN_NAMESPACE_SHAREIPC:<br>
> +        case VIR_DOMAIN_NAMESPACE_SHAREUTS:<br>
> +        case VIR_DOMAIN_NAMESPACE_SHARENET:<br>
> +            {<br>
> +                if (lxcDef->ns_inherit_fd[j] > 0) {<br>
> +                    virBufferAsprintf(buf, "<%s type='%s' value='%s'/>\n",<br>
> +                                      virDomainNamespaceTypeToString(j),<br>
> +                                      lxcDef->ns_type[j],<br>
> +                                      lxcDef->ns_val[j]);<br>
> +                }<br>
> +            }<br>
> +            break;<br>
> +        case VIR_DOMAIN_NAMESPACE_LAST:<br>
> +            break;<br>
> +        }<br>
> +    }<br>
> +<br>
> +    virBufferAdjustIndent(buf, -2);<br>
> +    virBufferAddLit(buf, "</lxc:namespace>\n");<br>
> +    return 0;<br>
> +}<br>
> +<br>
> +static const char *<br>
> +lxcDomainDefNamespaceHref(void)<br>
> +{<br>
> +    return "xmlns:lxc='" LXC_NAMESPACE_HREF "'";<br>
> +}<br>
> +<br>
> +<br>
> +virDomainXMLNamespace virLXCDriverDomainXMLNamespace = {<br>
> +    .parse = lxcDomainDefNamespaceParse,<br>
> +    .free = lxcDomainDefNamespaceFree,<br>
> +    .format = lxcDomainDefNamespaceFormatXML,<br>
> +    .href = lxcDomainDefNamespaceHref,<br>
> +};<br>
> +<br>
> +<br>
>  static void virLXCDomainObjPrivateFree(void *data)<br>
>  {<br>
>      virLXCDomainObjPrivatePtr priv = data;<br>
> @@ -73,7 +322,6 @@ static int virLXCDomainObjPrivateXMLParse(xmlXPathContextPtr ctxt, void *data)<br>
>      } else {<br>
>          priv->initpid = thepid;<br>
>      }<br>
> -<br>
>      return 0;<br>
>  }<br>
<br>
</div></div>Yet again, why is this change needed?<br>
<span><br>
><br>
> diff --git a/src/lxc/lxc_domain.h b/src/lxc/lxc_domain.h<br>
> index 751aece..25df999 100644<br>
> --- a/src/lxc/lxc_domain.h<br>
> +++ b/src/lxc/lxc_domain.h<br>
> @@ -41,6 +41,7 @@ struct _virLXCDomainObjPrivate {<br>
>      virCgroupPtr cgroup;<br>
>  };<br>
><br>
> +extern virDomainXMLNamespace virLXCDriverDomainXMLNamespace;<br>
>  extern virDomainXMLPrivateDataCallbacks virLXCDriverPrivateDataCallbacks;<br>
>  extern virDomainDefParserConfig virLXCDriverDomainDefParserConfig;<br>
><br>
><br>
<br>
</span>Frankly, I don't have good feeling about this. But maybe I'm missing<br>
some rationale behind. What's the usecase? You want two containers to<br>
share the same network namespace, or?<br>
I view two or more containers in the same namespace as a security flaw,<br>
not feature. With this mind set maybe I'm not the right person to review<br>
the patches. But hey, I'm open for persuasion :)<br>
<span><font color="#888888"><br>
Michal<br>
</font></span></div></div></blockquote></div><br></div></div>
</blockquote></div><br></div>