[Libguestfs] [PATCH nbdkit v2] curl: Implement header and cookie scripts.
Richard W.M. Jones
rjones at redhat.com
Wed Jul 15 20:53:48 UTC 2020
This rather complex feature solves a problem for certain web services
that require a cookie or token for access, especially one which must
be periodically renewed.
For motivation for this feature see the included documentation, and
item (1)(b) here:
https://www.redhat.com/archives/libguestfs/2020-July/msg00069.html
---
plugins/curl/nbdkit-curl-plugin.pod | 142 ++++++++++++
plugins/curl/Makefile.am | 2 +
tests/Makefile.am | 47 ++++
plugins/curl/curldefs.h | 76 +++++++
plugins/curl/curl.c | 116 +++++++---
plugins/curl/scripts.c | 330 ++++++++++++++++++++++++++++
tests/test-curl-cookie-script.c | 143 ++++++++++++
tests/test-curl-header-script.c | 165 ++++++++++++++
.gitignore | 2 +
9 files changed, 991 insertions(+), 32 deletions(-)
diff --git a/plugins/curl/nbdkit-curl-plugin.pod b/plugins/curl/nbdkit-curl-plugin.pod
index 22c07f39..52875988 100644
--- a/plugins/curl/nbdkit-curl-plugin.pod
+++ b/plugins/curl/nbdkit-curl-plugin.pod
@@ -81,6 +81,14 @@ command line is not secure on shared machines. Use the alternate
C<+FILENAME> syntax to pass it in a file, C<-> to read the cookie
interactively, or C<-FD> to read it from a file descriptor.
+=item B<cookie-script=>SCRIPT
+
+=item B<cookie-script-renew=>SECS
+
+Run C<SCRIPT> (a command or shell script fragment) to generate the
+HTTP/HTTPS cookies. C<cookie-script> cannot be used with C<cookie>.
+See L</HEADER AND COOKIE SCRIPTS> below.
+
=item B<header=>HEADER
(nbdkit E<ge> 1.22)
@@ -106,6 +114,14 @@ requests, even when following a redirect, which can cause headers
(eg. containing sensitive authorization information) to be sent to
hosts other than the one originally requested.
+=item B<header-script=>SCRIPT
+
+=item B<header-script-renew=>SECS
+
+Run C<SCRIPT> (a command or shell script fragment) to generate the
+HTTP/HTTPS headers. C<header-script> cannot be used with C<header>.
+See L</HEADER AND COOKIE SCRIPTS> below.
+
=item B<password=>PASSWORD
Set the password to use when connecting to the remote server.
@@ -226,6 +242,132 @@ user-agent header.
=back
+=head1 HEADER AND COOKIE SCRIPTS
+
+While the C<header> and C<cookie> parameters can be used to specify
+static headers and cookies which are used in every HTTP/HTTPS request,
+the alternate C<header-script> and C<cookie-script> parameters can be
+used to run an external script or program to generate headers and/or
+cookies. This is particularly useful to access services which require
+an authorization token. In addition the C<header-script-renew> and
+C<cookie-script-renew> parameters allow you to renew the authorization
+token by rerunning the script periodically.
+
+C<header-script> is incompatible with C<header>, and C<cookie-script>
+is incompatible with C<cookie>.
+
+=head2 Header script
+
+The header script should print zero or more HTTP headers, each line of
+output in the same format as the C<header> parameter. The headers
+printed by the script are passed to L<CURLOPT_HTTPHEADER(3)>.
+
+In the following example, an imaginary web service requires
+authentication using a token fetched from a separate login server.
+The token expires after 60 seconds, so we also tell the plugin that it
+must renew the token (by re-running the script) if more than 45
+seconds have elapsed since the last request:
+
+ nbdkit curl https://service.example.com/disk.img \
+ header-script='
+ echo -n "Authorization: Bearer "
+ curl -s -X POST https://auth.example.com/login |
+ jq -r .token
+ ' \
+ header-script-renew=50
+
+=head2 Cookie script
+
+The cookie script should print a single line in the same format as the
+C<cookie> parameter. This is passed to L<CURLOPT_COOKIE(3)>.
+
+=head2 Header and cookie script shell variables
+
+Within the C<header-script> and C<cookie-script> the following shell
+variables are available:
+
+=over 4
+
+=item C<$iteration>
+
+The number of times that the script has been called. The first time
+the script is called this contains C<0>.
+
+=item C<$url>
+
+The URL as passed to the plugin.
+
+=back
+
+=head2 Example: VMware ESXi cookies
+
+VMware ESXi’s web server can expose both VMDK and raw format disk
+images, but requires you to log in using HTTP Basic Authentication.
+While you can use the C<user> and C<password> parameters to send HTTP
+Basic Authentication headers in every request, tests have shown that
+it is faster to accept the cookie which the server returns and send
+that instead. (It is not clear why it is faster, but one theory is
+that VMware has to do a more expensive username and password check
+each time.)
+
+The web server can be accessed as below. Since the cookie expires
+after a certain period of time, we use C<cookie-script-renew>, and
+because the server uses a self-signed certificate we must use
+I<--insecure> and C<sslverify=false>.
+
+ SERVER=esx.example.com
+ DCPATH=data
+ DS=datastore1
+ GUEST=guest-name
+ URL="https://$SERVER/folder/$GUEST/$GUEST-flat.vmdk?dcPath=$DCPATH&dsName=$DS"
+
+ nbdkit curl "$URL" \
+ cookie-script='
+ curl --head -s --insecure -u root:password "$url" |
+ sed -ne '{ s/^Set-Cookie: \([^;]*\);.*/\1/ip }'
+ ' \
+ cookie-script-renew=500 \
+ sslverify=false
+
+=head2 Example: Docker Hub authorization tokens
+
+Accessing objects like container layers from Docker Hub requires that
+you first fetch an authorization token, even for anonymous access.
+These tokens expire after about 5 minutes (300 seconds) so must be
+periodically renewed.
+
+You will need this authorization script (F</tmp/auth.sh>):
+
+ #!/bin/sh -
+ IMAGE=library/fedora
+ curl -s "https://auth.docker.io/token?service=registry.docker.io&scope=repository:$IMAGE:pull" |
+ jq -r .token
+
+You will also need this script to get the blobSum of the layer
+(F</tmp/blobsum.sh>):
+
+ #!/bin/sh -
+ TOKEN=`/tmp/auth.sh`
+ IMAGE=library/fedora
+ curl -s -X GET -H "Authorization: Bearer $TOKEN" \
+ "https://registry-1.docker.io/v2/$IMAGE/manifests/latest" |
+ jq -r '.fsLayers[0].blobSum'
+
+Both scripts must be executable, and both can be run on their own to
+check they are working. To run nbdkit:
+
+ IMAGE=library/fedora
+ BLOBSUM=`/tmp/blobsum.sh`
+ URL="https://registry-1.docker.io/v2/$IMAGE/blobs/$BLOBSUM"
+
+ nbdkit curl "$URL" \
+ header-script=' echo -n "Authorization: Bearer "; /tmp/auth.sh ' \
+ header-script-renew=200 \
+ --filter=gzip
+
+Note that this exposes a tar file over NBD. See also
+L<nbdkit-tar-filter(1)>.
+
=head1 DEBUG FLAG
=over 4
diff --git a/plugins/curl/Makefile.am b/plugins/curl/Makefile.am
index ddf1a215..0dd78199 100644
--- a/plugins/curl/Makefile.am
+++ b/plugins/curl/Makefile.am
@@ -38,6 +38,8 @@ if HAVE_CURL
plugin_LTLIBRARIES = nbdkit-curl-plugin.la
nbdkit_curl_plugin_la_SOURCES = \
+ curldefs.h \
+ scripts.c \
curl.c \
$(top_srcdir)/include/nbdkit-plugin.h \
$(NULL)
diff --git a/tests/Makefile.am b/tests/Makefile.am
index b830d80e..2641910b 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -502,6 +502,10 @@ if HAVE_CURL
TESTS += test-curl-file.sh
EXTRA_DIST += test-curl-file.sh
LIBGUESTFS_TESTS += test-curl
+LIBNBD_TESTS += \
+ test-curl-header-script \
+ test-curl-cookie-script \
+ $(NULL)
test_curl_SOURCES = \
test-curl.c \
@@ -524,6 +528,49 @@ test_curl_LDADD = \
libtest.la \
$(LIBGUESTFS_LIBS) \
$(NULL)
+
+test_curl_header_script_SOURCES = \
+ test-curl-header-script.c \
+ web-server.c \
+ web-server.h \
+ $(NULL)
+test_curl_header_script_CPPFLAGS = \
+ -I$(top_srcdir)/common/utils \
+ $(NULL)
+test_curl_header_script_CFLAGS = \
+ $(WARNINGS_CFLAGS) \
+ $(LIBNBD_CFLAGS) \
+ $(PTHREAD_CFLAGS) \
+ $(NULL)
+test_curl_header_script_LDFLAGS = \
+ $(top_builddir)/common/utils/libutils.la \
+ $(PTHREAD_LIBS) \
+ $(NULL)
+test_curl_header_script_LDADD = \
+ $(LIBNBD_LIBS) \
+ $(NULL)
+
+test_curl_cookie_script_SOURCES = \
+ test-curl-header-script.c \
+ web-server.c \
+ web-server.h \
+ $(NULL)
+test_curl_cookie_script_CPPFLAGS = \
+ -I$(top_srcdir)/common/utils \
+ $(NULL)
+test_curl_cookie_script_CFLAGS = \
+ $(WARNINGS_CFLAGS) \
+ $(LIBNBD_CFLAGS) \
+ $(PTHREAD_CFLAGS) \
+ $(NULL)
+test_curl_cookie_script_LDFLAGS = \
+ $(top_builddir)/common/utils/libutils.la \
+ $(PTHREAD_LIBS) \
+ $(NULL)
+test_curl_cookie_script_LDADD = \
+ $(LIBNBD_LIBS) \
+ $(NULL)
+
endif HAVE_CURL
endif HAVE_MKE2FS_WITH_D
diff --git a/plugins/curl/curldefs.h b/plugins/curl/curldefs.h
new file mode 100644
index 00000000..5ec03231
--- /dev/null
+++ b/plugins/curl/curldefs.h
@@ -0,0 +1,76 @@
+/* nbdkit
+ * Copyright (C) 2014-2020 Red Hat Inc.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are
+ * met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ *
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ *
+ * * Neither the name of Red Hat nor the names of its contributors may be
+ * used to endorse or promote products derived from this software without
+ * specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+ * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+ * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+ * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+ * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+ * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#ifndef NBDKIT_CURLDEFS_H
+#define NBDKIT_CURLDEFS_H
+
+extern const char *url;
+
+extern const char *cainfo;
+extern const char *capath;
+extern char *cookie;
+extern const char *cookie_script;
+extern unsigned cookie_script_renew;
+extern struct curl_slist *headers;
+extern const char *header_script;
+extern unsigned header_script_renew;
+extern char *password;
+extern long protocols;
+extern const char *proxy;
+extern char *proxy_password;
+extern const char *proxy_user;
+extern bool sslverify;
+extern bool tcp_keepalive;
+extern bool tcp_nodelay;
+extern uint32_t timeout;
+extern const char *unix_socket_path;
+extern const char *user;
+extern const char *user_agent;
+
+/* The per-connection handle. */
+struct curl_handle {
+ CURL *c;
+ bool accept_range;
+ int64_t exportsize;
+ char errbuf[CURL_ERROR_SIZE];
+ char *write_buf;
+ uint32_t write_count;
+ const char *read_buf;
+ uint32_t read_count;
+ struct curl_slist *headers_copy;
+};
+
+/* scripts.c */
+extern int do_scripts (struct curl_handle *h);
+extern void scripts_unload (void);
+
+#endif /* NBDKIT_CURLDEFS_H */
diff --git a/plugins/curl/curl.c b/plugins/curl/curl.c
index 50eef1a8..8731a506 100644
--- a/plugins/curl/curl.c
+++ b/plugins/curl/curl.c
@@ -48,9 +48,11 @@
#include <nbdkit-plugin.h>
-#include "cleanup.h"
#include "ascii-ctype.h"
#include "ascii-string.h"
+#include "cleanup.h"
+
+#include "curldefs.h"
/* Macro CURL_AT_LEAST_VERSION was added in 2015 (Curl 7.43) so if the
* macro isn't present then Curl is very old.
@@ -61,24 +63,29 @@
#endif
#endif
-static const char *url = NULL; /* required */
+/* Plugin configuration. */
+const char *url = NULL; /* required */
-static const char *cainfo = NULL;
-static const char *capath = NULL;
-static char *cookie = NULL;
-static struct curl_slist *headers = NULL;
-static char *password = NULL;
-static long protocols = CURLPROTO_ALL;
-static const char *proxy = NULL;
-static char *proxy_password = NULL;
-static const char *proxy_user = NULL;
-static bool sslverify = true;
-static bool tcp_keepalive = false;
-static bool tcp_nodelay = true;
-static uint32_t timeout = 0;
-static const char *unix_socket_path = NULL;
-static const char *user = NULL;
-static const char *user_agent = NULL;
+const char *cainfo = NULL;
+const char *capath = NULL;
+char *cookie = NULL;
+const char *cookie_script = NULL;
+unsigned cookie_script_renew = 0;
+struct curl_slist *headers = NULL;
+const char *header_script = NULL;
+unsigned header_script_renew = 0;
+char *password = NULL;
+long protocols = CURLPROTO_ALL;
+const char *proxy = NULL;
+char *proxy_password = NULL;
+const char *proxy_user = NULL;
+bool sslverify = true;
+bool tcp_keepalive = false;
+bool tcp_nodelay = true;
+uint32_t timeout = 0;
+const char *unix_socket_path = NULL;
+const char *user = NULL;
+const char *user_agent = NULL;
/* Use '-D curl.verbose=1' to set. */
int curl_debug_verbose = 0;
@@ -98,11 +105,12 @@ curl_load (void)
static void
curl_unload (void)
{
- free (password);
- free (proxy_password);
free (cookie);
if (headers)
curl_slist_free_all (headers);
+ free (password);
+ free (proxy_password);
+ scripts_unload ();
curl_global_cleanup ();
}
@@ -202,6 +210,16 @@ curl_config (const char *key, const char *value)
return -1;
}
+ else if (strcmp (key, "cookie-script") == 0) {
+ cookie_script = value;
+ }
+
+ else if (strcmp (key, "cookie-script-renew") == 0) {
+ if (nbdkit_parse_unsigned ("cookie-script-renew", value,
+ &cookie_script_renew) == -1)
+ return -1;
+ }
+
else if (strcmp (key, "header") == 0) {
headers = curl_slist_append (headers, value);
if (headers == NULL) {
@@ -210,6 +228,16 @@ curl_config (const char *key, const char *value)
}
}
+ else if (strcmp (key, "header-script") == 0) {
+ header_script = value;
+ }
+
+ else if (strcmp (key, "header-script-renew") == 0) {
+ if (nbdkit_parse_unsigned ("header-script-renew", value,
+ &header_script_renew) == -1)
+ return -1;
+ }
+
else if (strcmp (key, "password") == 0) {
free (password);
if (nbdkit_read_password (value, &password) == -1)
@@ -300,6 +328,26 @@ curl_config_complete (void)
return -1;
}
+ if (headers && header_script) {
+ nbdkit_error ("header and header-script cannot be used at the same time");
+ return -1;
+ }
+
+ if (!header_script && header_script_renew) {
+ nbdkit_error ("header-script-renew cannot be used without header-script");
+ return -1;
+ }
+
+ if (cookie && cookie_script) {
+ nbdkit_error ("cookie and cookie-script cannot be used at the same time");
+ return -1;
+ }
+
+ if (!cookie_script && cookie_script_renew) {
+ nbdkit_error ("cookie-script-renew cannot be used without cookie-script");
+ return -1;
+ }
+
return 0;
}
@@ -307,7 +355,11 @@ curl_config_complete (void)
"cainfo=<CAINFO> Path to Certificate Authority file.\n" \
"capath=<CAPATH> Path to directory with CA certificates.\n" \
"cookie=<COOKIE> Set HTTP/HTTPS cookies.\n" \
+ "cookie-script=<SCRIPT> Script to set HTTP/HTTPS cookies.\n" \
+ "cookie-script-renew=<SECS> Time to renew HTTP/HTTPS cookies.\n" \
"header=<HEADER> Set HTTP/HTTPS header.\n" \
+ "header-script=<SCRIPT> Script to set HTTP/HTTPS headers.\n" \
+ "header-script-renew=<SECS> Time to renew HTTP/HTTPS headers.\n" \
"password=<PASSWORD> The password for the user account.\n" \
"protocols=PROTO,PROTO,.. Limit protocols allowed.\n" \
"proxy=<PROXY> Set proxy URL.\n" \
@@ -322,18 +374,6 @@ curl_config_complete (void)
"user=<USER> The user to log in as.\n" \
"user-agent=<USER-AGENT> Send user-agent header for HTTP/HTTPS."
-/* The per-connection handle. */
-struct curl_handle {
- CURL *c;
- bool accept_range;
- int64_t exportsize;
- char errbuf[CURL_ERROR_SIZE];
- char *write_buf;
- uint32_t write_count;
- const char *read_buf;
- uint32_t read_count;
-};
-
/* Translate CURLcode to nbdkit_error. */
#define display_curl_error(h, r, fs, ...) \
do { \
@@ -450,7 +490,11 @@ curl_open (int readonly)
/* Get the file size and also whether the remote HTTP server
* supports byte ranges.
+ *
+ * We must run the scripts if necessary and set headers in the
+ * handle.
*/
+ if (do_scripts (h) == -1) goto err;
h->accept_range = false;
curl_easy_setopt (h->c, CURLOPT_NOBODY, 1); /* No Body, not nobody! */
curl_easy_setopt (h->c, CURLOPT_HEADERFUNCTION, header_cb);
@@ -608,6 +652,8 @@ curl_close (void *handle)
struct curl_handle *h = handle;
curl_easy_cleanup (h->c);
+ if (h->headers_copy)
+ curl_slist_free_all (h->headers_copy);
free (h);
}
@@ -638,6 +684,9 @@ curl_pread (void *handle, void *buf, uint32_t count, uint64_t offset)
CURLcode r;
char range[128];
+ /* Run the scripts if necessary and set headers in the handle. */
+ if (do_scripts (h) == -1) return -1;
+
/* Tell the write_cb where we want the data to be written. write_cb
* will update this if the data comes in multiple sections.
*/
@@ -699,6 +748,9 @@ curl_pwrite (void *handle, const void *buf, uint32_t count, uint64_t offset)
CURLcode r;
char range[128];
+ /* Run the scripts if necessary and set headers in the handle. */
+ if (do_scripts (h) == -1) return -1;
+
/* Tell the read_cb where we want the data to be read from. read_cb
* will update this if the data comes in multiple sections.
*/
diff --git a/plugins/curl/scripts.c b/plugins/curl/scripts.c
new file mode 100644
index 00000000..5d961391
--- /dev/null
+++ b/plugins/curl/scripts.c
@@ -0,0 +1,330 @@
+/* nbdkit
+ * Copyright (C) 2014-2020 Red Hat Inc.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are
+ * met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ *
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ *
+ * * Neither the name of Red Hat nor the names of its contributors may be
+ * used to endorse or promote products derived from this software without
+ * specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+ * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+ * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+ * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+ * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+ * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+/* Header and cookie scripts. */
+
+#include <config.h>
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <stdbool.h>
+#include <string.h>
+#include <unistd.h>
+#include <time.h>
+#include <assert.h>
+#include <pthread.h>
+
+#include <curl/curl.h>
+
+#include <nbdkit-plugin.h>
+
+#include "ascii-ctype.h"
+#include "cleanup.h"
+#include "utils.h"
+
+#include "curldefs.h"
+
+/* This lock protects internal state in this file. */
+static pthread_mutex_t lock = PTHREAD_MUTEX_INITIALIZER;
+
+/* Last time header-script or cookie-script was run. */
+static time_t header_last = 0;
+static time_t cookie_last = 0;
+static bool header_script_has_run = false;
+static bool cookie_script_has_run = false;
+static unsigned header_iteration = 0;
+static unsigned cookie_iteration = 0;
+
+/* Last set of headers and cookies generated by the scripts. */
+static struct curl_slist *headers_from_script = NULL;
+static char *cookies_from_script = NULL;
+
+void
+scripts_unload (void)
+{
+ curl_slist_free_all (headers_from_script);
+ free (cookies_from_script);
+}
+
+static int run_header_script (struct curl_handle *);
+static int run_cookie_script (struct curl_handle *);
+
+/* This is called from any thread just before we make a curl request.
+ *
+ * Because the thread model is NBDKIT_THREAD_MODEL_SERIALIZE_REQUESTS
+ * we can be assured of exclusive access to curl_handle here.
+ */
+int
+do_scripts (struct curl_handle *h)
+{
+ time_t now;
+ struct curl_slist *p;
+
+ /* Return quickly without acquiring the lock if this feature is not
+ * being used.
+ */
+ if (!header_script && !cookie_script)
+ return 0;
+
+ ACQUIRE_LOCK_FOR_CURRENT_SCOPE (&lock);
+
+ /* Run or re-run header-script if we need to. */
+ if (header_script) {
+ time (&now);
+ if (!header_script_has_run ||
+ (header_script_renew > 0 && now - header_last >= header_script_renew)) {
+ if (run_header_script (h) == -1)
+ return -1;
+ header_last = now;
+ header_script_has_run = true;
+ }
+ }
+
+ /* Run or re-run cookie-script if we need to. */
+ if (cookie_script) {
+ time (&now);
+ if (!cookie_script_has_run ||
+ (cookie_script_renew > 0 && now - cookie_last >= cookie_script_renew)) {
+ if (run_cookie_script (h) == -1)
+ return -1;
+ cookie_last = now;
+ cookie_script_has_run = true;
+ }
+ }
+
+ /* Set headers and cookies in the handle.
+ *
+ * When calling CURLOPT_HTTPHEADER we have to keep the list around
+ * because unfortunately curl doesn't take a copy. Since we don't
+ * know which other threads might be using it, we must make a copy
+ * of the global list (headers_from_script) per handle
+ * (h->headers_copy). For CURLOPT_COOKIE, curl internally takes a
+ * copy so we don't need to do this.
+ */
+ if (h->headers_copy) {
+ curl_easy_setopt (h->c, CURLOPT_HTTPHEADER, NULL);
+ curl_slist_free_all (h->headers_copy);
+ h->headers_copy = NULL;
+ }
+ for (p = headers_from_script; p != NULL; p = p->next) {
+ h->headers_copy = curl_slist_append (h->headers_copy, p->data);
+ if (h->headers_copy == NULL) {
+ nbdkit_error ("curl_slist_append: %m");
+ return -1;
+ }
+ }
+ curl_easy_setopt (h->c, CURLOPT_HTTPHEADER, h->headers_copy);
+
+ curl_easy_setopt (h->c, CURLOPT_COOKIE, cookies_from_script);
+
+ return 0;
+}
+
+/* This is called with the lock held when we must run or re-run the
+ * header-script.
+ */
+static int
+run_header_script (struct curl_handle *h)
+{
+ int fd;
+ char tmpfile[] = "/tmp/errorsXXXXXX";
+ FILE *fp;
+ CLEANUP_FREE char *cmd = NULL, *line = NULL;
+ size_t len = 0, linelen = 0, nr_headers = 0;
+
+ assert (header_script != NULL); /* checked by caller */
+
+ /* Reset the list of headers. */
+ curl_slist_free_all (headers_from_script);
+ headers_from_script = NULL;
+
+ /* Create a temporary file for the errors so we can redirect them
+ * into nbdkit_error.
+ */
+ fd = mkstemp (tmpfile);
+ if (fd == -1) {
+ nbdkit_error ("mkstemp");
+ return -1;
+ }
+ close (fd);
+
+ /* Generate the full script with the local $url variable. */
+ fp = open_memstream (&cmd, &len);
+ if (fp == NULL) {
+ nbdkit_error ("open_memstream: %m");
+ return -1;
+ }
+ fprintf (fp, "exec </dev/null\n"); /* Avoid stdin leaking (nbdkit -s). */
+ fprintf (fp, "exec 2>%s\n", tmpfile); /* Catch errors to a temporary file. */
+ fprintf (fp, "url="); /* Set the shell variables. */
+ shell_quote (url, fp);
+ putc ('\n', fp);
+ fprintf (fp, "iteration=%u\n", header_iteration++);
+ putc ('\n', fp);
+ fprintf (fp, "%s", header_script); /* The script or command. */
+ if (fclose (fp) == EOF) {
+ nbdkit_error ("memstream failed");
+ return -1;
+ }
+
+ /* Run the script and read the headers. */
+ nbdkit_debug ("curl: running header-script");
+ fp = popen (cmd, "r");
+ if (fp == NULL) {
+ nbdkit_error ("popen: %m");
+ return -1;
+ }
+ while ((len = getline (&line, &linelen, fp)) != -1) {
+ /* Remove trailing \n and whitespace. */
+ while (len > 0 && ascii_isspace (line[len-1]))
+ line[--len] = '\0';
+ if (len == 0)
+ continue;
+
+ headers_from_script = curl_slist_append (headers_from_script, line);
+ if (headers_from_script == NULL) {
+ nbdkit_error ("curl_slist_append: %m");
+ pclose (fp);
+ return -1;
+ }
+ nr_headers++;
+ }
+
+ /* If the command failed, this should return EOF and the error
+ * message should be in the temporary file (but we only read the
+ * first line).
+ */
+ if (pclose (fp) == EOF) {
+ fp = fopen (tmpfile, "r");
+ if ((len = getline (&line, &linelen, fp)) >= 0) {
+ if (len > 0 && line[len-1] == '\n')
+ line[len-1] = '\0';
+ nbdkit_error ("header-script failed: %s", line);
+ }
+ else
+ nbdkit_error ("header-script failed");
+ return -1;
+ }
+
+ nbdkit_debug ("header-script returned %zu header(s)", nr_headers);
+ return 0;
+}
+
+/* This is called with the lock held when we must run or re-run the
+ * cookie-script.
+ */
+static int
+run_cookie_script (struct curl_handle *h)
+{
+ int fd;
+ char tmpfile[] = "/tmp/errorsXXXXXX";
+ FILE *fp;
+ CLEANUP_FREE char *cmd = NULL, *line = NULL;
+ size_t len = 0, linelen = 0;
+
+ assert (cookie_script != NULL); /* checked by caller */
+
+ /* Reset the cookies. */
+ free (cookies_from_script);
+ cookies_from_script = NULL;
+
+ /* Create a temporary file for the errors so we can redirect them
+ * into nbdkit_error.
+ */
+ fd = mkstemp (tmpfile);
+ if (fd == -1) {
+ nbdkit_error ("mkstemp");
+ return -1;
+ }
+ close (fd);
+
+ /* Generate the full script with the local $url variable. */
+ fp = open_memstream (&cmd, &len);
+ if (fp == NULL) {
+ nbdkit_error ("open_memstream: %m");
+ return -1;
+ }
+ fprintf (fp, "exec </dev/null\n"); /* Avoid stdin leaking (nbdkit -s). */
+ fprintf (fp, "exec 2>%s\n", tmpfile); /* Catch errors to a temporary file. */
+ fprintf (fp, "url="); /* Set the shell variable. */
+ shell_quote (url, fp);
+ putc ('\n', fp);
+ fprintf (fp, "iteration=%u\n", cookie_iteration++);
+ putc ('\n', fp);
+ fprintf (fp, "%s", cookie_script); /* The script or command. */
+ if (fclose (fp) == EOF) {
+ nbdkit_error ("memstream failed");
+ return -1;
+ }
+
+ /* Run the script and read the cookies. */
+ nbdkit_debug ("curl: running cookie-script");
+ fp = popen (cmd, "r");
+ if (fp == NULL) {
+ nbdkit_error ("popen: %m");
+ return -1;
+ }
+ len = getline (&line, &linelen, fp);
+ if (len > 0) {
+ /* Remove trailing \n and whitespace. */
+ while (len > 0 && ascii_isspace (line[len-1]))
+ line[--len] = '\0';
+ if (len > 0) {
+ cookies_from_script = strdup (line);
+ if (cookies_from_script == NULL) {
+ nbdkit_error ("strdup");
+ pclose (fp);
+ return -1;
+ }
+ }
+ }
+
+ /* If the command failed, this should return EOF and the error
+ * message should be in the temporary file (but we only read the
+ * first line).
+ */
+ if (pclose (fp) == EOF) {
+ fp = fopen (tmpfile, "r");
+ if ((len = getline (&line, &linelen, fp)) >= 0) {
+ if (len > 0 && line[len-1] == '\n')
+ line[len-1] = '\0';
+ nbdkit_error ("cookie-script failed: %s", line);
+ }
+ else
+ nbdkit_error ("cookie-script failed");
+ return -1;
+ }
+
+ nbdkit_debug ("cookie-script returned %scookies",
+ cookies_from_script ? "" : "no ");
+ return 0;
+}
diff --git a/tests/test-curl-cookie-script.c b/tests/test-curl-cookie-script.c
new file mode 100644
index 00000000..481207b5
--- /dev/null
+++ b/tests/test-curl-cookie-script.c
@@ -0,0 +1,143 @@
+/* nbdkit
+ * Copyright (C) 2013-2020 Red Hat Inc.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are
+ * met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ *
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ *
+ * * Neither the name of Red Hat nor the names of its contributors may be
+ * used to endorse or promote products derived from this software without
+ * specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+ * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+ * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+ * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+ * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+ * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#include <config.h>
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <stdbool.h>
+#include <stdint.h>
+#include <inttypes.h>
+#include <string.h>
+#include <unistd.h>
+
+#include <libnbd.h>
+
+#include "cleanup.h"
+#include "web-server.h"
+
+#include "test.h"
+
+static int iteration;
+
+#define SCRIPT \
+ "echo iteration=$iteration"
+
+static void
+check_request (const char *request)
+{
+ char expected[64];
+
+ /* Check the Cookie header. */
+ snprintf (expected, sizeof expected,
+ "\r\nCookie: iteration=%u\r\n", iteration);
+ if (strcasestr (request, expected) == NULL) {
+ fprintf (stderr, "%s: no/incorrect iteration cookie in request\n",
+ program_name);
+ exit (EXIT_FAILURE);
+ }
+}
+
+static char buf[512];
+
+int
+main (int argc, char *argv[])
+{
+ const char *sockpath;
+ struct nbd_handle *nbd;
+ CLEANUP_FREE char *usp_param = NULL;
+
+#ifndef HAVE_CURLOPT_UNIX_SOCKET_PATH
+ fprintf (stderr, "%s: curl does not support CURLOPT_UNIX_SOCKET_PATH\n",
+ program_name);
+ exit (77);
+#endif
+
+ sockpath = web_server ("disk", check_request);
+ if (sockpath == NULL) {
+ fprintf (stderr, "%s: could not start web server thread\n", program_name);
+ exit (EXIT_FAILURE);
+ }
+
+ nbd = nbd_create ();
+ if (nbd == NULL) {
+ fprintf (stderr, "%s\n", nbd_get_error ());
+ exit (EXIT_FAILURE);
+ }
+
+ /* We expect that connecting will cause a HEAD request (to find the
+ * size). $iteration will be 0.
+ */
+ iteration = 0;
+
+ /* Start nbdkit. */
+ if (asprintf (&usp_param, "unix-socket-path=%s", sockpath) == -1) {
+ perror ("asprintf");
+ exit (EXIT_FAILURE);
+ }
+ char *args[] = {
+ "nbdkit", "-s", "--exit-with-parent", "-v",
+ "curl",
+ "-D", "curl.verbose=1",
+ "http://localhost/disk",
+ "cookie-script=" SCRIPT,
+ "cookie-script-renew=1",
+ usp_param, /* unix-socket-path=... */
+ NULL
+ };
+ if (nbd_connect_command (nbd, args) == -1) {
+ fprintf (stderr, "%s\n", nbd_get_error ());
+ exit (EXIT_FAILURE);
+ }
+
+ /* Sleep the script will be called again. $iteration will be 1. */
+ sleep (2);
+ iteration = 1;
+
+ /* Make a request. */
+ if (nbd_pread (nbd, buf, sizeof buf, 0, 0) == -1) {
+ fprintf (stderr, "%s\n", nbd_get_error ());
+ exit (EXIT_FAILURE);
+ }
+
+ /* Sleep again and make another request. $iteration will be 2. */
+ sleep (2);
+ iteration = 2;
+
+ if (nbd_pread (nbd, buf, sizeof buf, 0, 0) == -1) {
+ fprintf (stderr, "%s\n", nbd_get_error ());
+ exit (EXIT_FAILURE);
+ }
+
+ nbd_close (nbd);
+ exit (EXIT_SUCCESS);
+}
diff --git a/tests/test-curl-header-script.c b/tests/test-curl-header-script.c
new file mode 100644
index 00000000..a151af05
--- /dev/null
+++ b/tests/test-curl-header-script.c
@@ -0,0 +1,165 @@
+/* nbdkit
+ * Copyright (C) 2013-2020 Red Hat Inc.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are
+ * met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ *
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ *
+ * * Neither the name of Red Hat nor the names of its contributors may be
+ * used to endorse or promote products derived from this software without
+ * specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+ * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+ * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+ * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+ * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+ * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#include <config.h>
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <stdbool.h>
+#include <stdint.h>
+#include <inttypes.h>
+#include <string.h>
+#include <unistd.h>
+
+#include <libnbd.h>
+
+#include "cleanup.h"
+#include "web-server.h"
+
+#include "test.h"
+
+static int iteration;
+
+#define SCRIPT \
+ "if [ $iteration -eq 0 ]; then echo X-Test: hello; fi\n" \
+ "echo X-Iteration: $iteration\n" \
+ "echo 'X-Empty;'\n"
+
+static void
+check_request (const char *request)
+{
+ char expected[64];
+
+ /* Check the iteration header. */
+ snprintf (expected, sizeof expected, "\r\nX-Iteration: %u\r\n", iteration);
+ if (strcasestr (request, expected) == NULL) {
+ fprintf (stderr, "%s: no/incorrect X-Iteration header in request\n",
+ program_name);
+ exit (EXIT_FAILURE);
+ }
+
+ /* Check the test header, only sent when $iteration = 0. */
+ if (iteration == 0) {
+ if (strcasestr (request, "\r\nX-Test: hello\r\n") == NULL) {
+ fprintf (stderr, "%s: no X-Test header in request\n", program_name);
+ exit (EXIT_FAILURE);
+ }
+ }
+ else {
+ if (strcasestr (request, "\r\nX-Test:") != NULL) {
+ fprintf (stderr, "%s: X-Test header sent but not expected\n",
+ program_name);
+ exit (EXIT_FAILURE);
+ }
+ }
+
+ /* Check the empty header. */
+ if (strcasestr (request, "\r\nX-Empty:\r\n") == NULL) {
+ fprintf (stderr, "%s: no X-Empty header in request\n", program_name);
+ exit (EXIT_FAILURE);
+ }
+}
+
+static char buf[512];
+
+int
+main (int argc, char *argv[])
+{
+ const char *sockpath;
+ struct nbd_handle *nbd;
+ CLEANUP_FREE char *usp_param = NULL;
+
+#ifndef HAVE_CURLOPT_UNIX_SOCKET_PATH
+ fprintf (stderr, "%s: curl does not support CURLOPT_UNIX_SOCKET_PATH\n",
+ program_name);
+ exit (77);
+#endif
+
+ sockpath = web_server ("disk", check_request);
+ if (sockpath == NULL) {
+ fprintf (stderr, "%s: could not start web server thread\n", program_name);
+ exit (EXIT_FAILURE);
+ }
+
+ nbd = nbd_create ();
+ if (nbd == NULL) {
+ fprintf (stderr, "%s\n", nbd_get_error ());
+ exit (EXIT_FAILURE);
+ }
+
+ /* We expect that connecting will cause a HEAD request (to find the
+ * size). $iteration will be 0.
+ */
+ iteration = 0;
+
+ /* Start nbdkit. */
+ if (asprintf (&usp_param, "unix-socket-path=%s", sockpath) == -1) {
+ perror ("asprintf");
+ exit (EXIT_FAILURE);
+ }
+ char *args[] = {
+ "nbdkit", "-s", "--exit-with-parent", "-v",
+ "curl",
+ "-D", "curl.verbose=1",
+ "http://localhost/disk",
+ "header-script=" SCRIPT,
+ "header-script-renew=1",
+ usp_param, /* unix-socket-path=... */
+ NULL
+ };
+ if (nbd_connect_command (nbd, args) == -1) {
+ fprintf (stderr, "%s\n", nbd_get_error ());
+ exit (EXIT_FAILURE);
+ }
+
+ /* Sleep the script will be called again. $iteration will be 1. */
+ sleep (2);
+ iteration = 1;
+
+ /* Make a request. */
+ if (nbd_pread (nbd, buf, sizeof buf, 0, 0) == -1) {
+ fprintf (stderr, "%s\n", nbd_get_error ());
+ exit (EXIT_FAILURE);
+ }
+
+ /* Sleep again and make another request. $iteration will be 2. */
+ sleep (2);
+ iteration = 2;
+
+ if (nbd_pread (nbd, buf, sizeof buf, 0, 0) == -1) {
+ fprintf (stderr, "%s\n", nbd_get_error ());
+ exit (EXIT_FAILURE);
+ }
+
+ nbd_close (nbd);
+ exit (EXIT_SUCCESS);
+}
diff --git a/.gitignore b/.gitignore
index a39aa675..255a97a5 100644
--- a/.gitignore
+++ b/.gitignore
@@ -113,6 +113,8 @@ plugins/*/*.3
/tests/stamp-ssh-user-key
/tests/test-connect
/tests/test-curl
+/tests/test-curl-cookie-script
+/tests/test-curl-header-script
/tests/test-data
/tests/test-delay
/tests/test-exit-with-parent
--
2.27.0
More information about the Libguestfs
mailing list