<div dir="ltr"><div>tl;dr I'm going back to the original plan to have Content Guard authorization in Pulp's processes and not the webserver. Also now we know a) to use urlencoded certificates instead of stripping newlines and b) how to make it work on both apache and nginx having tested it already.<br></div><div><br></div><div>After lots of testing and digging, I believe the best thing to do is to go back to the original idea of having nginx/apache forward a certificate used in a TLS connection with the client to pulp-container for authorization. Here's the summary of reasons why:</div><div><br></div><div>* Trying to use a WSGIAccessScript to check in Python in the webserver isn't an option when you're reverse proxying with either Apache or Nginx. For example, to even do it in Nginx you would have to write the script in Lua. Yikes. So moving the checking to apache/nginx isn't even possible in a reasonable way.<br></div><div>* Having the additional requirement to install all of Pulp on nginx or apache is undesirable at best. It's a requirement likely not easily achieved in deployments like k8s for example meaning maybe Pulp couldn't run there if we did this.<br></div><div>* Forward the certificate is not as bad as I originally thought. Yes you are forwarding the clients cert but you aren't forwarding or re-presenting the client's key so you can't impersonate the client like a MITM would. Also you are forwarding it to a trusted service so that is ok.</div><div>* stripping newlines from rhsm certs causes them to be unreadable by rhsm library so we need to use urlencoding instead of newline strippping to make rhsm certs transport safe for headers</div><div>* urlencoding a tls cert for forwarding is so common nginx even has a pre-defined variable for exactly this. For apache I have a simple mod_rewrite rule handling it. Both were working for me.<br></div><div><br></div><div>Feedback is welcome! Please send any thoughts, questions, or concerns you have. Next steps wise I will close the Epic I opened <a href="https://pulp.plan.io/issues/6323">https://pulp.plan.io/issues/6323</a> and it's subtasks as WONTFIX. I will continue the pulp-certguard work on <a href="https://pulp.plan.io/issues/4664">https://pulp.plan.io/issues/4664</a></div><div><br></div><div>-Brian</div><div><br></div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Mar 12, 2020 at 9:44 AM Brian Bouterse <<a href="mailto:bmbouter@redhat.com">bmbouter@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Mar 11, 2020 at 9:18 PM Justin Sherrill <<a href="mailto:jsherril@redhat.com" target="_blank">jsherril@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>Thanks and it does appears that we use certs that could be too
large for default header sizes (by several multiples). <br>
</p>
<p>Could you elaborate a bit about the design a bit more? I'm
curious what the requirements of web service layer (will it need
to talk to the pulp3 api? the db?) Will it just add some header
after reading the cert (and validating the path) and then pass it
on to the reverse proxy with apache?</p></div></blockquote><div>This was the idea, but in looking through the nginx and apache documentation it doesn't seem possible. I had planned to use WSGIAccessScript, but the pulp-content app isn't wsgi it's ProxyPass so I don't see an access script option for that which means my plan isn't achievable as is. At best nginx has the auth request model <a href="https://nginx.org/en/docs/http/ngx_http_auth_request_module.html" target="_blank">https://nginx.org/en/docs/http/ngx_http_auth_request_module.html</a> but that makes a subrequest and apache doesn't support it. We'd still have the "certs are too big for headers" problem to deal with. I'm configuring nginx and http and doing some testing. Articles are saying the header options would work for both nginx and httpd, so I'm going to do some testing with their configs. <a href="https://connect2id.com/products/server/docs/guides/tls-proxy" target="_blank">https://connect2id.com/products/server/docs/guides/tls-proxy</a></div><div><br></div><div>I'll post my findings and bump this list also for more input.<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>
<p>Thanks!<br>
</p>
<p>Justin<br>
</p>
<div>On 3/11/20 2:58 PM, Brian Bouterse
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr"><br>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Wed, Mar 11, 2020 at 2:34
PM Justin Sherrill <<a href="mailto:jsherril@redhat.com" target="_blank">jsherril@redhat.com</a>> wrote:<br>
</div>
<blockquote class="gmail_quote">
<div>
<p>We had discussed base64 encoding the cert in the
webserver on the way in and then letting cert guard
decode it. While that's not ideal I think it has some
advantages over moving the full auth into the
webserver. What was your motivation for going with that
approach over the base64 encoding approach? <br>
</p>
</div>
</blockquote>
<div>Thank you for this question! I ended up with a few
different concerns about the base64 encode-and-forward idea.
Architecturally the concern with it is that it's frowned
upon to forward the client's TLS cert beyond the TLS
termination point because that is what MITM software does.
Also, there are some practical concerns: one, I don't think
nginx can provide a similar runtime base64 encoding feature.
Also I was concerned with header length truncation and what
happens when the certificates get longer.</div>
<div><br>
</div>
<div>Overall having auth that is based on TLS certificates
brought me to the conclusion that we need to auth where the
TLS is terminated. What do you think?<br>
</div>
<div><br>
</div>
<div>More thoughts and questions are welcome. This is a good
discussion.<br>
</div>
<div> <br>
</div>
<blockquote class="gmail_quote">
<div>
<p> </p>
<div>On 3/11/20 2:11 PM, Brian Bouterse wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>tl;dr: What we have today cannot work with rhsm
certificates which Katello uses. To resolve, we need
to have content guard checking moved to the
webserver configs for apache and nginx and not done
in pulp-content as it is today. <a href="https://pulp.plan.io/issues/6323" target="_blank">https://pulp.plan.io/issues/6323</a></div>
<div><br>
</div>
<div>We need to bring the auth to where TLS is
terminated because we can't being the client certs
to pulp-content due to invalid header characters. As
is, pulp-certguard cannot work with Katello's cert
types (rhsm certs) so that is driving my changes.</div>
<div><br>
</div>
<div>If anyone has major concerns or other ideas
please let me know. In the meantime I'm proceeding
moving the authorization to the webserver and then
updating pulp-certguard to work with that. This will
make pulp-certguard's GA tied to pulpcore 3.3.0.
Feedback is welcome.<br>
</div>
<div><br>
</div>
<div>[0]: <a href="https://pulp.plan.io/issues/6323" target="_blank">https://pulp.plan.io/issues/6323</a><br>
<br>
</div>
<div>Thanks,</div>
<div>Brian<br>
</div>
<div><br>
</div>
</div>
<br>
<fieldset></fieldset>
<pre>_______________________________________________
Pulp-dev mailing list
<a href="mailto:Pulp-dev@redhat.com" target="_blank">Pulp-dev@redhat.com</a>
<a href="https://www.redhat.com/mailman/listinfo/pulp-dev" target="_blank">https://www.redhat.com/mailman/listinfo/pulp-dev</a>
</pre>
</blockquote>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
</blockquote></div></div>
</blockquote></div>