[Freeipa-devel] validating return values in XML-RPC

[Dmitri Pal]

John Dennis wrote:
> Sorry to harp on this :-) But the more I work with the XML-RPC
> interface from non-python code the more I think we've got a problem.
>
> The first problem is what was discussed in the team meeting. You don't
> know what a function is going to return and nothing enforces the
> consistency of return values. Jason has done an awesome job of
> enforcing the consistency of input arguments, but that's only half the
> battle. What gets returned is purely a function of what the plugin
> author happens to stuff into the plugin's return statement. There is
> no enforcement of how many values get returned, what their types are,
> what is optional, what is mandatory, etc. In other words everything
> which is enforced on the input side of the call is absent on the
> output side, it should be obvious why this is a problem, especially
> for any callers of XML-RPC which are *not* in the python plugin
> framework.
>
> The second problem I've run into with return values is especially
> pernicious because the plugin framework is hiding a very fundamental
> and apparently common error. Here is the issue:
>
> * We've adopted the convention that *all* strings will be unicode
> objects.
>
> * str objects will be treated as binary data
>
> * Python will in many instances freely convert between str objects and
> unicode objects.
>
> * If a plugin wants to return a string it *must* return a unicode
> object. If the plugin mistakenly returns a str object (a very very
> easy mistake to make) then what gets returned through XML-RPC is a
> *binary* base64 encoded blob, not an XML-RPC *string* value!
>
> The above is so critical let me repeat it: FAILING TO ASSURE A
> RETURNED STRING IS UNICODE AND NOT STR RESULTS IN BINARY BLOBS ON THE
> RECEIVING END INSTEAD OF A STRING.
>
> * However, the python framework *hides* the error on the return side
> because it decodes the base64 binary value back into a str object.
> Because str objects and unicode objects are often interchangeable the
> python code receiving the return value thinks it sees the right result
> even though it's not.
>
> If we're going to have other clients of the XML-RPC interface then
> that client *must* know what the return values are and what their type
> is. It can't (or shouldn't) do things like:
>
> * I was expecting a string but I got a binary blob so that must have
> been a mistake so I'll treat the binary blob as a string and hope it's
> correct.
>
> -or-
>
> * I was expecting an integer but I actually got a string (yes there
> are plugins which do this), so I'll try to read an integer value out
> of the string. But wait, suppose the plugin author who returned the
> integer as a string forget to assure that the string representing the
> integer was a unicode object and not a str object, then the receiver
> really has to start guessing because he's gotten back a binary blob.
> Is that binary blob a 2's compliment representation of a signed
> integer, is it unsigned integer, or is the binary blob a string
> representation of the integer? Clearly this doesn't work.
>
> Now let's suppose another common scenario. The plugin author discovers
> he has mistakenly returned a str object when it should have been a
> unicode object and corrects his mistake, seemingly innocent because
> everything continues to work (but only in python). We have a
> non-python client of the XML-RPC interface who has corrected for the
> mistake by expecting binary data for the string, now that client
> fails! Or let's say the plugin was correctly returning a unicode
> object but some seemingly innocent change is made and the value ends
> up being a str object instead. Once again the python code continues to
> work correctly but the non-python code fails.
>
> So how easy it for Python programmers to make the mistake between str
> and unicode? *VERY VERY EASY!* In fact it's so easy even Jason's
> documentation and examples sometimes make the mistake. It's especially
> easy mistake to make when calling another function because the vast
> majority of existing Python libraries return str objects for strings.
>
> If we don't validate the return values from the plugin's I don't see
> how we'll ever have non-python XML-RPC clients which are robust.
>
> I'm happy to work with Jason to get the return validation implemented.
> I now know the insides of the plugin rpc mechanism in gory detail :-)
>
> I don't think we should postpone this, I think there are all sorts of
> errors lurking in the code, or will get introduced in the code in the
> future. For robustness sake it's better to fix this sooner rather than
> later IMHO.
>
> Of course the other option is to do away with the rule that str
> objects will get converted into binary blobs in XML-RPC. I think there
> is a good argument for treating both str objects and unicode objects
> as strings in XML-RPC. How often do we really need to pass binary data
> back in XLM-RPC? I'm thinking it's not very often. For those limited
> cases wouldn't it be better to require the plugin to return a Binary
> object instead of a str?. This has the advantage that's it's explicit
> and removes the dangerous practice of automatic conversion of
> conventional python strings into binary blobs when 99% of the time
> that's not the intention of the programmer.
>
> BTW, removing the automatic conversion of str to binary is a simple
> one line change which would have a huge impact on robustness without
> any loss of functionality. However that still does not address the
> issue of return value validation, but it does remove the single most
> common error we have with return values at the moment.
>

DO IT!
:-)

-- 
Thank you,
Dmitri Pal

Engineering Manager IPA project,
Red Hat Inc.


-------------------------------
Looking to carve out IT costs?
www.redhat.com/carveoutcosts/