[Rhm-users] XML Exchanges

Tue Jul 22 15:02:49 UTC 2008

Hi Andrew,

The examples we ship are for C++ and Python. Setting up the queues and 
bindings for an XML Exchange requires either the C++, Python, or low 
level Java APIs, we will make it possible to set this up with the Java 
JMS URLs, but we aren't there yet.

I don't know where the breaking points are. I'm not trying to be coy, 
but the XML Exchange is a new exchange type that I wrote recently, and I 
don't have anything like the systematic performance profiles we have for 
other exchanges. Each queue binding is one query that must be applied to 
each message sent to the XML Exchange, I have not yet measured how 
increasing the number of queries affects performance. If you had said 
hundreds or thousands, I would certainly have encouraged you to measure 
this aspect carefully, since I haven't yet.

Jonathan

Andrew M wrote:
> Thanks. I would say just 2 or 3 dozen queues at once.  Is there a practical
> limit I'm likely to hit?  Are there any XML exchange examples(java) floating
> around?
>
> Thanks,
> Andrew
>
>
> -----Original Message-----
> From: Jonathan Robie [mailto:jonathan.robie at redhat.com] 
> Sent: Monday, July 21, 2008 7:18 PM
> To: Andrew M
> Cc: rhemrg-users-list at redhat.com; rhm-users at redhat.com
> Subject: Re: [Rhm-users] XML Exchanges
>
> Hi Andrew,
>
> You're right - the XML Exchange is the only exchange type that can look 
> at the content of the message. The other exchange types look at message 
> properties (the XML Exchange can look at both). If clients set up their 
> own queues and created queries that route messages to those queues, the 
> sending applications need no knowledge of the queries that exist or even 
> of which parts of the message are queried.
>
> Performance is not as good as for other exchange types, but you seem to 
> be aware of that already. At this point, we have less performance data 
> on the XML Exchange than for other exchange types (it's much newer!), 
> but from what I can tell, this architecture could well make sense. Any 
> idea how many queues might be active at the same time?
>
> Honestly, the XML Exchange is new enough that the best way to get a feel 
> for its performance with the kind of topology you suggest is probably to 
> come up with a simple prototype.
>
> Jonathan
>
> Andrew M wrote:
>   
>> Jonathan,
>> I'm checking out the FieldTable you mentioned last week. Thanks.
>>
>> I'd like to give you a little background on my application.  If you can
>>     
> make
>   
>> any suggestions I'd love to hear them.  I'm revamping my financial order
>> routing system.  I have trading applications and traders that generate
>> orders something like this:
>>
>> <side=buy share_qty=100 symbol=IBM source=Joe exchange=NYSE>
>> <side=buy share_qty=100 symbol=IBM source=Joe exchange=ArcaECN>
>> <side=buy share_qty=100 symbol=MSFT source=automated_strategy_1
>> exchange=ArcaECN>
>>
>> I have connections to multiple stock and options exchange destinations,
>>     
> some
>   
>> use FIX protocol, going forward some will use their own proprietary
>> protocol. To some exchanges I have multiple connections for redundancy and
>> load distribution.  Currently I'm using FIX protocol for everything both
>> internal and external and my goal is to use AMQP internally to carry order
>> data in a protocol independent format.  AMQP msgs bound for a destination
>> talking FIX would eventually hit my FIX engine for translation into
>>     
> whatever
>   
>> FIX version that counterparty wants.  
>>
>> For routing, I would like an app/trader to just place an order on the bus
>> and then based on the stock/options exchange destination specified, have
>>     
> the
>   
>> order routed to the appropriate destinations.  It would need to hit a
>>     
> queue
>   
>> for orders going to the exchange and also a queue for an order book app
>>     
> that
>   
>> needs all outgoing orders and incoming responses.  Also copies would hit
>> queues for other applications which display the order book on
>>     
> workstations,
>   
>> manage risk, etc.
>>
>> When an app/trader places an order on the bus he specifies an exchange
>> destination, say, NASDAQ.  There might be several FIX connections to
>>     
> NASDAQ,
>   
>> I assume each FIX engine would draw from its own queue on the broker.  I
>> would like the broker to route all orders generated by certain sources
>>     
> down
>   
>> specific exchange connections. In this way, latency sensitive applications
>> could have their traffic kept separate from other less sensitive stuff.
>>     
> Ie.
>   
>> Joe and Bill both send orders specifying NASDAQ.  Joe's orders need to go
>>     
> to
>   
>> the connection NASDAQ1 and Bill's to NASDAQ 2.  This would require routing
>> based on both exchange and order source. In order for the routing to be
>> based on both sources and destination exchanges I think the XQuery would
>>     
> be
>   
>> required, right? 
>>
>> Some things I'd like to do in the future:
>>
>> - Orders originating from some sources would go right to the exchange
>> without approval from a risk mgmt application.  Other traders' orders
>>     
> would
>   
>> be required to hit a risk mgmt application which would look at orders and
>> place approved ones back on the bus for delivery to the exchange.  Broker
>> would need to know which sources' orders require approval and route
>> accordingly.
>>
>> - Using multiple connections to an exchange, if the message
>> translation/gateway app talking to one of those links detects the link is
>> down it modifies the routing rules MRG is using so MRG routes all messages
>> for that exchange to queues for links which are up. 
>>
>> Speed should obviously be as good as possible with latency to a minimum
>>     
> but
>   
>> I'm not going to be shoveling OPRA message rates through it so I think the
>> XQuery performance hit would be ok.  I would think an 8 core box could
>>     
> still
>   
>> saturate my exchange links w/orders anyhow.
>>
>> Maybe I'm trying to do too much w/a broker?  Maybe building all this
>> functionality requires an ESB?
>>
>> I appreciate any suggestions...
>>
>> Andrew
>>
>>
>> -----Original Message-----
>> From: Jonathan Robie [mailto:jonathan.robie at redhat.com] 
>> Sent: Thursday, July 17, 2008 3:46 PM
>> To: Andrew M
>> Cc: rhemrg-users-list at redhat.com; rhm-users at redhat.com
>> Subject: Re: [Rhm-users] XML Exchanges
>>
>> Andrew - do you have to route based on the values in these maps?
>>
>> If not, the XML Exchange is not the most direct way to solve your 
>> problem. The XML Exchange should work fine in Java JMS, but you 
>> currently have to set up your queues in either the low level Java API or 
>> in C++ or Python. I haven't actually tested Java JMS programs that use 
>> the XML Exchange.
>>
>> Jonathan
>>
>> Andrew M wrote:
>>   
>>     
>>> I need to put Map objects on the bus and pass them around for use by 
>>> java, C++ and (potentially).NET applications.  Maps will definitely 
>>> consist of String-to-String mappings. There may also be 
>>> String-to-Float, or String-to-Long, etc.  What is the best way to do 
>>> this?  It does not appear that AMQP supports some native Map type like 
>>> JMS.    Is the correct plan to have C++ apps translate the Map to/from 
>>> XML using this:
>>>
>>> http://www.codesynthesis.com/products/xsd/
>>>
>>>  
>>>
>>> and java apps do the same using this:
>>>
>>> http://xerces.apache.org/
>>>
>>>  
>>>
>>> Then put those bits of XML on the bus destined for an XML Exchange?
>>>
>>>  
>>>
>>> This document...
>>>
>>>
>>>     
>>>       
> http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_MRG/1.0/pdf/Messaging_Tu
>   
>> torial/Messaging_Tutorial.pdf 
>>   
>>     
>>> talks about XML based routing in Python and C++ but not in Java.  Is 
>>> it not possible in Java?
>>>
>>>  
>>>
>>> Thanks,
>>>
>>> Andrew
>>>
>>> ------------------------------------------------------------------------
>>>
>>> _______________________________________________
>>> Rhm-users mailing list
>>> Rhm-users at redhat.com
>>> https://www.redhat.com/mailman/listinfo/rhm-users
>>>     
>>>       
>>   
>>     
>
>
>   