From aconway at redhat.com Mon Aug 4 19:57:43 2008 From: aconway at redhat.com (Alan Conway) Date: Mon, 04 Aug 2008 15:57:43 -0400 Subject: [Rhemrg-users-list] RE: [Rhm-users] how many queues can I have? In-Reply-To: <023e01c8ee76$c19554c0$44bffe40$@net> References: <011101c8edc4$cce4aa90$66adffb0$@net> <4888DDE7.6070702@redhat.com> <013201c8edc8$a000aca0$e00205e0$@net> <48893067.4060904@idi-middleware.com> <023e01c8ee76$c19554c0$44bffe40$@net> Message-ID: <1217879863.20894.166.camel@rolf> On Fri, 2008-07-25 at 11:51 -0500, Andrew M wrote: > I'm not sure if I understand. Let's say I have producers A and B sending > orders. A doesn't need to (and shouldn't) see B's executions returned from > the exchange and B shouldn't see A's. Some other application may need to > see both A's and B's. wuddyathink? > > Thanks, > Andrew > > > -----Original Message----- > From: rhm-users-bounces at redhat.com [mailto:rhm-users-bounces at redhat.com] On > Behalf Of Bill Whiting > Sent: Thursday, July 24, 2008 8:46 PM > To: rhm-users at redhat.com > Cc: rhemrg-users-list at redhat.com; rhm-users at redhat.com > Subject: Re: [Rhm-users] how many queues can I have? > > Unless the replies are being read by separate processes, then send all > of the replies back to a single queue. It's much more efficient, and > there's no benefit in that case to separating the reply messages. > > //Bill > > Andrew M wrote: > > > > Well let's say I have various producers sending stock orders before > > the open of trading. When the market opens those orders are filled or > > cancelled and execution reports come back. I want to make sure those > > executions get to the app who produced the order. Before the open I > > may have several hundred thousand orders pending. Is there a better > > way to go where I don't use an individual temp queue for each order's > > reply? To generalize on Bills suggestion: create a queue per producer process rather than per order. Each producer process subscribes to its own execution report temp queue and matches up the reports to the original orders. Each producer sees only the reports for orders that it issued itself. From aconway at redhat.com Mon Aug 4 20:28:17 2008 From: aconway at redhat.com (Alan Conway) Date: Mon, 04 Aug 2008 16:28:17 -0400 Subject: [Rhemrg-users-list] RE: [Rhm-users] load balancing In-Reply-To: <4886168E.3010906@redhat.com> References: <07cf01c8eb43$4d890260$e89b0720$@net> <4884ABDC.8090809@redhat.com> <07fe01c8eb50$b5a1f3e0$20e5dba0$@net> <48852D7A.6090205@redhat.com> <08ab01c8ec02$77db9c70$6792d550$@net> <4886168E.3010906@redhat.com> Message-ID: <1217881697.15148.11.camel@rolf> On Tue, 2008-07-22 at 18:19 +0100, Gordon Sim wrote: > Andrew M wrote: > > I?m thinking of two federated brokers at different data centers > > connected by a WAN link. An application would publish messages directly > > to a single queue existing on both of the federated brokers. A number > > of consumers would consume messages from that queue from either broker. > > Say I have 5 consumers talking to the broker at data center ?A? and 5 > > talking to data center ?B?. Assuming the WAN link is fast, could I > > expect each of the ten consumers to get roughly 10% of the traffic? > > No, not with the current implementation I'm afraid. I think it depends on the relative time spent in different parts of the system. The current impl will distribute unfairly if the application is doing no work except pulling messages. If the application is spending time processing each message and you use consumer flow control with a reasonably sized window then it may not be so bad. From andrew at oc384.net Thu Aug 7 16:22:03 2008 From: andrew at oc384.net (Andrew M) Date: Thu, 7 Aug 2008 11:22:03 -0500 Subject: [Rhm-users] RHM MRG vs OMG DDS Message-ID: <03ff01c8f8a9$bb033450$31099cf0$@net> I'm looking for a pub/sub message distribution system for high frequency market data. OPRA rates are heading for 1 million ticks/sec so it has to handle those volumes. Has anyone compared MRG messaging w/OMG's DDS? Can someone tell me if one is more suited to this application than another? I will have a number of applications subscribing to quote data based on stock/options symbol. http://portals.omg.org/dds/OMG_Data_Distribution_Portal http://portals.omg.org/dds/DataDistributionIntro http://www.ociweb.com/products/opendds http://www.omg.org/technology/documents/dds_spec_catalog.htm Thanks, Andrew -------------- next part -------------- An HTML attachment was scrubbed... URL: From bche at redhat.com Thu Aug 7 16:47:12 2008 From: bche at redhat.com (Bryan Che) Date: Thu, 07 Aug 2008 12:47:12 -0400 Subject: [Rhm-users] Re: [Rhemrg-users-list] RHM MRG vs OMG DDS In-Reply-To: <03ff01c8f8a9$bb033450$31099cf0$@net> References: <03ff01c8f8a9$bb033450$31099cf0$@net> Message-ID: <489B2710.1060101@redhat.com> Andrew, we recently did a benchmark of MRG Messaging in Intel's low latency lab, where on an 8-core Xeon box with 4 1GB NICs, we scaled to over 6 million OPRA messages/second. I'd be happy to send you a copy of the report if you'd like. Bryan Andrew M wrote: > I?m looking for a pub/sub message distribution system for high frequency > market data. OPRA rates are heading for 1 million ticks/sec so it has > to handle those volumes. Has anyone compared MRG messaging w/OMG?s > DDS? Can someone tell me if one is more suited to this application than > another? I will have a number of applications subscribing to quote data > based on stock/options symbol. > > http://portals.omg.org/dds/OMG_Data_Distribution_Portal > > http://portals.omg.org/dds/DataDistributionIntro > > http://www.ociweb.com/products/opendds > > http://www.omg.org/technology/documents/dds_spec_catalog.htm > > > > Thanks, > > Andrew > > > > > ------------------------------------------------------------------------ > > _______________________________________________ > Rhemrg-users-list mailing list > Rhemrg-users-list at redhat.com > https://www.redhat.com/mailman/listinfo/rhemrg-users-list -- Bryan Che Red Hat, Inc. 314 Littleton Rd Westford, MA 01886 978-392-3107 bche at redhat.com From bche at redhat.com Mon Aug 18 15:45:23 2008 From: bche at redhat.com (Bryan Che) Date: Mon, 18 Aug 2008 11:45:23 -0400 Subject: [Rhm-users] Re: [Rhemrg-users-list] C# API In-Reply-To: <532EAD9BE204294BABBC3147C2403A2901400535@FPNTEXCBE02.services.local> References: <03ff01c8f8a9$bb033450$31099cf0$@net> <489B2710.1060101@redhat.com> <532EAD9BE204294BABBC3147C2403A2901400535@FPNTEXCBE02.services.local> Message-ID: <48A99913.50008@redhat.com> Shahbaz Chaudhary wrote: > Unfortunately, there I can't find a lot of developer friendly > information on RHM site. > > Is there a C# API for RHM 1.0? If not, when is one expected? There isn't one for v1. We are planning to release a C# client for our 1.1 update later this year > > Which version of Qpid is closest to RHM 1.0? We released off a snapshot of qpid. M2 is closest. > > ------------- > I'm currently using Qpid's M2 Java broker; however, my queues keep > getting filled up. It is not easy for me to find out which queues are > filling up and why. I'm hoping the RHM version will have better tools > and/or have better error messages, better performance (if that is the > problem), etc. We do have a number of management tools for MRG, and we will have even more for our 1.1 release. Bryan > > Thanks > > _______________________________________________ > Rhemrg-users-list mailing list > Rhemrg-users-list at redhat.com > https://www.redhat.com/mailman/listinfo/rhemrg-users-list -- Bryan Che Red Hat, Inc. 314 Littleton Rd Westford, MA 01886 978-392-3107 bche at redhat.com From cctrieloff at redhat.com Thu Aug 21 00:48:21 2008 From: cctrieloff at redhat.com (Carl Trieloff) Date: Wed, 20 Aug 2008 20:48:21 -0400 Subject: [Rhm-users] Re: [Rhemrg-users-list] C# API In-Reply-To: <48A99913.50008@redhat.com> References: <03ff01c8f8a9$bb033450$31099cf0$@net> <489B2710.1060101@redhat.com> <532EAD9BE204294BABBC3147C2403A2901400535@FPNTEXCBE02.services.local> <48A99913.50008@redhat.com> Message-ID: <48ACBB55.5040203@redhat.com> Bryan Che wrote: > Shahbaz Chaudhary wrote: >> Unfortunately, there I can't find a lot of developer friendly >> information on RHM site. >> >> Is there a C# API for RHM 1.0? If not, when is one expected? > > There isn't one for v1. We are planning to release a C# client for > our 1.1 update later this year > >> >> Which version of Qpid is closest to RHM 1.0? > > We released off a snapshot of qpid. M2 is closest. Trunk is actually closer. Carl. From cctrieloff at redhat.com Wed Aug 27 13:50:29 2008 From: cctrieloff at redhat.com (Carl Trieloff) Date: Wed, 27 Aug 2008 09:50:29 -0400 Subject: [Rhm-users] Re: [Rhemrg-users-list] RHM Performence In-Reply-To: <4df0fedf0808270612s59f81c28pf2079c78d2e5c612@mail.gmail.com> References: <4df0fedf0808270612s59f81c28pf2079c78d2e5c612@mail.gmail.com> Message-ID: <48B55BA5.4000009@redhat.com> I can't comment specifically as I don't know how you have modified the example, however... Some of the differences between the example and perftest is that perftest sets a few additional things. These include flow control, accept mode, and by default is aggressive on consumer ack, etc. If you are tuning for a specific case, the easiest is play with perftest options till you get want you want and then have a look in the perftest code (it is all in one file) on how to set the options. However I expect you will get to perftest numbers with the above 4 options, I have mail you examples privately. In the case where you are chaining client, my first quest is that you have some logic that is introducing head of line blocking - just guessing again in absence of detail. Another easy way to see how MRG performs in a chain setup is to configure federation. regards Carl. mark yoffe wrote: > Hi > > i have been using the system for some and have some quetions regarding > performence > > ---- > 1 > ----- > i have used the PerfTest to benchmark the performance > and i receive results of around 70,000 transfers per second with the > default values (message size 1024, 1 publish , 1 subscribe, etc..) > with 1 Byte size messages i get around 160K transfers per second > > than i used the request response example privded with the system and > changed the message limit to be unlimited > and tested based on the example a transmission of 100,000 messages - > the best time i got was 10 seconds for round trip time > > so if i translate the numbers correctly 100K messages each way is 200K > messages total > which is 20K per seconds not close > > which translates to around 20,000 > > > can someone please explain the differences between the numbers , is > there something missing in the request response example to achieve hi > performance results? > > -- > 2 > --- > > i have made another experiment using three RHMclients in an experiment > > where one sends a message to the seconds and he second to the third , > than the third returns it to the first (1->2->3->1) > > each one employs a similar technique to the request response server > from the examples provided with the product > > the results i receive show that the performance is slower than the > request - response example > > the change is not linear what took 10 seconds in the first test (point > to point and back) now takes more than double the time > although only one more "communication step" was added to the scheme > > can someone please explain this behavior? is this correct? should the > system behave like this? > > Regards > > > Mark Yoffe > > ------------------------------------------------------------------------ > > _______________________________________________ > Rhemrg-users-list mailing list > Rhemrg-users-list at redhat.com > https://www.redhat.com/mailman/listinfo/rhemrg-users-list > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aconway at redhat.com Wed Aug 27 14:53:30 2008 From: aconway at redhat.com (Alan Conway) Date: Wed, 27 Aug 2008 10:53:30 -0400 Subject: [Rhm-users] Re: [Rhemrg-users-list] RHM Performence In-Reply-To: <4df0fedf0808270612s59f81c28pf2079c78d2e5c612@mail.gmail.com> References: <4df0fedf0808270612s59f81c28pf2079c78d2e5c612@mail.gmail.com> Message-ID: <1219848810.3198.29.camel@rolf> On Wed, 2008-08-27 at 16:12 +0300, mark yoffe wrote: > Hi > > i have been using the system for some and have some quetions regarding > performence > > ---- > 1 > ----- > i have used the PerfTest to benchmark the performance > and i receive results of around 70,000 transfers per second with the > default values (message size 1024, 1 publish , 1 subscribe, etc..) > with 1 Byte size messages i get around 160K transfers per second > > than i used the request response example privded with the system and > changed the message limit to be unlimited > and tested based on the example a transmission of 100,000 messages - > the best time i got was 10 seconds for round trip time > > so if i translate the numbers correctly 100K messages each way is 200K > messages total > which is 20K per seconds not close > > which translates to around 20,000 > > > can someone please explain the differences between the numbers , is > there something missing in the request response example to achieve hi > performance results? > > -- > 2 > --- > > i have made another experiment using three RHMclients in an experiment > > where one sends a message to the seconds and he second to the third , > than the third returns it to the first (1->2->3->1) > > each one employs a similar technique to the request response server > from the examples provided with the product > > the results i receive show that the performance is slower than the > request - response example > > the change is not linear what took 10 seconds in the first test (point > to point and back) now takes more than double the time > although only one more "communication step" was added to the scheme > > can someone please explain this behavior? is this correct? should the > system behave like this? > Not sure how you altered the examples but the request-response example as it is coded has an important performance error: it first sends all the messages from the main() thread, and then waits for all the responses in the main() thread. So there is no interleaving of sending and receiving. This is to avoid complicating the examples with threading details. Perftest on the other hand starts a separate thread for sending and receiving, so send & receive are done in parallel. The other examples have separate publish and subscribe clients so they can also be run in parallel. For any high-performance app you should be sending & receiving asynchronously in parallel streams: either separate threads with separate sessions or in separate processes.