[Pulp-list] Asynchronous Task Dispatching

[Jay Dobies]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

17 minutes.

I have no idea where I just pulled that number from.

Why do we need to query the database for tasks? Can we keep the task
stuff in memory and snap out its state when it changes? Or are you
trying to solve the cross-process task question at the same time?

(the above statement was made with absolutely no thought into task
persistence, so there's probably a million reasons that's a bad idea;
I'm just asking for a few of them)


On 04/14/2011 11:14 AM, Jason L Connor wrote:
> Hi All,
> 
> I'm looking for some feed back on the responsiveness of our system.
> 
> Currently asynchronous tasks are executed from an in-memory task queue.
> This task queue uses a dispatcher thread that wakes up every .5 seconds
> and checks to see if it needs to run anything.
> 
> We're currently in the process of making our tasks persistent by pushing
> them into the database. This means that the dispatcher thread must now
> query the database and deserialize tasks in order to check to see if it
> needs to run anything.
> 
> Personally, I think hitting the database and performing task
> deserialization twice a second is going to create a lot of io waits. I'd
> rather slow the dispatcher thread down to something more reasonable, but
> this will affect how responsive we are in kicking off async tasks.
> 
> I'm looking for suggestions for how long the dispatcher thread should
> sleep. Preferably balancing the io with responsiveness.
> 
> I've got a time in mind, but I'll refrain from suggesting it until I get
> some feedback.
> 
> 
> 
> 
> _______________________________________________
> Pulp-list mailing list
> Pulp-list at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-list


- -- 
Jay Dobies
RHCE# 805008743336126
Freenode: jdob
http://pulpproject.org
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.14 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJNpxfNAAoJEOMmcTqOSQHCYaUH/ixQbdPDIBXVNuwzyJlRzxVD
V2Hc/Nk1vgF06vLVhfzjVQPhX3AkVWPrZdxjHHNq3LXVxef3DqpV2NnB+f0jShFM
PizMpqmvELe0HalCJx/zOfxmCURUPs2kN4D80py2lzAwbn6YDCGBj4NFOPErZ0BY
MVHANtlpsLL48mjZTvcsqK5eYutnN8keYsn4vZgP9spWSxJZOzdiQSdYL57Bx5ii
1MyZkCpJOITwr/f63T02Kuu+Z++CQSHS9onnwo+ksZV/8K+YEOodrCL9SoKjTiFB
PjDuHebp+ihhCmZN+frTzm/QhzRbxzK7xaBUAk6WLf9jakmAg2GIQ1RxUmTdjUQ=
=w2MH
-----END PGP SIGNATURE-----