[Pulp-dev] Tasking System Changes in 3.0

Brian Bouterse bbouters at redhat.com
Thu Sep 15 21:46:29 UTC 2016


In porting the tasking system to work with the new models I have made 
some adjustments that I think are improvements. Please send feedback or 
leave it directly on the PR. I plan to turn this into documentation at 
some point as a separate effort. This applies to methods we will mark in 
the plugin API such as sync(), publish() which are run inside of a pulp 
task dedicated to calling into the plugin code.

Here is a recap so everyone is aware and can give feedback:

== Improvements over Fatal vs Non Fatal Exceptions in Tasking System ==
- The tasking system allows non_fatal_errors to be recorded as the task 
runs. This is different than before which required any error recording 
to be done using the task result. @dkliban pointed out that recording as 
you go is better in case you later experience an unexpected, fatal 
exception. See the new plugin controller here[0].

- Fatal exceptions are any exception raised during Task execution. This 
is roughly what we did before but we did a poor job of documenting it.

- Fatal exceptions are recorded with the exception traceback in the 
'result' attribute. Previously this was recorded in the 'errors' 
attribute which caused mixing/overwriting problems between fatal and 
non-fatal exceptions. In this new code, if you experience a fatal error 
the result *is* your exception. See the implementation here [1]

- The errors field is renamed to non_fatal_errors to clearly indicate 
its semantic purpose to collect non_fatal_errors[2]. It is still a JSON 
field but it is now structured due to the use of the plugin controller 
[0]. It probably could be more structured, or less. Input would be great 
here.

== Less work for plugin writers ==
- spawned_tasks are now populated as you dispatch tasks from within a 
task. Previously it was the plugin writer's job to name any spawned 
tasks. This is one more responsibility moved to platform. See the 
implementation wherever this method is used [3]. If you want to dispatch 
tasks and not have them in the spawned_tasks list making it inherit from 
PulpTask[4] directly will do that.

== Less code to Maintain ==
- We no longer support sigterm handlers and are going to assume that a 
task can be killed at any moment. We deprecated this part of the old 
Plugin API on the 2.y line even. On 2.y, the tasking system used to have 
code which would "install" a sigterm handler callback for that specific 
task. That installation code is now deleted [5].

- The Task.results field is a JSON field and it stores anything returned 
by the task [6]. We used to maintain an object called TaskResult which 
no longer has value with all the changes ^. Plugin writers can just 
return a JSON serializable python object as a task result.

[0]: 
https://github.com/pulp/pulp/pull/2752/files#diff-d378ced2dc71e64a3ff4985690c331b4R5
[1]: 
https://github.com/pulp/pulp/pull/2752/files#diff-a5f1119f1229212c0027870093b9356cR418
[2]: 
https://github.com/pulp/pulp/pull/2752/files#diff-f2e147f346d6cb888f808c77a58e84bdR134
[3]: 
https://github.com/pulp/pulp/pull/2752/files#diff-a5f1119f1229212c0027870093b9356cR426
[4]: 
https://github.com/pulp/pulp/pull/2752/files#diff-a5f1119f1229212c0027870093b9356cR33
[5]: 
https://github.com/pulp/pulp/pull/2752/files#diff-abee42f9c3a1b498c07cb513ca7cb974L629
[6]: 
https://github.com/pulp/pulp/pull/2752/files#diff-a5f1119f1229212c0027870093b9356cR366

-Brian




More information about the Pulp-dev mailing list