[Pulp-list] Verifying component services when creating a new Pulp service

[Brian Bouterse]

Hi Mark,

The resource_manager and all pulp_workers emit celery heartbeats which are monitored by celerybeat component. You could subscribe to these heartbeats also through the Qpid broker and process them, but you could get a lot of assurance through easier means.

Each pulp worker has a name that starts with 'reserved_resource_worker-' and then a number. The resource manager has a name that starts with 'resource_manager'. Both of these components automatically create a queue on the broker that starts with their name and ends in '.dq'. Once the process disconnects from the broker for any reason the queue is automatically deleted by the broker. Using this reasoning you list the queues that are available on the broker using `qpid-stat -q` and you should see them. I've included my output [0] as an example.

The pulp_celerybeat component has a similar behavior, but it creates and deletes with a queue that starts with 'celeryev' and ends in a uuid. This is the queue that is monitoring the celery events, one of which is a heartbeat.

For verifying the httpd servers you could use wget to verify they are online and functioning, although I don't have a specific recommendation for that.

The database is less useful for determining which components are actually running because those components "work out of" the database, but don't necessarily note their own existence in the database. You should at least have a pulp_database inside MongoDB created.

-Brian

[0]:  http://ur1.ca/hz95m


----- Original Message -----
> From: "Mark Lamourine" <markllama at redhat.com>
> To: pulp-list at redhat.com
> Sent: Thursday, August 14, 2014 10:31:27 AM
> Subject: [Pulp-list] Verifying component services when creating a new Pulp	service
> 
> I'm working to get a Pulp service built in Docker containers and created
> using Kubernetes.  I'm building on Aaron Weitekamp's initial docker work.
> 
> When putting a Pulp service up in containers it is important to know the
> requirements and boundaries of each component service.  I'd like to have a
> way to verify each service component before moving on to the next.
> 
> With MongoDB and QPID/AMQP, I know how do do this:
> 
> Mongo:
> 
>   mongo <hostname or IP address>
>   > show dbs
> 
> I can verify a connnection to the mongodb server, confirm that the pulp
> database has or has not been created.
> 
> QPID:
> 
> For a qpid service I can connect and send/recieve a message.
> 
> https://gist.github.com/markllama/e0a5db776d2a1c876ff6
> 
>   test_qpid.py <hostname or IP address>
> 
> 
> What I'd like is some incantation to check that the pulp components have
> connected to the database.  Do they heartbeat? The same for QPID.   Do they
> create a set
> of topics or queues who's presence and activity would indicate that the
> components have connected?  Do the pulp services have the equivalent of an
> echo or ping message which could be used to query the connected processes?
> 
> 
> 1) Mongo service exists
>  a) can connect
>  b) database initialized
>  c) * celerybeat service connected, heartbeating ??
> 
> 2) QPID service exists
>  a) can connect
>  b) can echo
> 
> 3) Celery beat
>  a) Mongodb: db exists and is initialized
>  a) Mongodb: Beat heartbeats?
>  b) QPID:  Pulp topics/queues exist
>  c) QPID:  celery beat service connected, heartbeating ??
>  d) QPID:  celery beat service responds to echo request
> 
> 4) Resource Manager worker
>  ...
> 
> 5) Apache server - content
>  ...
> 
> 6) Apache server - admin
>  ...
> 
> 
> What I'm really looking for is an internal behavior profile for a working
> service.
> 
> I'd also like a comprehensive list of the expected startup and lifecycle
> syslog entries for each service.  When a service starts and attaches to
> mongo/qpid/beat what would the signatures of success look like?  What would
> be signatures of common failure modes?
> 
> Does anything like this exist?  Pointers?
> 
> How are people verifying the operation of the service both new and over its
> life-cycle?
> 
> - Mark
> 
> --
> Mark Lamourine <mlamouri at redhat.com>
> Sr. Software Developer, Cloud Strategy
> Red Hat, 314 Littleton Road, Westford MA 01886
> Voice: +1 978 392 1093
> http://people.redhat.com/~mlamouri
> markllama @ irc://irc.freenod.org*lopsa
> 
> _______________________________________________
> Pulp-list mailing list
> Pulp-list at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-list
>