[Pulp-list] Verifying component services when creating a new Pulp service

[Mark Lamourine]

I'm working to get a Pulp service built in Docker containers and created using Kubernetes.  I'm building on Aaron Weitekamp's initial docker work.

When putting a Pulp service up in containers it is important to know the requirements and boundaries of each component service.  I'd like to have a way to verify each service component before moving on to the next.

With MongoDB and QPID/AMQP, I know how do do this:

Mongo:

  mongo <hostname or IP address>
  > show dbs

I can verify a connnection to the mongodb server, confirm that the pulp database has or has not been created.

QPID:

For a qpid service I can connect and send/recieve a message.

https://gist.github.com/markllama/e0a5db776d2a1c876ff6

  test_qpid.py <hostname or IP address>


What I'd like is some incantation to check that the pulp components have connected to the database.  Do they heartbeat? The same for QPID.   Do they create a set
of topics or queues who's presence and activity would indicate that the components have connected?  Do the pulp services have the equivalent of an echo or ping message which could be used to query the connected processes?


1) Mongo service exists
 a) can connect
 b) database initialized
 c) * celerybeat service connected, heartbeating ??

2) QPID service exists
 a) can connect
 b) can echo

3) Celery beat
 a) Mongodb: db exists and is initialized
 a) Mongodb: Beat heartbeats?
 b) QPID:  Pulp topics/queues exist
 c) QPID:  celery beat service connected, heartbeating ??
 d) QPID:  celery beat service responds to echo request

4) Resource Manager worker
 ...

5) Apache server - content
 ...

6) Apache server - admin
 ...


What I'm really looking for is an internal behavior profile for a working service.

I'd also like a comprehensive list of the expected startup and lifecycle syslog entries for each service.  When a service starts and attaches to mongo/qpid/beat what would the signatures of success look like?  What would be signatures of common failure modes?

Does anything like this exist?  Pointers?

How are people verifying the operation of the service both new and over its life-cycle?

- Mark

-- 
Mark Lamourine <mlamouri at redhat.com>
Sr. Software Developer, Cloud Strategy
Red Hat, 314 Littleton Road, Westford MA 01886
Voice: +1 978 392 1093
http://people.redhat.com/~mlamouri
markllama @ irc://irc.freenod.org*lopsa