[EnMasse] Test documentation

[Jiri Danek]

Hello, I would like to discuss one thing. And that is the amount of
comments in tests.

One thing I dislike about our Polarion-inspired way of doing things in QE
is what I consider excessive test documentation. See example of a test at
the end.

In contrast with that, from what little I've seen, it seems to be that
EnMasse has too little test documentation. If somebody from outside wants
to figure out what test does, say when it fails for them, some
documentation in right place does wonders.

Problems I observed in Dtests tests and some may be affecting EnMasse too

* Nothing explains the purpose of the test, necessary especially if the
test is a multi step scenario which "asserts more than one thing" and there
are actions interspersed with asserts.
* Test names are not descriptive. "testDurableConsumer". Ok, test how? test
what exactly?
* Some comments add nothing of value: "This testDurableConsumer test tests
a durable consumer"
* The test steps  (self.test.add_doc()) just retell the test steps in
English before the code. Now you have to read two tests, the English one
and the Python one.
* If the feature itself is insufficiently explained (or the test author
feels this way). then the author may decide to dump all the knowledge into
comment on test and the test comments may become essentially feature
documentation

is there some arcane Agile knowledge for this?

The example of Dtests test

def test_non_shared_durable_subscription(self):
  """ 1. Non-sharing test:
      Subscribe by the durable subscription to the prepared queue and verify
      that no other subscription can be created.
      2. Durability test:
      Wait for the consumer closing, send some messages and check if
the messages
      can be delivered to another instance of a consumer instance.
  """
  self.test.add_doc(
    '#. Subscribe by a durable subscription to the prepared
"nonshared_subs::nsQueue".',
    'The durable subscription is created.')
  self.test.add_doc('#. Try to create another durable subscription to
the same queue.',
                    'No subscription is created.')
  self.test.add_doc('#. Send a message into "nonshared_subs" address.',
                    'The message is received by the subscriber.')
  self.test.add_doc(
    '#. Wait for the consumer timeout, send another message, subscribe
to the topic again.',
    'The message is received by the consumer.')
  self.do_test_non_shared_subscription()

defopts_test_non_shared_non_durable_subscription = metadata.metadata(
  authors=['xxx'],
  tags=[],
  defects=[
    'ENTMQBR-684',  # More consumers than allowed by queue
"max-consumers" property can be subscribed to the queue in one time
  ],
  importance=metadata.Importance.HIGH,
  level=metadata.Levels.INTEGRATION,
  components=None,  # added by injector,
  subcomponents=metadata.BRsub.MSG_DELIVERY + metadata.CLsub.MSG_DELIVERY,
  type=metadata.Types.NON_FUNCTIONAL,
  subtype1=metadata.Subtype1NF.USABILITY,
  subtype2=metadata.Subtype2.NONE,
  requirements=[
    # requirements filled by injector
  ],
  positive=True,
  dschema_compatible=[],
  approver='amq_machine',
  approved=True,
)
def set_up_test_non_shared_non_durable_subscription(self):
  """ Add a queue to support non-shared non-durable subscription
  into broker.xml configuration file.
  """
  self.dschema.back_up_confs(['artemis-brokers'])
  new_brk_opts = {'core':
                   {'addresses': {'address': [{'@name':'nonshared_subs',
                                  'multicast': {'queue': [{'@name': 'nsQueue',

'@max-consumers': '1',

'@purge-on-no-consumers': 'true'}]}}]}}}
  for self.i_broker in
self.get_srv_iterator(self.bp.artemis_broker_only_interator):
    self.bp.change_broker_config(new_brk_opts, self.bp.BROKER_CONF_KEY)

def tear_down_test_non_shared_non_durable_subscription(self):
  """ Clean up the configuration changes. """
  self.dschema.restore_confs(['artemis-brokers'])
  self.bp.restore_broker_configs()

Thanks,
-- 
Jiri Daněk
Messaging QA
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/enmasse/attachments/20171108/323bed8f/attachment.htm>