[almighty] [Planner] demo.almighty Dropout Yesterday

[Michael Kleinhenz]

Hi Team,

the demo.almighty.io service went down yesterday for a couple of
minutes. The problem is now fixed and the server is back up, but I
found two underlying, more general problems. Here's my findings in
bullet points:

- We had a problem on code level yesterday in master. That problem
should have been easily
  detected by our test chain. But it was not.

- currently, the functional tests are run against an intermediate
build. We're using the
  WebPack dev/debug function for that. Only after they succeed, the
final build is done
  and the output artifact created. The reason is that we don't need a
test-only http-server setup
  as WebPack already includes that.

- I found out that the error caused by the problem in master was
apperaring ONLY when
  doing a production build, NOT when doing a debug/dev build. The
result was, that the problem
  was completely invisible in a dev environment and for the functional
tests. I don't know yet, why
  the dev and prod build done by WebPack are behaving differently at a
such elementary level.

- The above WebPack mystery would have not affected demo.almighty
without a second problem:
  the build was deployed to demo.almighty althought it failed
building. Resulting in just index.html
  deployed to the server (!).

- I suspect that errors appearing inside the build container are
somehow not propagated to the
  outside Jenkins build. They may just be consumed somewhere.

- The second problem (build errors do not get propagated) is the more
severe one and also may
  affect other builds in other projects as we basically just re-used that setup.

Action points:

- We need to check if we can run the functional tests against the
production build instead of the
  intermediate build.
  -> https://github.com/fabric8io/fabric8-planner/issues/741

- We need to check the build process to find out why errors are not
propagated to the outside
  world (or being recognized as a fail by Jenkins).
  -> https://github.com/fabric8io/fabric8-planner/issues/742

- We need to fix the actual code problems that triggered the above.
  -> https://github.com/fabric8io/fabric8-planner/issues/732

-- Michael

-- 
Michael Kleinhenz
Principal Software Engineer

Red Hat Deutschland GmbH
Werner-von-Siemens-Ring 14
85630 Grasbrunn
Germany

RED HAT | TRIED. TESTED. TRUSTED.
Red Hat GmbH, www.de.redhat.com,
Registered seat: Grasbrunn, Commercial register: Amtsgericht München,
HRB 153243,
Managing Directors: Paul Argiry, Charles Cachera, Michael Cunningham,
Michael O'Neill