From rhallise at redhat.com Wed Nov 1 13:34:42 2017 From: rhallise at redhat.com (Ryan Hallisey) Date: Wed, 1 Nov 2017 09:34:42 -0400 Subject: [Ansible-service-broker] New CI framework Message-ID: Morning team, I've migrated the new ci framework PR to 3.7. The goal of the new framework is to make our CI much more flexible than it was before. We will be able to test many different provision and bind combinations in the broker's Travis CI job. This will come in handy especially when get to more complex multi-container APBs. We will be able to add a new and more complex job without having to hack up shell scripts. If folks have a chance to try it out, here's how CI is controlled: [rhallisey at rhev-i16c-04 ansible-service-broker]$ cat config.yaml # Travis CI provision: mediawiki123 provision: postgresql bind: postgresql verify: rthallisey/service-broker-ci/verify-mediawiki-postgresql.sh unbind: postgresql | mediawiki123 deprovision: mediawiki123 deprovision: postgresql New CI framework PRs - https://github.com/openshift/ansible-service-broker/pull/463 mediawiki PR - https://github.com/ansibleplaybookbundle/mediawiki123-apb/pull/8 postgresql PR - https://github.com/ansibleplaybookbundle/rhscl-postgresql-apb/pull/7 Thanks, Ryan From dwhatley at redhat.com Wed Nov 1 15:55:52 2017 From: dwhatley at redhat.com (Derek Whatley) Date: Wed, 1 Nov 2017 11:55:52 -0400 Subject: [Ansible-service-broker] Multi-bind machine-learning demo with AWS S3, SNS, SQS and scikit-learn Message-ID: Hey all, I've been working on a demo showing a "Stock Analysis APB" powered by scikit-learn binding with 3 AWS services to provide a stock analysis UI for calculating likely relationships between publicly traded companies based on stock price data. S3 provides file storage for results, SNS provides email notifications of available results, and SQS provides a FIFO work queue connecting the worker and webapp containers. You can view the video here: https://www.youtube.com/watch?v=BB0gLQiAnyM I'd like to add support for connection with an RDS-MySQL database in the future, which will allow for caching of retrieved data. Best, Derek -------------- next part -------------- An HTML attachment was scrubbed... URL: From rhallise at redhat.com Wed Nov 1 17:10:36 2017 From: rhallise at redhat.com (Ryan Hallisey) Date: Wed, 1 Nov 2017 13:10:36 -0400 Subject: [Ansible-service-broker] 3.8 label Message-ID: Hey folks, I added a new label '3.8' to temporarily mark bugs that will be targeted for 3.8 until we release. This should help us triage any new issues that come in. Let me know if anyone disagrees with the issues marked for 3.8. Here's the list of 3.7 issues: https://github.com/openshift/ansible-service-broker/issues?utf8=%E2%9C%93&q=is%3Aissue%20is%3Aopen%20label%3Abug%20-label%3A3.8 Here's the list of 3.8 issues: https://github.com/openshift/ansible-service-broker/issues?utf8=%E2%9C%93&q=is%3Aissue%20is%3Aopen%20label%3A3.8 Thanks, -Ryan From shurley at redhat.com Wed Nov 1 19:57:12 2017 From: shurley at redhat.com (Shawn Hurley) Date: Wed, 1 Nov 2017 15:57:12 -0400 Subject: [Ansible-service-broker] asb and etcd in separate pods Message-ID: <31CEA567-36E9-4D6D-A7F5-70936D39128D@redhat.com> Hello Alll, A quick heads up on what is happening. 1. We are merging in CATASB pr: https://github.com/fusor/catasb/pull/173 to test the gate on asb PR: https://github.com/openshift/ansible-service-broker/pull/522 2. We will merge in https://github.com/openshift/ansible-service-broker/pull/522 once it has been tested and the gate is green. 3. We will need to create a release build for the broker. catasb will be in a bad state for a second, once the PR is merged you should be able to use the canary image of broker to deploy with success. run_latest_build.sh will not be working until the latest build is published. Thanks, Shawn -------------- next part -------------- An HTML attachment was scrubbed... URL: From shurley at redhat.com Thu Nov 2 17:07:19 2017 From: shurley at redhat.com (Shawn Hurley) Date: Thu, 2 Nov 2017 13:07:19 -0400 Subject: [Ansible-service-broker] asb and etcd in separate pods In-Reply-To: <31CEA567-36E9-4D6D-A7F5-70936D39128D@redhat.com> References: <31CEA567-36E9-4D6D-A7F5-70936D39128D@redhat.com> Message-ID: Hello All, Currently, the ASB canary container image is working with catasb. Still working on doing the release. Thanks, Shawn > On Nov 1, 2017, at 3:57 PM, Shawn Hurley wrote: > > Hello Alll, > > A quick heads up on what is happening. > > 1. We are merging in CATASB pr: https://github.com/fusor/catasb/pull/173 to test the gate on asb PR: https://github.com/openshift/ansible-service-broker/pull/522 > 2. We will merge in https://github.com/openshift/ansible-service-broker/pull/522 once it has been tested and the gate is green. > 3. We will need to create a release build for the broker. > > catasb will be in a bad state for a second, once the PR is merged you should be able to use the canary image of broker to deploy with success. > > run_latest_build.sh will not be working until the latest build is published. > > > Thanks, > > Shawn -------------- next part -------------- An HTML attachment was scrubbed... URL: From shurley at redhat.com Thu Nov 2 20:15:25 2017 From: shurley at redhat.com (Shawn Hurley) Date: Thu, 2 Nov 2017 16:15:25 -0400 Subject: [Ansible-service-broker] asb and etcd in separate pods In-Reply-To: References: <31CEA567-36E9-4D6D-A7F5-70936D39128D@redhat.com> Message-ID: Hello All, Last email, The release build has been made, and you may need to re pull your images to take the latest changes. Sorry about the delay. ** Note: Dylan?s PR I think is also apart of this release, if you start to see issues with docker hub credentials this could it. Thanks, Shawn > On Nov 2, 2017, at 1:07 PM, Shawn Hurley wrote: > > Hello All, > > Currently, the ASB canary container image is working with catasb. > > Still working on doing the release. > > Thanks, > > Shawn > >> On Nov 1, 2017, at 3:57 PM, Shawn Hurley > wrote: >> >> Hello Alll, >> >> A quick heads up on what is happening. >> >> 1. We are merging in CATASB pr: https://github.com/fusor/catasb/pull/173 to test the gate on asb PR: https://github.com/openshift/ansible-service-broker/pull/522 >> 2. We will merge in https://github.com/openshift/ansible-service-broker/pull/522 once it has been tested and the gate is green. >> 3. We will need to create a release build for the broker. >> >> catasb will be in a bad state for a second, once the PR is merged you should be able to use the canary image of broker to deploy with success. >> >> run_latest_build.sh will not be working until the latest build is published. >> >> >> Thanks, >> >> Shawn > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dzager at redhat.com Fri Nov 10 16:33:01 2017 From: dzager at redhat.com (David Zager) Date: Fri, 10 Nov 2017 16:33:01 +0000 Subject: [Ansible-service-broker] Release Notes Message-ID: Greetings, The past few sprints have been action packed in preparation for OpenShift 3.7 and creating a process for generating and communicating release notes has been on our mind. You can find our first attempt at a release notes generator here . Below, you will find a summary of the changes from late September through yesterday. Let us know what you think and where can improve. Bugs - Bug 1498203 - Extracted Credentials were leaking into new bindings (#469) - Bug 1497819 - Broker should not rely on image field of APB yaml (#433) - Bug 1498185 - Adjust versioning check so that it is done in the registry package (#468) - Bug 1498618 - Support bind parameters. (#467) - Bug 1498992 - Ansible Service Broker template should default (#478) - Bug 1498933 - Do not delete apb-push sourced specs when bootstrapping (#477) - Bug 1498954 - Broker in developer mode must support apb push (#476) - Bug 1500048 - make plan ids globally unique (#480) - Bug 1500934 - Dynamic broker ns for secrets (#482) - Bug 1501512 - bind issue when multiple calls to create the same binding (#486) - Bug 1500930 - Prevent multiple deprovision pods from spawning if a job is already in progress (#488) - Bug 1496572 - Clean up error message for invalid registry credentials. (#490) - Bug 1497766 - Adding ablity to specify keeping namespace alive (#474) - Bug 1497839 - copy secrets to transient namespace and always run (#473) - Bug 1501523 - Set plan name for APB push sourced specs (#495) - Bug 1502044 - deprovision fixes (#494) - Bug 1503233 - Add liveness and readiness checks to ASB dc (#500) - Bug 1499622 - Return 202 if provisioning job is in progress (#498) - Bug 1504729 - Log job state when getting last op (#505) - Bug 1504957 - Broker should use recreate strategy (#511) - Bug 1504250 - Keep listening for deprovision messages (#508) - Bug 1506713 - handle updatable enum parameters properly in schema output (#517) - Bug 1476173 - Skip deprovision if the namespace is being deleted since we (#520) - Bug 1503289 - Move registry credentials to a secret (#502) - Bug 1502044 - add buffer size and work_engine test (#510) - Bug 1507617 - Adding SSL and Authentication to etcd (#522) - Bug 1501523 - Add spec plan to image during apb push (#533) - Bug 1476173 - Cleanup deleting namespaces (#529) - Bug 1507111 - Add support for a local OpenShift Registry adapter (#527) - Bug 1504927 - if apbs fail, mark them as failed. (#534) - Bug 1507111 - Update docs and example configs for local openshift adapter (#538) - Bug 1507111 - Do not force image tag to be IP + Port (#540) Other Enhancements - Remove provision parameters from being reused as binding parameters. (#456) - User Impersonation Implementation (#428) - fix asbcli to work with bearer auth (#455) - Added versioning check to Broker on bootstrap (#457) - changing default for 3.6 run_latest_build to function correctly (#458) - Bump wait times (#461) - Split the deploy.sh script to work with both kube & openshift (#432) - Bearer auth documentation (#460) - Make the prep_local_devel_env script work for Kubernetes & Openshift (#434) - Pass in args to the deploy scripts (#462) - Provide an environment variable to deploy latest with run_latest_build (#466) - add 3.7 releaser to releasers.conf (#465) - Creating proposals for keeping transient namespace alive (#464) - Update run_latest_build w/ origin latest default (#471) - Add troubleshooting documentation to the broker (#479) - Update deployment template to match latest service-catalog in origin (#485) - Update secrets docs to account for new fqname. (#487) - Move the gate to 3.7 (#489) - Fix api auth for ci test (#492) - Adding prometheus metrics for ASB (#497) - Look at the apbs in the catalog for a matching name when creating a secret (#438) - remove trailing spaces from supporting files (#493) - Update schema for instance-update (#444) - Fix gate for Openshift 3.7 (#513) - accept update with bad params and log warnings instead of erroring (#516) - Adding docs for prometheus. (#507) - update resource field names (#519) - Remove checks for DOCKER_USER and DOCKER_PASSWORD (#523) - Call the correct service-catalog namespace (#524) - when building the broker for image also build for linux OS. (#525) - add ServiceClassID and ServiceInstanceID parameters during provision and bind (#515) - Changing the default for auto escalate to false (#503) - grep for correct asb-token for local dev. (#526) - setting default value for the deployment template. (#528) - Setting generated local dev template to autoescalate: false (#532) - Look for the url in the proper place (#535) - Attempting fix for image name. (#539) - Improve logging for missing tags (#536) --- David Zager -------------- next part -------------- An HTML attachment was scrubbed... URL: From dzager at redhat.com Fri Nov 10 16:56:33 2017 From: dzager at redhat.com (David Zager) Date: Fri, 10 Nov 2017 16:56:33 +0000 Subject: [Ansible-service-broker] Release Notes In-Reply-To: References: Message-ID: Thank you to Derek for pointing out a bug in the script. Let us try that again. Bugs - Bug 1498203 - Extracted Credentials were leaking into new bindings (#469) - Bug 1497819 - Broker should not rely on image field of APB yaml (#433) - Bug 1498185 - Adjust versioning check so that it is done in the registry package (#468) - Bug 1498618 - Support bind parameters. (#467) - Bug 1498992 - Ansible Service Broker template should default (#478) - Bug 1498933 - Do not delete apb-push sourced specs when bootstrapping (#477) - Bug 1498954 - Broker in developer mode must support apb push (#476) - Bug 1500048 - make plan ids globally unique (#480) - Bug 1500934 - Dynamic broker ns for secrets (#482) - Bug 1501512 - bind issue when multiple calls to create the same binding (#486) - Bug 1500930 - Prevent multiple deprovision pods from spawning if a job is already in progress (#488) - Bug 1496572 - Clean up error message for invalid registry credentials. (#490) - Bug 1497766 - Adding ablity to specify keeping namespace alive (#474) - Bug 1497839 - copy secrets to transient namespace and always run (#473) - Bug 1501523 - Set plan name for APB push sourced specs (#495) - Bug 1502044 - deprovision fixes (#494) - Bug 1503233 - Add liveness and readiness checks to ASB dc (#500) - Bug 1499622 - Return 202 if provisioning job is in progress (#498) - Bug 1504729 - Log job state when getting last op (#505) - Bug 1504957 - Broker should use recreate strategy (#511) - Bug 1504250 - Keep listening for deprovision messages (#508) - Bug 1506713 - handle updatable enum parameters properly in schema output (#517) - Bug 1476173 - Skip deprovision if the namespace is being deleted since we (#520) - Bug 1503289 - Move registry credentials to a secret (#502) - Bug 1502044 - add buffer size and work_engine test (#510) - Bug 1507617 - Adding SSL and Authentication to etcd (#522) - Bug 1501523 - Add spec plan to image during apb push (#533) - Bug 1476173 - Cleanup deleting namespaces (#529) - Bug 1507111 - Add support for a local OpenShift Registry adapter (#527) - Bug 1504927 - if apbs fail, mark them as failed. (#534) - Bug 1507111 - Update docs and example configs for local openshift adapter (#538) - Bug 1507111 - Do not force image tag to be IP + Port (#540) Other Enhancements - Remove provision parameters from being reused as binding parameters. (#456) - User Impersonation Implementation (#428) - fix asbcli to work with bearer auth (#455) - Added versioning check to Broker on bootstrap (#457) - changing default for 3.6 run_latest_build to function correctly (#458) - Bump wait times (#461) - Split the deploy.sh script to work with both kube & openshift (#432) - Bearer auth documentation (#460) - Make the prep_local_devel_env script work for Kubernetes & Openshift (#434) - Pass in args to the deploy scripts (#462) - Provide an environment variable to deploy latest with run_latest_build (#466) - add 3.7 releaser to releasers.conf (#465) - Creating proposals for keeping transient namespace alive (#464) - Update run_latest_build w/ origin latest default (#471) - Add troubleshooting documentation to the broker (#479) - Update deployment template to match latest service-catalog in origin (#485) - Update secrets docs to account for new fqname. (#487) - Move the gate to 3.7 (#489) - Fix api auth for ci test (#492) - Adding prometheus metrics for ASB (#497) - Look at the apbs in the catalog for a matching name when creating a secret (#438) - remove trailing spaces from supporting files (#493) - Update schema for instance-update (#444) - Fix gate for Openshift 3.7 (#513) - accept update with bad params and log warnings instead of erroring (#516) - Adding docs for prometheus. (#507) - update resource field names (#519) - Remove checks for DOCKER_USER and DOCKER_PASSWORD (#523) - Call the correct service-catalog namespace (#524) - when building the broker for image also build for linux OS. (#525) - add ServiceClassID and ServiceInstanceID parameters during provision and bind (#515) - Changing the default for auto escalate to false (#503) - grep for correct asb-token for local dev. (#526) - setting default value for the deployment template. (#528) - Setting generated local dev template to autoescalate: false (#532) - Look for the url in the proper place (#535) - Attempting fix for image name. (#539) - Improve logging for missing tags (#536) --- David Zager -------------- next part -------------- An HTML attachment was scrubbed... URL: From rhallise at redhat.com Thu Nov 16 01:41:10 2017 From: rhallise at redhat.com (Ryan Hallisey) Date: Wed, 15 Nov 2017 20:41:10 -0500 Subject: [Ansible-service-broker] CI framework Message-ID: Hey folks, The new CI framework is in place. I expect there to be a few bumps at first, but we'll work through those as they come. The new framework should make testing much easier and we can now add gates to every APB we have! Jason, I'll catch up with you tomorrow and we'll talk about migrating the Jenkins jobs. Also, any gate bugs should be reported here: https://github.com/rthallisey/service-broker-ci Thanks, Ryan From dzager at redhat.com Thu Nov 16 14:39:43 2017 From: dzager at redhat.com (David Zager) Date: Thu, 16 Nov 2017 14:39:43 +0000 Subject: [Ansible-service-broker] Broker Bind Credentials Change Message-ID: Greetings, I have, just yesterday, submitted a PR to allow our Ansible Broker to work with bind credentials via secrets. This change is a meaningful one if for no other reason than APBs whose version is not "2.0" will fail validation (if/when you run into this it will look like "Spec [ etherpad-apb ] failed validation for the following reason: [ Specs must be at least version 2.0 ]. It will not be made available."). There are a handful of dependencies: - Update the ansible-asb-module to create a secret in our APB sandbox on `asb-encode-binding` - Remove scripts no longer needed in our apb-base and don't try to keep the pod alive - Bump the version to "2.0" in the apb tool so newly built APBs will be accepted by the broker - Update the broker to handle bind-credentials as a secret Once all of these changes are merged and the associated docker images are built and tagged, APB developers (in order to work with latest broker) must: 1. Get the latest apb-base 2. Bump the version in the apb.yml to "2.0" 3. Rebuild the APB using the latest apb-base *I will send a follow-up email* when these changes have been made. This change has an impact on broker users and APB developers tracking the latest and greatest and wanted to take an opportunity to give advance warning. Respectfully, David Zager -------------- next part -------------- An HTML attachment was scrubbed... URL: From dzager at redhat.com Mon Nov 20 19:27:34 2017 From: dzager at redhat.com (David Zager) Date: Mon, 20 Nov 2017 19:27:34 +0000 Subject: [Ansible-service-broker] Sprint 140 Release Notes Message-ID: Greetings, Please find a summary of changes to the Ansible Service Broker in Sprint 140 below. Regards, David Zager Enhancements - Create release notes script to grab changes (#545) - Integrate with coveralls for code coverage (#548) - remove unneeded paragraph from license header (#549) - Expose the pod name/namespace to APB (#546) - Proposal to improve bind credential extraction (#550) - Setup tls support for k8s deployments (#496) - using table driven testing (#551) - Use the Kubernetes API for namespaces check (#552) - Integrate a new ci framework for travis (#463) - Update copr link in Makefile comments (#559) - adding ability to connnect over SSL w/o authentication. (#558) - Pull apb templates from their respective git repos (#560) - First pass at last_operation description proposal (#537) - Update vendor directory (#562) - Create a Kubernetes Client struct (#561) - Rebase k8s templates to pickup etcd name change (#563) -------------- next part -------------- An HTML attachment was scrubbed... URL: From rhallise at redhat.com Mon Nov 20 20:43:43 2017 From: rhallise at redhat.com (Ryan Hallisey) Date: Mon, 20 Nov 2017 15:43:43 -0500 Subject: [Ansible-service-broker] Openshift 3.8 Message-ID: Hey folks, If you're using catasb with the "latest" tag from origin, it uses kubernetes 1.8 and there are a few issues folks are hitting during deployment. Until https://github.com/openshift/origin/pull/17385 merges, you need to disable swap on your host before running catasb. `swapoff -a` After that change merges, we'll need to add `--fail-swap-on=false` to /etc/systemd/system/kubelet.service.d/10-kubeadm.conf as an operation in catasb. If you're running the broker locally, you will hit the same issue I'm hitting in kubernetes. We run `oc login` to the cluster and your project will be changed to the default namespace. This causes some issues with permissions. I think in the near term, we need to: patch the gate to keep it green 3.8. v3.7.0-rc.0 From rhallise at redhat.com Mon Nov 20 20:48:28 2017 From: rhallise at redhat.com (Ryan Hallisey) Date: Mon, 20 Nov 2017 15:48:28 -0500 Subject: [Ansible-service-broker] Openshift 3.8 In-Reply-To: References: Message-ID: Sorry pressing enter sent the message.... I think in the near term, we need to patch the gate to keep it green 3.8. If you need a working env and don't want to deal with the above workarounds/issues use the origin tag: v3.7.0-rc.0 Thanks, -Ryan On Mon, Nov 20, 2017 at 3:43 PM, Ryan Hallisey wrote: > Hey folks, > > If you're using catasb with the "latest" tag from origin, it uses > kubernetes 1.8 and there are a few issues folks are hitting during > deployment. > > Until https://github.com/openshift/origin/pull/17385 merges, you need > to disable swap on your host before running catasb. > `swapoff -a` > > After that change merges, we'll need to add `--fail-swap-on=false` to > /etc/systemd/system/kubelet.service.d/10-kubeadm.conf as an operation > in catasb. > > If you're running the broker locally, you will hit the same issue I'm > hitting in kubernetes. We run `oc login` to the cluster and your > project will be changed to the default namespace. This causes some > issues with permissions. > > I think in the near term, we need to: > patch the gate to keep it green 3.8. > > v3.7.0-rc.0 From ernelson at redhat.com Thu Nov 30 18:18:27 2017 From: ernelson at redhat.com (Erik Nelson) Date: Thu, 30 Nov 2017 13:18:27 -0500 Subject: [Ansible-service-broker] Catalog / OSB F2F Summary Message-ID: Hello all, Over the last 3 days, the Service Catalog and OSB working groups held a face to face meeting here in Raleigh, here are some of the highlights as they relate to the Ansible Broker. https://github.com/eriknelson/cat-osb-f2f-2017/blob/master/README.md Some new, interesting stuff coming up: - Async bind is basically here; we'll be working with the upstream catalog and broker spec groups validating the proposals through our implementation and providing feedback so we can get it released in the master spec. Target is the end of January for the OSB group. - Bind Credential Remapping: often a service may return a set of credentials with service specific keys (MYSQL_HOST), when a binding consumer may expect the credentials in the form of something more generic (DB_HOST). This feature will allow for remapping the keys from source to destination once injected into an application. - Generic Actions - The OSB working group is exploring formal ways extend the default OSB actions (provision, bind), allowing for brokers to implement custom actions such as backup/restore, metrics/health, and other arbitrary features. This one is a little farther down the road, but there are proposals actively being worked on that we hope to submitted in the near future. ...and many more! - Erik