extras-buildsys ChangeLog,1.7,1.8 README,1.2,1.3
Daniel Williams (dcbw)
fedora-extras-commits at redhat.com
Sun Jun 12 08:23:39 UTC 2005
Author: dcbw
Update of /cvs/fedora/extras-buildsys
In directory cvs-int.fedora.redhat.com:/tmp/cvs-serv28967
Modified Files:
ChangeLog README
Log Message:
2005-06-12 Dan Williams <dcbw at redhat.com>
* Require a "hostname" argument to the build server. But we get to remove
it as a config option from CONFIG.py then, which is good :)
* Implement Additional Package Architectures support. This is essentially
an look-aside file listing packages and other arches these packages should
build on, but that a whole distribution isn't built on. Examples include
OpenSSL on SPARC, where sparcv8 and sparcv9 optimized builds are produced,
but not an entire sparcv9 distro. See the "addl_pkg_arches" directory
for an example file, and the CONFIG.py file for more information.
* Simplify build client tracking on the server. There is now only 1 instance
of a BuildClient (formerly BuildClientInstance) class per build client.
Previously there was one instance for each arch the client supported.
Since at this time we don't allow more than one build job per client,
its un-necessary to track anything per-client-per-arch.
* Return the build job UID when queuing a job
* Start jobs on clients immediately when creating them, instead of putting
the server-side tracking object in 'initialize' state first and then
starting the job on the client in the next process() iteration.
* Do some locking when the BuildClient class gets the job the client
is currently running. Because BuildJob instances run in their own threads,
the BuildJob could be starting a new job on the BuildClient while the
BuildClient instance is getting status. This could result in a race
condition and potentially two jobs thinking they have started concurrently.
* Add a "Getting Started" section to the README file
Index: ChangeLog
===================================================================
RCS file: /cvs/fedora/extras-buildsys/ChangeLog,v
retrieving revision 1.7
retrieving revision 1.8
diff -u -r1.7 -r1.8
--- ChangeLog 10 Jun 2005 01:35:31 -0000 1.7
+++ ChangeLog 12 Jun 2005 08:23:37 -0000 1.8
@@ -1,3 +1,35 @@
+2005-06-12 Dan Williams <dcbw at redhat.com>
+
+ * Require a "hostname" argument to the build server. But we get to remove
+ it as a config option from CONFIG.py then, which is good :)
+
+ * Implement Additional Package Architectures support. This is essentially
+ an look-aside file listing packages and other arches these packages should
+ build on, but that a whole distribution isn't built on. Examples include
+ OpenSSL on SPARC, where sparcv8 and sparcv9 optimized builds are produced,
+ but not an entire sparcv9 distro. See the "addl_pkg_arches" directory
+ for an example file, and the CONFIG.py file for more information.
+
+ * Simplify build client tracking on the server. There is now only 1 instance
+ of a BuildClient (formerly BuildClientInstance) class per build client.
+ Previously there was one instance for each arch the client supported.
+ Since at this time we don't allow more than one build job per client,
+ its un-necessary to track anything per-client-per-arch.
+
+ * Return the build job UID when queuing a job
+
+ * Start jobs on clients immediately when creating them, instead of putting
+ the server-side tracking object in 'initialize' state first and then
+ starting the job on the client in the next process() iteration.
+
+ * Do some locking when the BuildClient class gets the job the client
+ is currently running. Because BuildJob instances run in their own threads,
+ the BuildJob could be starting a new job on the BuildClient while the
+ BuildClient instance is getting status. This could result in a race
+ condition and potentially two jobs thinking they have started concurrently.
+
+ * Add a "Getting Started" section to the README file
+
2005-06-09 Dan Williams <dcbw at redhat.com>
* client/buildclient.py
Index: README
===================================================================
RCS file: /cvs/fedora/extras-buildsys/README,v
retrieving revision 1.2
retrieving revision 1.3
diff -u -r1.2 -r1.3
--- README 9 Jun 2005 01:57:34 -0000 1.2
+++ README 12 Jun 2005 08:23:37 -0000 1.3
@@ -1,10 +1,60 @@
Fedora Extras Build System
-The build system is composed of a single build server, and multiple build clients. Both clients and server must be on the same LAN, or at least have access to the same shared storage. Clients run an XMLRPC server to which the build-server delivers build jobs. The build server runs an XMLRPC server to allow submission of jobs, and to retrieve basic status information about both clients and the build system as a whole.
+System Requirements:
+ - Python 2.3 or 2.4
+ - m2crypto
+ - sqlite python bindings
+ - mock (http://www.dulug.duke.edu/~skvidal/mock/)
-The Build Client:
+
+Getting Started
+------------------------------------------
+
+
+0) Follow the instructions at the bottom of this file titled "Configuring SSL for your Build System"
+1) Put the server somewhere, lets call it SERVERDIR
+2) Copy the server Key, Cert, and CA Cert to the SERVERDIR/certs directory
+3) Things to modify in the server's CONFIG.py:
+ - Update the Key, Cert, and CA Cert file options to point to the files in step (2)
+ - Modify the 'targets' option to add/remove the arches and targets you'll be building
+ - Modify the 'builders' option to point to the build clients you'll be using. Note the "https".
+ - If you want to do simple SRPM builds, set the 'use_srpm_not_cvs' option to true
+4) On the client, put the client somehwere, lets call it CLIENTDIR
+5) Create a user for the client. The client _cannot_ run as 'root' (since mock cannot run as 'root')
+6) Copy the client Key, Cert, and CA Cert to the CLIENTDIR/certs directory
+7) Things to modify in the client's CONFIG.py:
+ - Modify the 'distro' and 'repo' options to match the targets you've configured in /etc/mock/. These targets are usually in the form of "distro-target-arch-repo". 'arch' and 'target' are passed by the build system dynamically, but 'distro' and 'repo' are hardcoded in the config file. Examples are "fedora-development-i386-core" and "fedora-development-i386-extras".
+8) Configure the mock target files in /etc/mock. You only need one target file for each major arch you support. For example, you don't need separate 'ia32e' or 'amd64' config files, since these just use the normal 'x86_64' config file
+8) Start the client, NOT as 'root'. ex: "build-client 127.0.0.1 i386 i686"
+9) Start the server. ex: "build-server 127.0.0.1"
+
+If the server finds the client and prints out its address, then everything is well and you can start to submit jobs:
+
+[localhost]# python
+>>> import xmlrpclib
+>>> s = xmlrpclib.ServerProxy("http://<address of your build server>:8887")
+
+If you are using CVS builds, you then do:
+>>> s.enqueue("dcbw", "NetworkManager", "NetworkManager-0_4-11_cvs20050404", "devel")
+
+If you are using simple SRPM builds, you do:
+>>> s.enqueue_srpm("dcbw", "NetworkManager", "/path/to/NetworkManager-0.4-11_cvs20050404.src.rpm", "devel")
+
+The format of the enqueue command is:
+
+enqueue[_srpm](username, package_name, CVS tag _OR_ SRPM path, target)
+
+
+
+Architectural Overview:
------------------------------------------
+The build system is composed of a single build server, and multiple build clients. Clients run an XMLRPC server to which the build-server delivers build jobs. The build server runs an XMLRPC server to allow submission of jobs, and to retrieve basic status information about both clients and the build system as a whole.
+
+
+
+The Build Client:
+
usage: build-client <address> <architectures>
ie : build-client localhost sparc sparcv9 sparcv8
@@ -20,8 +70,9 @@
All communication with the build server is done through SSL to ensure the identity of each party. Both the XMLRPC server and the result file server are SSL-enabled, and require SSL certificates and keys to operate. See later section in this document on how to configure SSL certificates for your build system.
+
+
The Build Server:
-------------------------------------------
usage: build-server
@@ -30,16 +81,16 @@
main()
|- Creates: XMLRPCBuildMaster
|- Creates: BuildClientManager
- |- `- Creates: BuildClientInstance (one for each arch on each ArchWelder)
+ |- `- Creates: BuildClient (one for each remote build client)
|- `- Creates: BuildClientJob (one for each build job on each arch)
`- Creates: BuildMaster
`- Creates: BuildJob (one for each build job)
-The BuildClientManager object serves as a central location for all tracking and status information about each build job on each arch. It creates an BuildClientInstance for each supported architecture of each build client. The BuildClientInstance keeps track of specific jobs building on that single architecture on that single build client. It also serves as the XMLRPC client of the remote build client, proxying status information from it.
+The BuildClientManager object serves as a central location for all tracking and status information about each build job on each arch. It creates an BuildClient instance for each remote build client. The BuildClient instance keeps track of specific jobs building on all architectures on that remote build client. It also serves as the XMLRPC client of the remote build client, proxying status information from it.
-BuildJobs must request that the BuildClientManager create a new BuildClientJob for each build on each architecture the BuildJob needs. If there is an available build client (since build clients only build one job at a time across all arches they support), the BuildClientManager will pass the request to the arch-specific BuildClientInstance, which creates the new arch-specific BuildClientJob, and pass it back through the BuildClientManager to the parent BuildJob. If there is no available build client for the request, the BuildJob must periodically re-issue the build request to the BuildClientManager.
+BuildJobs must request that the BuildClientManager create a new BuildClientJob for each build on each architecture the BuildJob needs. If there is an available build client (since build clients only build one job at a time across all arches they support), the BuildClientManager will pass the request to the arch-specific BuildClient instance, which creates the new arch-specific BuildClientJob, and pass it back through the BuildClientManager to the parent BuildJob. If there is no available build client for the request, the BuildJob must periodically re-issue the build request to the BuildClientManager.
-BuildClientManager has a periodic processing routine that is called from the BuildMaster thread. This processing routine calls the BuildClientInstance.process() routine on each BuildClientInstance, which in turn updates its view of the remote build client's status. Thus, the BuildClientManager, through each BuildClientInstance, knows the status and currently building job on each remote build client.
+BuildClientManager has a periodic processing routine that is called from the BuildMaster thread. This processing routine calls the BuildClient.process() routine on each BuildClient instance, which in turn updates its view of the remote build client's status. Thus, the BuildClientManager, through each BuildClient instance, knows the status and currently building job on each remote build client.
BuildJobs track a single SRPM build through the entire build system. They are created from the BuildMaster thread whenever the BuildMaster finds a job entry in the sqlite database with the status of 'waiting'. BuildJobs proceed through a number of states: "initialize", "checkout", "make_srpm", "prep", "building", "finished", "cleanup", "failed", and "needsign".
More information about the fedora-extras-commits
mailing list