[Repoze-checkins] r658 - www/trunk/blog.repoze.org/entries
Tres Seaver
tseaver at palladion.com
Tue Feb 5 02:56:44 UTC 2008
Author: Tres Seaver <tseaver at palladion.com>
Date: Mon Feb 4 21:56:44 2008
New Revision: 658
Log:
Blog about buildbot setup.
Added:
www/trunk/blog.repoze.org/entries/continuous_integration_via_buildbot-20080214.txt
Added: www/trunk/blog.repoze.org/entries/continuous_integration_via_buildbot-20080214.txt
==============================================================================
--- (empty file)
+++ www/trunk/blog.repoze.org/entries/continuous_integration_via_buildbot-20080214.txt Mon Feb 4 21:56:44 2008
@@ -0,0 +1,152 @@
+Continuous Integration via Buildbot
+#label general
+<h2> Or, How I learned to stop worrying ... </h2>
+
+<h3> Rationale </h3>
+
+<p> After running across a portability / build problem with a distutils
+ package recently, I realized that the testing we are doing for Repoze
+ may not have adequately covered our needs to keep the various Repoze
+ components running against all supported platforms (basically, anything
+ where packages with "C" extensions can be built easily).</p>
+
+<p> After casting around a bit, I decided to bite the bullet and try out
+ <a href="http://buildbot.sourceforge.net">Buildbot</a>, the same tool
+ used to run the Zope unit tests.</p>
+
+<h3> Setting Up the Master </h3>
+
+<p> First, I decided to set up the buildbot "master", running on the
+ repoze.org server. This process tracks all pending changes which
+ require builds, and doles the builds out to one or more designated
+ "buildslaves".</p>
+
+<p> Because bulidbot runs as one or more long-running
+ daemon processes, I decided to create a new account for it, to enforce
+ a bit of privilege isolation:</p>
+
+<pre>
+ $ sudo /usr/sbin/groupadd buildbot
+ $ sudo /usr/sbin/useradd -g bulidbot -c "Buildbot" -m -s /bin/bash \
+ buildbot
+</pre>
+
+ Then, I set up the new user's home directory as a
+ <a href="http://pypi.python.org/pypi/virtualenv">virtualenf</a>.
+
+<pre>
+ $ sudo su - buildbot
+ $ /usr/bin/virtualenv ~
+</pre>
+
+<p> I then installed buildbot and its setuptools-compatible dependencies
+ (note that those dependencies aren't spelled out in its
+ <code>setup.py</code>, but are document in its <code>INSTALL.txt</code>.</p>
+
+<pre>
+ $ bin/easy_install buildbot zope.interface
+</pre>
+
+<p> The remaining dependency is <a href="twistedmatrix.org">Twisted</a>,
+ which <a href="http://twistedmatrix.com/trac/ticket/1286"
+ >can't be installed via <code>easy_install</code></a> (its setup is
+ complex, and hasn't yet adapted to the setuptools regime). So, I
+ installed it the old-fashined distutils way:</p>
+
+<pre>
+ $ mkdir src
+ $ cd src
+ $ wget http://tmrc.mit.edu/mirror/twisted/Twisted/2.5/Twisted-2.5.0.tar.bz2
+ $ tar xjf Twisted-2.5.0.tar.bz2
+ $ cd Twisted-2.5.0
+ $ ../../bin/python setup.py install
+ ...
+</pre>
+
+<p> Now, I could verify that the buildbot application was working:</p>
+
+<pre>
+ $ bin/buildbot --version
+ Buildbot version: 0.7.6
+ Twisted version: 2.5.0
+</pre>
+
+<p> I then created and initiailzed the buildbot "master" work area, and
+ set up a configuration (lots of iterative tweakage elided here): </p>
+
+<pre>
+ $ mkdir -p var/masters/repoze
+ $ bin/buildbot create-master var/masters/repoze
+ $ cp var/masters/repoze/master.cfg{.sample,}
+ $ vim var/masters/repoze/master.cfg
+ $ bin/buildbot start var/masters/repoze
+</pre>
+
+<h3> Setting up a "buildslave"</h3>
+
+<p> Next, I could work on setting up the first "buildslave":<p>
+
+<pre>
+ $ mkdir -p var/slaves/repoze
+ $ bin/buildbot create-slave var/slaves/repoze repoze.org:9980 \
+ laguna <password>
+</pre>
+
+<p> I used the slavename / password which I had created in the
+ <code>master.cfg</code> file. Then, I edited the two "info" files
+ identifying me and my host, and then started the buildbot "slave"
+ daemon:</p>
+
+<pre>
+ $ vim slave/repoze/info/{admin,host}
+ $ bin/buildbot start slave/repoze
+</pre>
+
+<p> It connected, and I could see it on the
+ <a href="http://repoze.org:8010/buildslaves">status page</a>.</p>
+
+<p> Now, I tried a "forced" build, which worked (after more iterative
+ tweakage). The "builder recipe" I'm using does the following whenever
+ it is triggered, either manually or when its "change source" notices a
+ change:</p>
+
+<ol>
+ <li> checks out the branch containing the change
+ (e.g., 'repoze.kiss/trunk');</li>
+
+ <li> runs 'bin/python setup.py test' in the checkout.</li>
+</ol>
+
+<p> The <a href="http://repoze.org:8010/waterfall">"waterfall"</a> page
+ shows the build status.</p>
+
+<p> I then set up another pair of buildslaves, one on my Ubuntu Gutsy laptop
+ and one on a Centos4 server, and ensured that the builds worked with
+ them. Carlos de la Guardia has since volunteered an Ubunty Gutsy server,
+ which is great, as my laptop may or may not be online when a change
+ is triggered.</p>
+
+<h3> Conclusions </h3>
+
+<p> First, the product does what it is designed for: it handles the complex
+ process of dispatching asynchronous builds to the various platforms
+ based on changes to the repository.</p>
+
+<p> Second, the configuration format is an <code>execfile</code'ed Python
+ file, which I consider a bit <strong>too</strong> powerful: I would
+ prefer a purely declarative configuration language. Many users, however,
+ won't need to write any really imperative code in this file, which
+ mitigates the risk somewhat. I chose to add some code here, rather
+ than move it out to an importable module, largely to deal with having
+ a single "scheduler" handle changes to any of the supported branches,
+ and to arrange that each of those branches build in a separate
+ subdirectory.</p>
+
+<p> Third, the code is definitely "frameworky", with lots of base classes
+ and policy objects. It succeeds in hiding a lot of that from the
+ user, however, which covers a lot of sins.</p>
+
+<p> Overall, I like it for what it does, and wouldn't hesitate to use it
+ in other settings.</p>
+
+<p><cite>Tres.</cite></p>
More information about the Repoze-checkins
mailing list