[Repoze-checkins] r609 - www/trunk/blog.repoze.org/entries

[Tres Seaver]

Author: Tres Seaver <tseaver at palladion.com>
Date: Fri Dec  7 20:14:24 2007
New Revision: 609

Log:
Record notes from yesterday's highway jam. ;)

Added:
   www/trunk/blog.repoze.org/entries/faster_application_restart.txt   (contents, props changed)

Added: www/trunk/blog.repoze.org/entries/faster_application_restart.txt
==============================================================================
--- (empty file)
+++ www/trunk/blog.repoze.org/entries/faster_application_restart.txt	Fri Dec  7 20:14:24 2007
@@ -0,0 +1,81 @@
+Faster Application Restart
+#label general
+<p>
+   While driving back from North Carolina, Chris McDonough and I spent
+   the time talking about ways to make the Repoze developer more
+   productive.
+</p>
+
+<p>
+   We discussed, and then discarded, the idea of trying to fix Python's
+   reload story:  it seems just too painful and error-prone (like the
+   <code>refresh.txt</code> story in Zope).  Instead, we realized that
+   making restarting the application go <bold>really</bold> fast would
+   be a big win:  this observation was highlighted especially by the
+   restart-inducde delays during the demos we did at UNc and CPCC.
+</p>
+
+<p>In no particular order, here are the ideas we came up with:</p>
+
+<ul>
+ <li>Move the current profiling support out of Zope-the-application
+     and make it middleware, so we can measure the various bits
+     more cleanly.  Chris says Ian Bicking has some support already,
+     but only for one request at a time.</li>
+
+ <li>
+     Finish making ZCML parsing side-effect free, so that we can cache
+     the action list (to a ".pyc"-like file) across application runs
+     (a "pickle file").</li>
+
+ <li>
+     That would imply preventing the "eager validation / resolution"
+     of dotted names, currently done at parse-time.  Instead, the parser
+     would create "deferreds" representing the target objects, which would
+     then be resolved / replaced at the point of use.  We would presumably
+     be able to serialize the deferreds in the "pickle file" mentioned above.
+ </li>
+
+ <li>
+     When doing a "fast" restart, signal first an event which allows
+     listeners to dump any global state to a cache on the filesystem;
+     then, on restarting, arrange to signal another event which allows
+     listeners to reload that state, short-circuiting the whole application
+     initialization phase.
+ </li>
+
+ <li>
+     One particularly interesting bit of global state is the set of
+     pickles cached by each ZODB connection.  Repopulating those caches
+     after a restart is one reason that Plone, for instance, takes so
+     long to render the homepage after a restart.  So, check into whether
+     it is feasible to dump and reload the pickles maintained by each
+     ZODB database connection to disk during a "fast restart".
+ </li>
+
+ <li>
+     As an alternative to dumping / reloading the ZODB Connection pickles,
+     investigate whether multiple connections (or even appserver processes)
+     might be able to shart them via a setup using
+     <a href="http://danga.com/memcached/">memcached</a>
+ </li>
+
+ <li>
+     Look into whether we can replace the ZEO disk cache with memcahed.
+ </li>
+
+ <li>
+     Measure "slow" product initializations, and find ways to speed or
+     bypass them on restart (e.g., PlacelessTranslationService).
+ </li>
+
+ <li>
+     In development mode, maybe we can trigger the fast restart whenever
+     any of
+     <code> [getattr(x, '__file__', None) for x in sys.modules.values()]</code>
+     changes.  Maybe find a way to include all parsed ZCML files, too.
+     Can we somehow arrange for the triggering request to be retried after
+     the restart?  Does it matter?
+ </li>
+
+</ul>