[pygame] run_tests.py subprocess mode and Build Page extensions

[Nicholas Dudfield <ndudfield@xxxxxxxxx>]

Greetings all,

I recently gave the run_tests.py a bit of an upgrade to run each test in 
subprocesses. I wrote some tests and I think it's ready to use on the 
build page. (It requires win32file and win32pipe though on windows)

There is a post on my blog that goes into a little bit of detail. Also a 
post re: the stub generator.

* http://blog.akalias.net/2008/07/subprocessed.html

* http://blog.akalias.net/2008/07/stubby.html

=== Build Page Extensions ===

I have also been thinking about making some extensions to the build page.

Below are a few idea/points that I would appreciate comments on. Any ideas?

== Raw Data ==

   * Keep raw_data to process at any time. No need to discount old data 
collected from buggy analysis.

== Profiling ==
   
   * Use function wrappers, that log profiling of each test and 
multiple calls.
   
   *  -p|--profile command line mode
   
== Tests ==

   *  Use subprocess mode by default for run_tests.py (on build page)
   *  Web interface for ticketing off tests

== Build information  ==

   * Post compiler version
   * Post complete Setup file
   * Post complete build output
   * Post complete test output
   * Python sys.path
   * Environment variables
   * As much as possible, unprocessed for archives

== Machine information ==

   * Processor speed
   * Cdrom availibility
   * etc. etc

== Breaking up tests  ==
   
Should the tests fail if a machine doesn't have a CD drive (assuming 
stubs were filled out) for example?

Should tests that require Numeric or NumPy fail if neither available?

There are some classes of tests that it seems to make sense to split 
apart from the main "base" group of tests. What should be the "base" 
group of tests to automate with the run_tests.py test runner?

What about tests that require human verification? For the build page a 
"base" group of tests should be specified.

What should be the requirements for machines sending results to the 
build page? Numeric, Numpy? win32 extensions on windows? A CD rom drive? 
32 bit color display?

Cheers,

Nicholas Dudfield.