Re: [pygame] TODO: add pygame unittests, and test runner stuff to pygame.test package.

[Nicholas Dudfield <ndudfield@xxxxxxxxx>]

Jotted down some thoughts.

Overview
========

Modifying Installer:
  
   pygame.tests
   pygame.examples
   build manifest

Build Script

Test Hub Web App

Whats the point
===============

How many developers out there use PyGame? How large a percentage of those
developers have commit rights? Of those who don't, how many when 
discovering a
bug have the time and inclination to go through the process of getting a 
patch
commited and writing a test for it?

So we have people with no commit rights communicating with those who do, who
finally have to verify the patch/test and merge it in.

Those with commit rights, volunteers all, have other demands on their 
time and
some times it takes a while for this process to complete. Patchers can be
discouraged by this.

How to grease the wheels here?

What information is usually provided from patcher to commiter?

   *   patch
   *   platform
   *   build info
   *   proof of existing bug --> test code

What would an ideal process look like? How to enable it?

Ideally, pygame would contain a build manifest somewhere, so the 
communication
of platform + build info could be automated.

For proof of existing bugs, if the test code was in the form of a 
unittest this
would be ideal.

During the GSOC I created a unittest stub generator, that would create the
skeleton code for a test. It also includes in the comments the 
documentation for
the callable being tested, sourced both from the __doc__ strings and the DOC
files. (The __doc__s tends to be short, containing only function 
signatures and
the makeref contains more extensive documentation)

eg gen_stubs sprite.Sprite >> stub.py

(DOC FILES WOULD NEED TO BE INCLUDED FOR GEN STUBS AS PART OF PACKAGE?)

Having the very latest documentation available in the test code is handy 
so you
can determine whether the function or the documentation is buggy as well as
being very convenient in general while writing tests.

********************************************************************************
* End Users /  Developers  /  Commiters *
*****************************************

It is possible in this context to categorize users of pygame into:

   *    End Users

        People who download a game.

   *    Stable version developers.

        Developers who are working from "installers" eg stable versions of
        pygame.

   *    SVN Developers

        All of these people can be assumed to have a subversion 
checkout of the
        source.

   *    Commiters

        Commit Rights - Duh!

What does that imply?

Py2Exe
======

Running tests from py2exe builds?

Subprocessed tests? How? Dependency on win32pipe for async_sub.py?
(subprocesses are killed after 30 seconds)

Build Info
==========

I modified Brian's build page script to collect build information during the
build + meta info (build script configuration info)

http://pygame-testify.net/results/py25_exe/1642/setup

If each "stable installer" could include something similar that could be 
useful.

Build Script
============

A build script that dumps output and records build information could be 
handy
for debugging builds by making it easier to communicate build information.

POSTing Tests
=============

If there was a pygame.test package that any "End User" could use to post 
test
results + build manifest, that could be useful.

Things that must stay the same
==============================

   *   Conveniently run tests from the trunk dir  eg run_tests.py

   *   Must not have to rebuild/reinstall pygame each time running new 
tests

           >>> development_of_tests == PITA == no_tests_written
           True

Things to consider
==================

   *   Adding size to the installer

       What is __too__ much size? The tests tend to use a lot of the 
data from
       the examples directory for fixtures. So these will by necessity 
have to
       be provided.

What will be included
=====================

Test Runner

Unit Tests

Test Fixtures

Examples

Stub Generator  (DOCS?)

Test POSTer

Keep build information/manifest in the installer so can post a long the 
details.

Test Hub
========

Where to POST the results to? How to view the results? A web interface.

################################################################################
# Conclusion #
##############

Ideas needs further fleshing out, thinking through.

Modifying Installer
  
   pygame.tests
   pygame.examples
   build manifest

Build Script

Test Hub Web App