Hans-Christoph Steiner wrote:
On Dec 21, 2007, at 10:30 AM, IOhannes m zmölnig wrote:
Mathieu Bouchard wrote:
On Mon, 10 Dec 2007, Hans-Christoph Steiner wrote:
What we really need is regression testing, I think that would
make it easier to convince Miller to accept patches, and would be a great tool for finding bugs. Matju started that project, any word on that Matju?No, I haven't done much progress. The problem with that thing is
that no-one wants to do it, not even me. It's hard to consider it
a priority as much as it should be.at least i am using "kind-of" regression tests in zexy and iemlib. the "suite" is rather simplistic, but works well for my purposes.
see zexy/tests for how it works
Any interest in making this a general purpose testing framework? I
have you had a look at it?
would like to come up with a suite that is run every night
automatically, like the nightly builds. It sounds very similar to
what I was thinking.If we just come up with a standard set of inlets, outlets, and a
standard output value,
well, i think i ahve documented this in the README. since i am using the same framework on several projects (not just the 2 mentioned), i would prefer to keep it rather fixed like it is. writing tests is not that much fun for me. right now there are more than 300 tests in iemmatrix...
then it will be easy to write a script to run
thru them all and report on them. Sounds like you might already have
that working to some degree.
have you had a look at it? i am using my suites for >2years now.
I would like to make a set that tests for the strict aliasing bugs so
I can safely use gcc optimizations. This means testing tilde objects.
while there are no explicit tests for ~-objects in any of my suites (i think), they are thought of.
I think it might make sense to start a section in CVS for this called
'tests', then we can start gathering tests there into subfolders
organized by library. Another option would be a standard folder in
each library called 'tests' like you have.
why would you keep them apart?
That's my two bits, but we really should be doing a lot more of this.
i agree.
and despite of saying how great my regression-test-suite is, in reality it is not. and i am using several different versions of it out there [1]
my advice:
- keep the tests as simple as possible, but not simpler
one should be able to write a test fairly easily; i think shortcuts are most likely not needed most of the times (e.g. iirc, matju introduced several shortcuts in his unity-test frameworks to test for mathematically important concepts (like assosiativity of operations); i think it is way simpler to just program 3 tests manually instead)
- as for results i have found it sufficient to have the following
return-states: SUCCESS, FAIL, WAIT i found it makes writing tests simpler if all of them implicitely fail if they do not respond with a return-state _immediately_ (this might make the FAIL-state unnecessary; but again it makes tests simpler if you can provide an explicit FAIL) if the test takes some time (e.g. for signal-objects or timing objects), i return the WAIT state which halts the testing-framework until the currently tested object returns a value.
i usually use "1" for success, "0" for fail, "-1" for wait, the rest is fail too.
write tests to test the framework
allow fail-tests which have to return FAIL-state in order to success
handle crash-tests separately
allow "manual" tests (not "scriptable" (as in "as fast as possible"))
i found that tests-patches should not have arguments (especially:
default-arguments, like "$0"), as this sometimes makes testing of special features unneccessarily complicated.
- ...
fmga.sdr IOhannes
[1] the most current incarnation of my framework is currently used in http://ftm.cvs.sourceforge.net/ftm/ftm/externals/pd/regression/