On Oct 25, 2011, at 1:26 PM, F.J. Kraan wrote:
On 2011-10-25 19:01, Hans-Christoph Steiner wrote:
On Oct 25, 2011, at 11:44 AM, F.J. Kraan wrote:
On Mon, Oct 24, 2011 at 7:35 PM, Hans-Christoph Steiner<hans@at.or.at
wrote:
Ah, course, makes sense. The third item there, the IIR filters, it should be not too hard to reproduce the exact same operation with them too. With the tests, each one is run in a new Pd instance, so they're always starting from scratch. Pd is then quit, and restarted for the next test.
But how do you want to create and check the test if the only environment the test will produce proper results is the unattended automatic run? If the test would fail, it would be hard to find out why, as the manual run would always produce a different result.
I guess I don't quite follow what you mean here. I am thinking that for a more elaborate test, the test patch would would run thru the IIR filter a few times. Each iteration of the IIR filter test should produce the same result.
Ok, I see. Here you are testing for repeatability. As an addition to a test for regression.
One of the 'requirements' for the test-patch was something that makes it easy to create tests, validate and run them, and find the cause if the test fails.
Yes, making it easy is the primary concern right now so we can get lots of tests written and running. But for very specific cases like the IIR case, we would need to test for repeatability of multiple iterations. Pd is designed to be 'deterministic', meaning that a patch should produce the same result every time its run, so things should be very repeatable.
This does remind me tho, the original load_every_help.py script would load each help patch into the same instance of Pd, just continually reusing it. That would trigger a couple hard-to- reproduce bugs that basically only happened when loading every help patch in a certain order. I was focused on getting a report on each help patch, so I changed the script to make a new Pd instance per test. But Pd should be able to load every help patch without crashing, so perhaps we need a second test mode where the same Pd is reused again and again.
The possibilities are endless here. We should try to stick to a sensible subset of these and focus on that. Even then the test set can grow larger that Pd itself.
Yes, definitely lets limit the scope to get tests running first. I was just thinking there could be two modes for running the tests: every test in a single Pd instance; and, each test in its own Pd instance. I have one of those cases working, doing the other would not be too hard, I think. But yes, lower priority.
.hc
----------------------------------------------------------------------------
"A cellphone to me is just an opportunity to be irritated wherever you are." - Linus Torvalds
Hello,
Here is a .zip with an improved method for unit-testing of signal objects. Instead of a template, there is now an abstraction 'unit-test-frame~.pd' handling the tests. The input of that abstraction consists of:
- name of the reference file (32 bit float .wav file, like Roman suggested) - signal (or system) under test - optional delay between test-trigger and test-snapshot (for filters etc.)
The reference file can simply be created from within the unit test patch (running in a 'good' Pd), and must be kept with the test and the abstraction in a directory. The abstraction compares signal under test with reference, computes standard deviation, and sends a tagged result to Pd window and stdout:
STDDEV-ZERO 0 reference: <reference> (when stddev = 0) STDDEV-SMALL <stddev> reference: <reference> (when 0 < stddev < 1e+05) STDDEV-LARGE <stddev> reference: <reference> (when stddev >= 1e+05)
The tags facilitate grepping different categories of 'errors' from a test log. Note that a small stddev may sometimes indicate improved accuracy in an object! Like when Pd-double is tested against current Pd.
By the way I did not find a way to set samplerate for a test. I definitely do not want the test patch to mess up other audio settings, like number of channels. Therefore an audio-dialog message to pd is not an option. However there's an inlet reserved on unit-test-frame~.pd for possible future solution of this issue. From a remark by Roman Haefeli in another thread, http://lists.puredata.info/pipermail/pd-dev/2011-10/017694.html, I learnt that -noaudio may be a useful option for automated signal object testing.
Please report if you have comments on the proposed test stuff. I should mention a hidden agenda behind the efforts: the intention is to produce dozends or even hundreds of unit-tests in short time, and use these for testing Pd-double. Then we could finally make some systematic progress in that field. Apart from that, the tests could be useful for rewrites/refactorings in general, and for tracking down optimizer-troubles.
Katja
.
Sorry, there's a bizarre bug in unit-test-frame~.pd. Don't try to use it at the moment, it's confusing. Loadbanged delay time is different from buttonbanged delay time, even when loadbang activates buttonbang directly. Need to find a workaround.
Katja
Its looking really good, hope the bug doesn't drive you nuts. The only thing I object to is the -utest.pd name. Perhaps I have a really bad memory, but I never remember what stuff like that means. That why I am a big fan of using full words. Typing really doesn't take very long, but if you can't remember what particular abbreviation to use, it can be maddening. How about -unittest.pd?
.hc
On Oct 26, 2011, at 1:29 PM, katja wrote:
Hello,
Here is a .zip with an improved method for unit-testing of signal objects. Instead of a template, there is now an abstraction 'unit-test-frame~.pd' handling the tests. The input of that abstraction consists of:
- name of the reference file (32 bit float .wav file, like Roman
suggested)
- signal (or system) under test
- optional delay between test-trigger and test-snapshot (for filters
etc.)
The reference file can simply be created from within the unit test patch (running in a 'good' Pd), and must be kept with the test and the abstraction in a directory. The abstraction compares signal under test with reference, computes standard deviation, and sends a tagged result to Pd window and stdout:
STDDEV-ZERO 0 reference: <reference> (when stddev = 0) STDDEV-SMALL <stddev> reference: <reference> (when 0 < stddev < 1e+05) STDDEV-LARGE <stddev> reference: <reference> (when stddev >= 1e+05)
The tags facilitate grepping different categories of 'errors' from a test log. Note that a small stddev may sometimes indicate improved accuracy in an object! Like when Pd-double is tested against current Pd.
By the way I did not find a way to set samplerate for a test. I definitely do not want the test patch to mess up other audio settings, like number of channels. Therefore an audio-dialog message to pd is not an option. However there's an inlet reserved on unit-test-frame~.pd for possible future solution of this issue. From a remark by Roman Haefeli in another thread, http://lists.puredata.info/pipermail/pd-dev/2011-10/017694.html, I learnt that -noaudio may be a useful option for automated signal object testing.
Please report if you have comments on the proposed test stuff. I should mention a hidden agenda behind the efforts: the intention is to produce dozends or even hundreds of unit-tests in short time, and use these for testing Pd-double. Then we could finally make some systematic progress in that field. Apart from that, the tests could be useful for rewrites/refactorings in general, and for tracking down optimizer-troubles.
Katja
. <unit-tests.zip>_______________________________________________ Pd-dev mailing list Pd-dev@iem.at http://lists.puredata.info/listinfo/pd-dev
----------------------------------------------------------------------------
Man has survived hitherto because he was too ignorant to know how to realize his wishes. Now that he can realize them, he must either change them, or perish. -William Carlos Williams
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 2011-10-26 19:29, katja wrote:
Hello,
Here is a .zip with an improved method for unit-testing of signal
before the list gets flooded with updates of updates of zip archives with the current installment of the unit-testing framework, how about getting write access to the repository (katja, you already have that, don't you?) and create a new folder "unittests"?
i would probably put that into "TRUNK/externals/unittests" (which really should read "TRUNK/libraries" :-)) for now, and later move it into "TRUNK/". or start right away with "TRUNK/unittests"
mghadf IOhannes
On Oct 27, 2011, at 3:49 AM, IOhannes m zmoelnig wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 2011-10-26 19:29, katja wrote:
Hello,
Here is a .zip with an improved method for unit-testing of signal
before the list gets flooded with updates of updates of zip archives with the current installment of the unit-testing framework, how about getting write access to the repository (katja, you already have that, don't you?) and create a new folder "unittests"?
i would probably put that into "TRUNK/externals/unittests" (which really should read "TRUNK/libraries" :-)) for now, and later move it into "TRUNK/". or start right away with "TRUNK/unittests"
Yes, I agree, that's why we have the SVN. Katja, just post a little intro to pd-dev, and we can get you commit access: http://puredata.info/docs/developer/SVNCommitAccess
Or alternatively, you could also make a new git repo on the github/pd- projects or wherever and host it there.
.hc
----------------------------------------------------------------------------
Terrorism is not an enemy. It cannot be defeated. It's a tactic. It's about as sensible to say we declare war on night attacks and expect we're going to win that war. We're not going to win the war on terrorism. - retired U.S. Army general, William Odom
Reorganized the unit test abstraction once more, now I'm really happy with it. Thanks for all the useful suggestions everyone. I will not spam the list with my patches anymore, it is here, with screenshots, explanation and .zip:
http://www.katjaas.nl/pdunittests/pdunittests.html
Please have a look at it, try it out. There's a few examples which should clarify the idea. In my experience, it takes on average a couple of minutes to create a test patch this way. I hope it is now decent enough to become part of a standardized method for testing Pd-extended.
Katja
-
Hans-Christoph Steiner -
IOhannes m zmoelnig -
katja