On Oct 25, 2011, at 1:26 PM, F.J. Kraan wrote:
> On 2011-10-25 19:01, Hans-Christoph Steiner wrote:
>>
>> On Oct 25, 2011, at 11:44 AM, F.J. Kraan wrote:
>>
>>>
>>>>> On Mon, Oct 24, 2011 at 7:35 PM, Hans-Christoph Steiner<hans(a)at.or.at
>>>>> > wrote:
>>>>>
>>>>>> Ah, course, makes sense. The third item there, the IIR
>>>>>> filters, it
>>>>>> should be not too hard to reproduce the exact same operation
>>>>>> with them
>>>>>> too. With the tests, each one is run in a new Pd instance, so
>>>>>> they're
>>>>>> always starting from scratch. Pd is then quit, and restarted
>>>>>> for the
>>>>>> next test.
>>>>>>
>>>>
>>> But how do you want to create and check the test if the only
>>> environment the test will produce proper results is the unattended
>>> automatic run? If the test would fail, it would be hard to find
>>> out why, as the manual run would always produce a different result.
>>
>> I guess I don't quite follow what you mean here. I am thinking
>> that for a more elaborate test, the test patch would would run thru
>> the IIR filter a few times. Each iteration of the IIR filter test
>> should produce the same result.
> Ok, I see. Here you are testing for repeatability. As an addition to
> a test for regression.
>
> One of the 'requirements' for the test-patch was something that
> makes it easy to create tests, validate and run them, and find the
> cause if the test fails.
Yes, making it easy is the primary concern right now so we can get
lots of tests written and running. But for very specific cases like
the IIR case, we would need to test for repeatability of multiple
iterations. Pd is designed to be 'deterministic', meaning that a
patch should produce the same result every time its run, so things
should be very repeatable.
>>
>> This does remind me tho, the original load_every_help.py script
>> would load each help patch into the same instance of Pd, just
>> continually reusing it. That would trigger a couple hard-to-
>> reproduce bugs that basically only happened when loading every help
>> patch in a certain order. I was focused on getting a report on each
>> help patch, so I changed the script to make a new Pd instance per
>> test. But Pd should be able to load every help patch without
>> crashing, so perhaps we need a second test mode where the same Pd
>> is reused again and again.
> The possibilities are endless here. We should try to stick to a
> sensible subset of these and focus on that. Even then the test set
> can grow larger that Pd itself.
Yes, definitely lets limit the scope to get tests running first. I
was just thinking there could be two modes for running the tests:
every test in a single Pd instance; and, each test in its own Pd
instance. I have one of those cases working, doing the other would
not be too hard, I think. But yes, lower priority.
.hc
----------------------------------------------------------------------------
"A cellphone to me is just an opportunity to be irritated wherever you
are." - Linus Torvalds