Re: A proposal...
On Mon, Apr 23, 2018 at 11:17 AM, Christophe Jaillet
> Le 23/04/2018 à 16:00, Jim Jagielski a écrit :
>> It seems that, IMO, if there was not so much concern about "regressions"
>> in releases, this whole revisit-versioning debate would not have come up.
>> This implies, to me at least, that the root cause (as I've said before)
>> appears to be one related to QA and testing more than anything. Unless we
>> address this, then nothing else really matters.
>> We have a test framework. The questions are:
>> 1. Are we using it?
>> 2. Are we using it sufficiently well?
>> 3. If not, what can we do to improve that?
>> 4. Can we supplement/replace it w/ other frameworks?
>> It does seem to me that each time we patch something, there should be a
>> test added or extended which covers that bug. We have gotten lax in that.
>> Same for features. And the more substantial the change (ie, the more core
>> code it touches, or the more it refactors something), the more we should
>> envision what tests can be in place which ensure nothing breaks.
>> In other words: nothing backported unless it also involves some changes to
>> the Perl test framework or some pretty convincing reasons why it's not
> +1000 on my side for more tests.
> But, IMHO, the perl framework is complex to understand for most of us.
> Last week I tried to play with it. I tried to update proxy_balancer.t
> because only lbmethod=byrequests is tested.
> The current test itself is really simple. It just checks if the module
> didn't crashed (i.e we receive 200).
> I tried to extend it for the other lbmethod available. This looked as an
> easy task. But figuring the relation between:
> <VirtualHost proxy_http_bal1>
> BalancerMember http://@SERVERNAME@:@PROXY_HTTP_BAL1_PORT@
> still remains a mystery to me.
> The ./test framework could be useful as well.
> At least it is written in C, so the entry ticket should be cheaper for most
> of us.
> But every thing can't be done with it, I guess.
> Maybe, we should at least have some unit testing for each ap_ function? The
> behavior of these function should not change as it can be used by 3rd party
I agree that having a quick way to make function level tests would be
very helpful. It's something largely missing from httpd. (APR has
Even in making mod_log_json, testing it via the test framework is
complicated, as its not a module that changes the output of an HTTP
Request, vs I could make a few quick C-based tests that make sure
things are being serialized correctly very easily.
> The more tests, the better, but I believe that most regressions come from
> interaction between all what is possible with httpd.
> A test-suite is only a test-suite. Everything can't be tested.
> IMHO, as a minimum, all CVE should have their dedicated test which
> explicitly fails with version n, and succeeds with version n+1.
> It would help to make sure than known security issues don't come back.
> Another question with the perl framework.
> Is there a way to send "invalid" data/request with it?
> All, I see is some GET(...). I guess that it sends well formed date.
> Checking the behavior when invalid queries are received would be great.
> Some kind of RFC compliancy check.
> just my 2c,