git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Does NeedsRunner work on parameterized tests?


This is a long-standing JUnit bug, and I suspect these tests have never run.

I've tested JUnit 4.13-beta1, and it partially fixes the issue. Parameterized tests now work as long as the include category is set on the entire test class. However individual methods marked with the category are still not executed. So, this works:
@RunWith(Parameterized.class)
@Category(NeedsRunner.class)
public static class NeedsRunnerTests {
@Test
public void testFOO() throws Throwable {
}
}
But the following does not work:
@RunWith(Parameterized.class)
public static class NeedsRunnerTests {
  @Test
@Category(NeedsRunner.class)
public void testWriteThenReadJavaClass() throws Throwable {
  }
}

Since no more JUnit 4 releases are planned (and last I checked we had issues moving to JUnit 5), we might need to refactor
the parameterized tests so that that the NeedsRunner (or ValidatesRunner) tests are in their own test class.

Reuven

On Tue, Dec 18, 2018 at 10:45 AM Kenneth Knowles <kenn@xxxxxxxxxx> wrote:
Upon digging in, it appears to be a JUnit issue. Did it ever work? The includes/excludes are passed along from Gradle to JUnit. Fixed in the as-yet-still-beta 4.13 (notably in RC stages since Feb, and declared to be the last release of JUnit 4).

Kenn

On Tue, Dec 18, 2018 at 12:45 PM Scott Wegner <scott@xxxxxxxxxx> wrote:
We hit a similar issue when trying to refactor some tests to use the 'Enclosed' runner. It seems the Gradle test filtering is not compatible with the Enclosed runner See BEAM-4375 [1] and pr/5440 [2]. 

I'm not familiar with the Parametrized runner, but perhaps it has similar issues.

[2] https://github.com/apache/beam/pull/5440

On Mon, Dec 17, 2018 at 6:15 PM Kenneth Knowles <kenn@xxxxxxxxxx> wrote:
Also FWIW when I tried to repro this, I consistently got many failures and a hang when I tried to run the whole suite via `./gradlew needsRunnerTests`. Filed https://issues.apache.org/jira/browse/BEAM-6256 with some details.

Kenn

On Mon, Dec 17, 2018 at 8:54 PM Reuven Lax <relax@xxxxxxxxxx> wrote:
It looks like maybe this is a JUnit bug: https://github.com/gradle/gradle/issues/1153

The fix is targeted for JUnit 4.13, which is still not fully released (and is intended to be the final JUnit 4 release). 4.13.-beta1 is out though, and it might be worth trying that out to see if it fixes this. Does anyone know how we control the version of JUnit used?

On Mon, Dec 17, 2018 at 5:39 PM Reuven Lax <relax@xxxxxxxxxx> wrote:
Ok, this is worrying - we're silently skipping some important test suites. Does anyone know what might be causing us to skip them?

On Mon, Dec 17, 2018 at 5:03 PM Kenneth Knowles <kenn@xxxxxxxxxx> wrote:
Flipping through the latest Java PreCommit Cron, it does look like NeedsRunner tests within the Parameterized sub-suites of TextIOReadTest are not executed.


Kenn

On Sat, Dec 15, 2018 at 12:16 PM Reuven Lax <relax@xxxxxxxxxx> wrote:
Hi,

I tried to switch one of our tests (AvroIOTest) over to be a parameterized test. However after doing so, I cannot get the NeedsRunner tests to execute! All attempts fail with the following:

No tests found for given includes: [org.apache.beam.sdk.io.AvroIOTest](filter.includeTestsMatching)

I also can't find evidence that Jenkins will execute this either.

Am I missing something, or do our NeedsRunner tests no longer work for parameterized tests? If the latter this is concerning, as there are a number of NeedsRunner paramaterized tests in our codebase (e.g. the TextIO tests).

Reuven


--