Solr Test Report Results & Notes 02/17/2017

Test Results

Lucene-Solr Master Test Beast Results 02/17/2017:19c8ec2bf1882bed1bb34d0b55198d03f2018838 Level Hard Running 100 iterations, 12 at a time, 8 cores

Thanks to those that are helping in this endeavor.


Ignored

The following tests are currently ignored and not run at all. Many of them should probably be changed to BadApple or AwaitsFix and explicitly linked to JIRA issues.


Failing Badly with AwaitsFix, BadApple, or Other Circumstances

The following tests are failing for known reasons and are either incompatible with proper parallel test beasting or awaiting fixes. HdfsChaosMonkeyNothingIsSafeTest is an exception, seems to have hung on 1 run in 100, and will be at least BadApple’d shortly.


Worse than Flakey Tests

The following tests failed over 10% of the time. I have made some improvements to all of them for the next run except LeaderInitiatedRecoveryOnShardRestartTest which I have @AwaitsFix’d. TestReloadDeadlock was hitting it’s custom test suite timeout under this test setup.


Flakey Tests

The following tests failed less than 10% of the time. Some that failed only once or twice did so for common issues unrelated to that specific test. For example the better Overseer close test tracking in SOLR-9846 Overseer is not always closed after being started caused a few fails. Many should go back to passing in the next report. Beyond that I am targeting those that failed 5% or greater on this list next (and have already made improvements to a couple of those).


The Newest Flakey Tests

These tests will get pushback first as things get under control.


Worst Average Test Fail % Over Multiple Reports

This shows the tests with overall fail averages greater than 1% and less than 100%. This highlights tests that flap across reports and those that have been improved or worsened over time.


The Most Expensive Tests

These tests are time expensive to run or beast. The worst of them should be improved, even for Nightly runs.