JetBrains Rider is a fantastic tool, and full disclosure, as of writing this post, I am a developer advocate for JetBrains. XUnit is also an excellent unit testing framework, and their combined powers create an out-of-this-world testing experience.

One feature JetBrains Rider users have fallen in love with is Run Selected Tests Until Failure, which is what it sounds like. JetBrains Rider will continue executing your unit test until the test fails. The test runner feature can help diagnose finicky unit tests that work intermittently. Nothing is worse than an inconsistent bug, am I right?!

When performing an investigation, you want as much information as possible. For example, a common question might be, “How many times did my test run before failing?”.

In this post, we’ll build an XUnit fixture you can use in your tests to determine the number of iterations before a failure. Let’s get started.

XUnit Unit Tests and Fixtures

XUnit is a unit testing library that provides a framework for building a test suite. It’s a popular tool for many developers and is supported by all .NET IDE vendors. A technique to share functionality across unit tests is building a fixture. A fixture is a shared context between tests. This feature can help reduce the cost of expensive dependencies such as databases, Docker containers, and more.

I prefer fixtures, but admittedly this approach could forgo fixtures in favor of a static class and static methods. If you like that approach, please modify the code accordingly.

I’ll be using SQLite to store the count for each test execution. You’ll need to install the NuGet package of Microsoft.Data.Sqlite. Let’s take a look at the fixture.

using System.Runtime.CompilerServices;
using Microsoft.Data.Sqlite;

namespace FailCounter;

public class TestRunsCounter
{
    private const string ConnectionString = "Data Source=tests.db";

    public TestRunsCounter()
    {
        using var connection = new SqliteConnection(ConnectionString);
        connection.Open();
        using var command = connection.CreateCommand();
        command.CommandText =
            """
            CREATE TABLE IF NOT EXISTS Counts
            (
                Name  TEXT not null
                    constraint Name primary key,
                Count integer
            );
            """;
        command.ExecuteNonQuery();
    }

    public long Increment([CallerMemberName] string? testName = null)
    {
        if (testName is null)
            return 0;

        using var connection = new SqliteConnection(ConnectionString);
        connection.Open();
        var command = connection.CreateCommand();
        command.CommandText =
            """
            INSERT Into Counts (Name, Count) Values($name, 1)      
            On CONFLICT(Name) Do Update Set
                Count = Count + 1
                Where Name = $name;
            Select Count from Counts Where Name = $name;
            """;
        command.Parameters.AddWithValue("$name", testName);
        var count = (long)command.ExecuteScalar()!;

        return count;
    }

    public void Reset([CallerMemberName] string? testName = null)
    {
        if (testName is null)
            return;

        using var connection = new SqliteConnection(ConnectionString);
        connection.Open();
        var command = connection.CreateCommand();
        command.CommandText =
            """
            INSERT Into Counts (Name, Count) Values($name, 0)      
            On CONFLICT(Name) Do Update Set
                Count = 0
                Where Name = $name;
            """;
        command.Parameters.AddWithValue("$name", testName);
        command.ExecuteNonQuery();
    }
}

The calls to Increment will either increment an existing row’s count or insert a new row with a count of 1. I also included a Reset method for resetting counters during investigations. The Reset method must be called manually by you.

Using The Test Runs Counter Fixture

Utilizing the TestRunsCounter fixture is as straightforward as implementing the IClassFixture interface. Let’s look at an example of using the counter.

using Xunit.Abstractions;

namespace FailCounter;

public class UnitTest1 : IClassFixture<TestRunsCounter>
{
    private readonly TestRunsCounter _fixture;
    private readonly ITestOutputHelper _output;

    public UnitTest1(TestRunsCounter fixture, ITestOutputHelper output)
    {
        _fixture = fixture;
        _output = output;
    }

    [Fact]
    public void Test1()
    {
        var count = _fixture.Increment();
        _output.WriteLine($"Current count is \"{count}\".");

        if (count > 10)
        {
            throw new Exception();
        }

        Assert.True(count < 10);
    }

    [Theory]
    [InlineData(nameof(Test1))]
    public void Reset(string testName)
        => _fixture.Reset(testName);
}

Now we can let JetBrains Rider run our test until failure. We can also use the unit test of Reset to reset our failing test. Great!

Conclusion

Intermittently failing tests are frustrating, but with JetBrains Riders “Run Selected Tests Until Failure”, you can diagnose those pesky situations. By using SQLite, we can track the iterations it took to fail. You can also use this as a starting to serialize other stateful information from a unit test and store them in SQLite.

I hope you enjoyed this post, and let me know how you end up using this code in your code bases. As always, thank you.