How to set up a testing workflow for a .NET library in 2024

Building an app? Bitwarden Passwordless.dev is a developer friendly API that allows users to sign in using passkeys with biometrics. Get started for free and see for yourself.

Developing a library involves a lot of moving pieces, and not all of them are about writing code. In a modern software development environment, building, testing, and deploying code efficiently and with as much automation as possible is very important. Setting up such processes is not exactly rocket science, but with the plethora of different options available and the various strategies to consider, it can be easy to get lost in the details. And while there is no one-size-fits-all solution, there are still a few common approaches that can be applied to most projects out there.

Let’s  take a look at the approaches Bitwarden uses to set up a testing workflow for the Passwordless .NET SDK. Hopefully, this will give a decent starting point for creating your own CI/CD workflows, and help you avoid some of the common pitfalls along the way.

First, let's get briefly acquainted with how the solution is organized. Passwordless .NET SDK is a .NET library that allows developers to integrate passwordless authentication into their .NET applications. In essence, it's a tight wrapper around the endpoints provided by the Passwordless Server, with some additional features to make integration with various .NET technologies easier.

The repository structure is pretty typical, as far as .NET projects go, and looks like this:

Here are two library projects, Passwordless and Passwordless.AspNetCore, and the corresponding two test projects, Passwordless.Tests and Passwordless.AspNetCore.Tests. There are also other items, such as the examples directory, which contains sample applications that demonstrate how to use the library, and the Directory.Build.props file, which contains some common configurations for all projects in the solution.

The library projects target multiple frameworks at the same time, most notably .NET Standard 2.0, .NET 4.6.2, and .NET 6.0. This allows them to be used in a wide range of applications, from legacy .NET Framework apps to the more modern .NET Core platforms, while utilizing polyfills to provide a consistent API surface across all of them. The test projects also target a range of different frameworks, to ensure that the corresponding libraries work as expected in all officially supported environments.

To build and test the entire solution, in all available targeting configurations, all we have to do is navigate to the root directory and run the following command:

Behind the scenes, .NET CLI (dotnet) identifies the projects referenced by the solution, restores their dependencies, builds the corresponding run-time artifacts, and then executes the tests for each available target framework. If all tests pass, the command returns the 0 exit code, otherwise it returns 1, indicating an error. In the event that the runtime for one of the target frameworks is not available on the machine, the corresponding tests will be automatically skipped.

Passwordless SDK tests are also particularly involved because they spawn a real instance of the Passwordless Server to perform complete end-to-end behavioral testing. This approach is orchestrated by TestContainers, so we also need Docker installed on the machine in order to run the tests.

While it's nice that this can be done  locally, the goal is to automate this process and run it on every push to the repository — as part of a  continuous integration workflow. This ensures that the code is always in a working state, and that new changes don't introduce any unwanted regressions.

Since GitHub hosts Bitwarden code, it makes sense to also take advantage of GitHub Actions to implement the CI workflow for the repository. GitHub Actions workflows are conceptually based around events — so you can listen to specific types of events that indicate that something happened in the repository, and then run a series of commands in response to that event. While it's completely free for open-source projects, it also comes with a generous monthly allowance of free minutes for private repositories as well.

For a typical testing workflow, it is standard to run dotnet test on every push to the repository, as well as on every pull request. To that end, you can create a workflow file that looks something like this:

Just like that, there is a basic CI workflow that will run dotnet test on every push and pull request. Once this file is committed to the repository, GitHub will automatically detect it and start running the workflow each time the corresponding events occur.

Using GitHub Actions as the CI platform is particularly convenient because it comes with a lot of typical developer workloads preinstalled, including .NET SDK, Docker, and many others. However, it does make sense to specify, at least, the versions of the .NET SDK you want to use, to ensure that the workflow is reproducible across different machines. Simply add the setup-dotnet action to the workflow:

To  develop a library that utilizes platform-specific APIs, or whose behavior otherwise depends on the underlying operating system, it's also a good idea to run the tests on multiple platforms. This can be done very easily in GitHub Actions by specifying a parameterized job via a matrix. For example, here's how to run the tests from before on Windows, Ubuntu, and macOS simultaneously:

So far, this works great, but this workflow is a bit too simple. For one, it doesn't do a very good job at reporting the test results — see the output of the dotnet test command in the workflow logs, it includes a lot of unnecessary information, and it's not particularly easy to navigate. GitHub Actions doesn't provide any built-in functionality to parse .NET test results and display them in a more user-friendly way, but there are a few third-party solutions that can help with that.

One of these solutions is the dorny/test-reporter action. This action can parse test results in a variety of different formats, including .NET's TRX files, and then report them in a separate check run. To use it, configure the dotnet test command to output the results in the TRX format, and then feed the resulting files to the test-reporter action:

Now, when you push a commit to the repository,  the test results can be seen in a separate check run. This allows failing tests to be easily identified, and also provides a convenient way to explore the test suite in general.

However, the issue with dorny/test-reporter is that it relies on GitHub's Check API to render its reports. This API requires the invoker of the workflow to have write access to the repository's checks, which is not always the case. For example, if the above workflow is run as a result of a push event into the main branch, the implicitly provided GITHUB_TOKEN will have the required permissions, but not if it's instead triggered by an outside contributor in a pull_request event.

To work around this issue, split the testing and reporting parts of the pipeline into two separate workflows. The testing workflow will run dotnet test and output the results in the TRX format, and the reporting workflow will then parse the results and publish them as a check run. This way, the reporting workflow will run with a token that has the required permissions, while the testing workflow can run with a token that only has read access to the repository.

Although this works quite well, using two separate workflows for testing and reporting is a bit clunky. Another approach to consider is to rely on the GitHubActionsTestLogger package, as an alternative to dorny/test-reporter. This package provides a custom logger for the dotnet test command, which can publish test results directly to GitHub Actions through the Job Summary API, without the need for elevated permissions.

To use this package, simply install it in the test projects and consolidate the workflows back into the following single file:

With this, the test results should be reported like so:

Finally, no testing workflow is complete without some kind of code coverage reporting. When writing tests, it's beneficial to be able to see which areas of the codebase lack coverage, which conditional branches are not being tested, and which lines of code are not being executed at all. This allows potential areas of improvement to be identified, and ensures the tests are actually doing what they're supposed to.

On the .NET's side, the most popular tool for collecting coverage is Coverlet, which can automatically instrument the assemblies and generate coverage reports in a variety of different formats. It integrates well within the dotnet test pipeline and comes pre-installed when creating a new test project using dotnet new xunit.

Since the coverage reports will be produced on CI, you'll need a place to view them. One option is to upload them as artifacts, of course, but that's not very convenient. Instead, use a third-party service like Coveralls or Codecov to store and visualize the coverage reports. Both of these services are free for open-source projects, have a powerful set of features, and are fairly popular within the .NET community — so it's really just a matter of personal preference which one to use.

For the Passwordless SDK, this example leverages Codecov, because it's a bit more straightforward to use. In order to integrate Codecov with the testing workflow, produce a coverage report in one of the supported formats, and then upload it to the service using the codecov/codecov-action. Here's how that looks:

After that, commit some new changes to the repository to see the corresponding coverage report uploaded on Codecov:

It's important to note that coverage isn't just about the numbers. Services like Codecov show exactly which directories, files, and even lines of code are covered by the tests, and which ones aren't. This can be a great way to identify areas of the codebase that are lacking in test coverage, and to ensure there aren’t any important edge cases missing.

GitHub Actions is a well-tested and secure platform, but it's still important to be mindful of the security implications of the workflows created. For example, when using a third-party action, always make sure that it's coming from a trusted source, and that it's not doing anything malicious. Also be careful about granting permissions to the required actions, and make sure that they don't have access to any sensitive information.

When it comes to permissions, GitHub Actions provides a few different ways to control them. The most common way is to use the permissions parameter of a job, which allows the user to specify exactly which permissions it should have. For example, to set permissions to allow the job to read from the repository, but not write to it, set it up like this:

Coincidentally, that's the only scope of permissions this workflow requires so far. Update it to reflect this, like so:

Now, whatever third-party actions used in this workflow won't be able to perform any write operations on the repository, or read anything that they're not supposed to. To make full use of this, another option is to reduce the default runner permissions to read-only in the repository settings, so that any workflows that don't explicitly require write access will be restricted by default.

While setting up permissions does help limit the threat surface of third-party actions, it doesn't completely eliminate it. Compromised actions can still be used to perform supply-chain attacks, for example by injecting malicious code after checkout, relaying sensitive information to a remote server, or just by slowing down the workflow to waste resources and incur additional costs.

The only way to mitigate these risks is by manually reviewing the actions to make sure they're not doing anything suspicious. Once that's done, an action reference can be pinned to a specific commit hash, to ensure that the underlying code never changes (unlike a tag, which can be moved to point to a different commit). The final workflow would then look like so:

With that, you can be confident the workflow is secure, and that it's not doing anything it's not supposed to.

In the modern world of software development, it's important to have a solid testing workflow in place. This allows users to ensure that code is always in a working state, and new changes don't introduce any unwanted regressions. This is particularly important when working on projects with many collaborators, especially in an open-source environment.

Setting up a testing workflow is not that hard, but it helps to know the right tools and approaches to use. Hopefully, this was a useful read, and you've learned something new today. If you have any questions or feedback, feel free to reach out on Bitwarden Community Forums. Thanks for reading and check out Passwordless.dev to get started with building passwordless authentication into you applications.