Traceability between code and requirements in .NET with Azure DevOps
In safety-critical software development within regulated fields like medical devices, there is often a stringent requirement for formal proof of the application’s correct operation. This article will illustrate a straightforward method for programmatically establishing a connection between tests and requirements, along with automated checks to assist developers in ensuring that their code aligns with these requirements using .NET and Azure DevOps.
During development, we identify risks, from which we derive requirements to mitigate these risks and ensure the product’s safety. To verify these requirements, we create a set of test cases aimed at demonstrating the functionality of the design. Requirements may exist at different levels, with user requirements typically serving as the basis for more formal system requirements. Some requirements can also be decomposed into module or component-level specifications.
In the standard development process, we often conduct system-level tests to validate the system requirements, integration tests to assess module-level functionality, and finally, unit tests to confirm compliance with component requirements. The exact distinction between system-level tests and integration tests is not always evident and may not be necessary to clarify.
The connection between tests and the requirements they verify is typically illustrated in a Requirements Traceability Matrix. More often than not, this matrix is created as a basic Excel table where developers and requirements specialists must manually establish and maintain the links between tests and requirements. Ensuring the constant accuracy of this document can be challenging, as is understanding the impact on requirements when you make changes to your implementation. While there are tools available for this purpose, integrating them directly into the code can sometimes pose difficulties. This article will introduce a straightforward alternative for those working with .NET and Azure DevOps.
So lets make an example. We have a product that interacts with patients in various ways. We’ve identified a safety risk wherein only patients whose vital signs consistently meet or exceed a specific threshold should use our product. To address this concern, we have formulated a requirement similar to the following:
In this project we have chosen to use Azure DevOps, were we can keep our requirements as work items which can be linked to our code and tests.
The implementation to satisfy this requirement could look something like this:
public class Patient
{
private readonly int vitalSign;
public Patient(int vitalSign)
{
this.vitalSign = vitalSign;
}
public bool CanUseDevice()
{
return vitalSign > 80;
}
}Testing this simple implementation is straight forward:
public class PatientTests
{
[Theory]
[InlineData(80, false)]
[InlineData(10, false)]
[InlineData(81, true)]
public void OnlyPatientsAboveACertainThreshold(int vitalSign, bool expectedResult)
{
var sut = new Patient(vitalSign);
var result = sut.CanUseDevice();
result.Should().Be(expectedResult);
}
}To establish a connection between our requirement and its implementation, we create a custom attribute that we can employ in our tests to verify the requirement:
[AttributeUsage(AttributeTargets.Method, AllowMultiple = false)]
public class Requirement : Attribute
{
public int Number;
public Requirement(int number)
{
Number = number;
}
}And then update our test method accordingly:
public class PatientTests
{
[Theory]
[Requirement(3038)] // Added the new attribute!
[InlineData(80, false)]
[InlineData(10, false)]
[InlineData(81, true)]
public void OnlyPatientsAboveACertainThreshold(int vitalSign, bool expectedResult)
{
var sut = new Patient(vitalSign);
var result = sut.CanUseDevice();
result.Should().Be(expectedResult);
}
}So how can this attribute to help us improve our traceability of requirements?
Lets start by creating a simple class which can scan our assemblies for the newly created attribute using a bit of reflection:
public class RequirementAssemblyScanner
{
public Dictionary<int, List<string>> GetRequirementWithAssociatedTestCases(Predicate<string> filter)
{
var dict = new Dictionary<int, List<string>>();
foreach (Type type in AppDomain.CurrentDomain.GetAssemblies().SelectMany(x => x.GetTypes().Where(y => filter(y.Name))))
{
var classMethods = type.GetMethods().Where(x => x.GetCustomAttribute(typeof(Requirement)) != null);
foreach (var method in classMethods)
{
var requirementAttribute = method.GetCustomAttribute<Requirement>();
var testAndClassName = type.Name + "_" + method.Name;
AddOrUpdate(dict, requirementAttribute.Number, testAndClassName);
}
}
return dict;
}
private static void AddOrUpdate(Dictionary<int, List<string>> dict, int requirementNumber, string testAndClassName)
{
if (dict.TryGetValue(requirementNumber, out var methodNames))
{
methodNames.Add(testAndClassName);
}
else
{
dict.Add(requirementNumber, new List<string> { testAndClassName });
}
}
}We store all requirement numbers, along with the corresponding class and method in which they are tested. Given that multiple tests can be associated with the same requirement, we store them in a list. The filter parameter allows us to selectively search for the attribute in specific files, which can be useful if, for instance, we wish to restrict the search to files modified by a pull request (PR) rather than scanning the entire repository.
We can then retrieve all requirements from Azure directly in our app using these packages from Microsoft: (Nuget #1) (Nuget #2) (NuGet #3)
public class RequirementAzureClient
{
private readonly VssConnection connection;
public RequirementAzureClient(string orgUrl, string pat)
{
connection = new VssConnection(new Uri(orgUrl), new VssBasicCredential(string.Empty, pat));
}
public async Task<IEnumerable<WorkItem>> GetRequirementsFromAzure()
{
var witClient = connection.GetClient<WorkItemTrackingHttpClient>();
Wiql wiql = new()
{
Query = "Select [State], [Title] " +
"From WorkItems " +
"Where [Work Item Type] = 'Requirement' " +
"Order By [State] Asc, [Changed Date] Desc"
};
WorkItemQueryResult queryResult = await witClient.QueryByWiqlAsync(wiql);
int[] workItemIds = queryResult.WorkItems.Select(wif => { return wif.Id; }).ToArray();
string[] fieldNames = new string[] {
"System.Id",
"System.Title",
"System.WorkItemType",
"System.Description"
};
IEnumerable<WorkItem> workItems = await witClient.GetWorkItemsAsync(workItemIds, fields: fieldNames);
return workItems;
}
public async Task CreateCommentOnPullRequest(string content, int pullRequestId)
{
var gitClient = connection.GetClient<GitHttpClient>();
var pullrequest = await gitClient.GetPullRequestByIdAsync(pullRequestId);
Comment comment = new() { Author = new IdentityRef() { DisplayName = "RequirementsBot", UniqueName = "RequirementsBot" }, CommentType = CommentType.Text, Content = content, };
var commentThread = new GitPullRequestCommentThread() { Comments = new List<Comment>() { comment }, Status = CommentThreadStatus.Active };
await gitClient.CreateThreadAsync(commentThread, pullrequest.Repository.Id, pullRequestId);
}We can then make a comparison to see if there are any requirements which are not under tests:
public static async Task Main(string[] args)
{
var orgUrl = args[0]; // e.g. www.dev.azure.com/MyOrg
var pat = args[1]; // Personal access token
var pullRequestId = int.Parse(args[2]);
var requirementsFromCode = new RequirementAssemblyScanner().GetRequirementWithAssociatedTestCases(x => true);
var azureRequirementsClient = new RequirementAzureClient(orgUrl, pat);
var requirementsFromAzure = await azureRequirementsClient.GetRequirementsFromAzure();
var requirementsWithoutAnyTests = requirementsFromAzure.Where(x => !requirementsFromCode.ContainsKey(x.Id.Value));
foreach (var requirement in requirementsWithoutAnyTests)
{
await azureRequirementsClient.UpdatePRWithUntestedRequirementsComment(requirement, pullRequestId);
}
}Creating a pipeline is straight forward, we just need to run our application with some arguments specifying where and how to access the code:
trigger:
- none
pool:
vmImage: 'windows-latest'
steps:
- task: DotNetCoreCLI@2
displayName: Run RequirementsClient
inputs:
command: run
projects: requirementsClient\requirementsClient.csproj
arguments: https://dev.azure.com/MyOrganization/ $(System.AccessToken) $(System.PullRequestId)The result could look something like this:
This can be useful in later stages of a project where we want to make sure that all requirements are tested before a release. But as a developer, I would be frustrated if I would get messages that there are 100 untested requirements in every pull request that I create.
For day-to-day development, our primary focus is likely to ensure that our changes in a pull request do not break any existing tests linked to requirements, or that we want to add new test cases and establish the necessary links.
Within our Azure pipelines, we can identify the modified files in a PR using a straightforward bash script and store them in a file for our program to later read:
git diff --name-only origin/master > tmp.txt
## Example tmp.txt:
src/Patient.cs
src/AnotherModifiedFile.cs
tests/PatientTests.cs
tests/AnotherTestFile.csBy utilizing the filter parameter within our RequirementsAssemblyScanner class, we can develop a small program that serves as a reminder to both the PR creator and the reviewer. Its purpose is to prompt them to verify whether the tests remain adequate for confirming compliance with the requirements.
public static async Task Main(string[] args)
{
var orgUrl = args[0]; // e.g. www.dev.azure.com/MyOrg
var pat = args[1]; // Personal access token
var pullRequestId = int.Parse(args[2]);
var modifiedFiles = ReadFilesNamesFromTmpTxt();
Predicate<string> filter = x => modifiedFiles.Contains(x);
var requirementsFromCode = new RequirementAssemblyScanner().GetRequirementWithAssociatedTestCases(filter);
var azureRequirementsClient = new RequirementAzureClient(orgUrl, pat);
var requirementsFromAzure = await azureRequirementsClient.GetRequirementsFromAzure();
var requirementsFromTests = requirementsFromAzure.Where(x => requirementsFromCode.ContainsKey(x.Id.Value));
foreach (var requirement in requirementsFromTests)
{
var testedInMethods = requirementsFromCode[requirement.Id.Value];
await azureRequirementsClient.UpdatePRWithRequirementsComment(requirement, testedInMethods, pullRequestId);
}
}This could look sometime like this:
In this article, we have demonstrated a method that involves creating a straightforward program with a touch of reflection magic to establish a linkage between a requirement and its corresponding verifying test. This approach can be employed to implement automated checks in projects, ensuring that all requirements are thoroughly tested. Additionally, it can be utilized to instate a procedural reminder for pull request reviewers to verify the ongoing fulfillment of our safety-critical requirements