|
 Introduction |
FireBenchmarks is a
NUnit
Addin able to record execution time of unit tests and generate XML, CSV, HTML performances reports with charts and history tracking.
Its' main purpose is to enable a developer or a team to integrate performances metrics and analysis into the unit testing environment, to easily control and monitor the evolution of a system in terms of algorithmic complexity and system load.
It's based on
NUnit
, one of the best and free unit-testing frameworks for .NET |
|
| Assumptions and basic principles |
In a XP-oriented project, the code is covered by
NUnit
tests, so the best solution to integrate performance metrics and analysis, is to use a tool able to directly work into
NUnit
and execute the performances benchmarks in the easy-to-manage shape of
unit tests
.
FireBenchmarks is so developed as a NUnit
Addin
able to:
- be driven by the smallest possible chunk of code into the test method to benchmark. We decided to follow the general
NUnit
extension strategy, developing a brand new attribute (
NJC_TestPerformanceRecorderAttribute
) able to instruct the
Addin
without affecting the code inside the test and so without touching its execution flow.
- track the test execution info with a descriptor (
NJC_TestPerformanceDescriptor
) initialized through the parameters declared in the tracking attribute
NJC_TestPerformanceRecorderAttribute
- connect to the
NUnit
infrastructure through the
EventListener
interface
- Makes the architecture extensible enough to make us able to generate useful data reports reusable in external environments. The current implementation supports 3 types of output report formats:
-
Csv:
Write a CSV file, where every row is a single benchmark with the description values written in is columns
-
Xml:
Write an XML file, where every child node of the root is a single benchmark
-
Html:
Writes both an XML file and an HTML file generated by the transformation of the XML file with the embedded. The HTML report contains graphs and visual aids that helps reading the benchmarks results and taking further decisions about tweaks to adopt.
|
|
| Global architecture |
Here below there is a simplified diagram that shows the global architecture of the AddIn.
As stated above, the main components are the main AddIn class (
NJC_TestPerformanceAddIn
), the execution descriptor class (
NJC_TestPerformanceDescriptor
) and the test attribute class (
NJC_TestPerformanceRecorderAttribute
)
|
|
The attribute NJC_TestPerformanceRecorderAttribute |
The attribute used to define the behaviour of the benchmarking workflow has a really simple structure, so let's jump directly to it's usage demonstration and the meaning of every parameter:
[Test]
// Mandatory declaration of the attribute to makes the Addin able to recognize it as a benchmark test
[NJC_TestPerformanceRecorder(
// Optional. The logic/descriptive name associated to the test
// Default value = ""
TestName = "Test 1",
// Optional. a desciption of the test
// Default value = ""
TestDescription = "Test description",
// Optional. The output of the benchmark must overwrite (Overwrite) the previous trackings, or must be appended to them (Append) ?
// Default value = NJC_TestPerformanceTargetWriteMode.Append
OutputTargetWriteMode = NJC_TestPerformanceTargetWriteMode.Append,
// Optional. Where will be written the output ? By now, the only possibile output value is "FileSystem"
// Default value = NJC_TestPerformanceTargetKind.FileSystem
OutputTargetKind = NJC_TestPerformanceTargetKind.FileSystem,
// Optional Flag. Which is the output format of the benchmark ? By now there are 3 values that can be combined together:
// Csv -> Write a CSV output, where every row is a single benchmark with the description values written in its columns
// Xml -> Write an XML output, where every child node of the root is a single benchmark
// HTML -> Writes both an XML output and an HTML output generated by the transformation of the XML output with the embedded
// transformation style sheet NJC_TestPerformanceDescriptorWriterHTML.xsl
// Default value = NJC_TestPerformanceTargetFormat.Xml
OutputTargetFormat = NJC_TestPerformanceTargetFormat.Csv | NJC_TestPerformanceTargetFormat.Xml,
// Optional. The storage location of the benchmark output.
// Since by now the only supported OutputTargetKind is "FileSystem", the OutputTarget value represents the target folder
// in which the output files are written.
// Default value = the test assembly folder
OutputTarget = TARGET_FOLDER,
// Optional. Every benchmarked test is identified by a unique key.
// With the OutputTargetKind set to FileSystem, this identification key corresponds to the file name where the output
// is stored with the OutputTargetWriteMode.
// Default value = NJC_TestPerformanceTargetIdentificationFormat.ClassFullNameAndMethodName
OutputTargetIdentificationFormat = NJC_TestPerformanceTargetIdentificationFormat.ClassFullNameAndMethodName
)]
public void My_Method_Test_That_Needs_For_Benchmarking()
{
/* place here the code to test AND benchmark*/
}
Here there is a representation of the internal structure of the NJC_TestPerformanceRecorderAttribute attribute with the enumeration associated to the configuration properties
 |
|
The descriptor NJC_TestPerformanceDescriptor |
The performance descriptor implemented in the class NJC_TestPerformanceDescriptor is just an information and parameters holder, which only functionality is to represent the timing and execution state of a NJC_TestPerformanceRecorderAttribute applied to a test method.
Due to this motivations all its properties and private members are exposed to the public environment as read-only.
The only exception is the EndTime property, that's used to implements the time-tracking functionality of the instance, exposed by the readonly property ExecutionTime.
 |
|
The EventListenter interface |
The EventListener interface is very useful because it makes custom code able to transparently connect to the NUnit environment events in an asynchronous way: every method in the interface implementor is called by the framework without waiting its execution to end.
It's definition is (in NUnit 2.4.4) declared as follows:
using System;
namespace NUnit.Core
{
public interface EventListener
{
void RunFinished(Exception exception);
void RunFinished(TestResult result);
void RunStarted(string name, int testCount);
void SuiteFinished(TestResult result);
void SuiteStarted(TestName testName);
void TestFinished(TestResult result);
void TestOutput(TestOutput testOutput);
void TestStarted(TestName testName);
void UnhandledException(Exception exception);
}
}
For simplicity and clearness in code writing, I created the core of the NJC_TestPerformanceAddIn directly implementing the IAddin interface (needed by NUnit to recognize the class as an Addin) and the EventListener interface (that drives the connection implementation of the Addin into the NUnit infrastructure)
///
/// This is the main Addin class
///
[NUnitAddin(Name = "NinjaCross Test Performance Extension")]
public sealed class NJC_TestPerformanceAddIn : IAddin, EventListener
{
...
}
|
|
The AddIn NJC_TestPerformanceAddIn and its workflow |
The workflow of the Addin is controlled by the sequential invocation of these methods implementations:
Step 1) NJC_TestPerformanceAddIn.RunStarted and/or NJC_TestPerformanceAddIn.SuiteStarted
load the tests executing assembly
public void RunStarted(string name, int testCount)
{
if (Debugger.IsAttached)
Log("RunStarted", name, testCount);
EnsureCacheAssembly(name);
}
//----------------------------------------------------------------------------
public void SuiteStarted(TestName testName)
{
if (Debugger.IsAttached)
Log("SuiteStarted", testName.FullName, testName.Name, testName.RunnerID, testName.TestID, testName.UniqueName);
EnsureCacheAssembly(testName.FullName);
}
//----------------------------------------------------------------------------
private Assembly EnsureCacheAssembly(String fullName)
{
try
{
Assembly a = GetCachedAssembly(fullName);
if (a != null)
return a;
if (fullName.EndsWith(".dll", StringComparison.CurrentCultureIgnoreCase) || fullName.EndsWith(".exe", StringComparison.CurrentCultureIgnoreCase))
{
Assembly asm = Assembly.LoadFile(fullName);
_assembliesCache.Add(asm);
return asm;
}
else
{
return null;
}
}
catch (Exception exc)
{
Log("AddSession", exc);
return null;
}
}
Step 2) NJC_TestPerformanceAddIn.TestStarted
Check the existence of the NJC_TestPerformanceRecorderAttribute attribute into the test method decorations.
If it exists, the method is a suitable benchmarking candidate, so acquires the description and output information/parameters defined in the attribute, and store them into an execution descriptor of type NJC_TestPerformanceDescriptor.
The execution start time is set into NJC_TestPerformanceDescriptor.StartTime using the DateTime.Now value.
public void TestStarted(TestName testName)
{
if (Debugger.IsAttached)
Log("TestStarted", testName.FullName, testName.Name, testName.RunnerID, testName.TestID, testName.UniqueName);
MethodInfo method = NJC_TestPerformanceUtils.GetMethodInfo(testName.FullName, _assembliesCache);
if (method != null)
{
if (NJC_TestPerformanceUtils.GetPerformanceAttribute(method) != null)
{
// generate a descriptor for the current test method
NJC_TestPerformanceDescriptor descriptor = new NJC_TestPerformanceDescriptor(method);
// cache the generated descriptor
_runningTests.Add(descriptor);
}
}
}
Step 3) NJC_TestPerformanceAddIn.TestFinished
Stop the timer for the currently running test setting NJC_TestPerformanceDescriptor.EndTime to DateTime.Now, and write the output in the way specified by the NJC_TestPerformanceRecorderAttribute, that was stored inside the execution descriptor NJC_TestPerformanceDescriptor.
The descriptor serialization is delegated to a specific set of classes, also known as "writers" (see next paragraph)
public void TestFinished(TestResult result)
{
if (Debugger.IsAttached)
Log("TestFinished", result.FullName, result.HasResults, result.AssertCount, result.Description, result.Executed, result.FailureSite);
if (result.Executed)
{
// resolve the test method using its full name
MethodInfo method = NJC_TestPerformanceUtils.GetMethodInfo(result.FullName, _assembliesCache);
if (method != null)
{
// get the descriptor from the cached running tests
NJC_TestPerformanceDescriptor descriptor = _runningTests.GetItem(method);
// if a descriptor is found for the reflected method, ands the time measurement procedure
if (descriptor != null)
{
// end the time measurement, setting the EndTime property to DateTime.Now
descriptor.EndTime = DateTime.Now;
// loops inside the list of avaiable writers
foreach (KeyValuePair writerKVP in _writers)
{
// check if the given format is supported by the current writer
if ((descriptor.TestMethodAttribute.OutputTargetFormat & writerKVP.Key) == writerKVP.Key)
{
// serialize the descriptor into the suitable target using the selected writer
writerKVP.Value.Write(descriptor);
}
}
// the method terminated it's execution, so it's not more needed into the cache
_runningTests.Remove(method);
}
}
}
}
|
|
| Output writers |
As previously stated, the last step of the workflow is the storage of the output information related to the test, in the way specified by the NJC_TestPerformanceRecorder attribute parameters.
To do this, every test-execution descriptor (NJC_TestPerformanceDescriptor) is serialized into a target file using a specific "writer" that implements the suitable business logic to serialize the descriptor data in the appropriate format.
The Addin selects the suitable writer to use looking at the value of the NJC_TestPerformanceRecorder.OutputTargetFormat (of type NJC_TestPerformanceTargetFormat).
Every value of the enumeration NJC_TestPerformanceTargetFormat is mapped one-on-one in a "writers list" inside the NJC_TestPerformanceAddIn class.
The writers list is declared as class member...
///
/// Writers avaiable into this addin implementation
///
private NJC_TestPerformanceDescriptorWriterList _writers = new NJC_TestPerformanceDescriptorWriterList();
... and is initialized in the constructor with an instance for every avaiable writer.
public NJC_TestPerformanceAddIn()
{
_writers.Add(NJC_TestPerformanceTargetFormat.Xml, new NJC_TestPerformanceDescriptorWriterXml());
_writers.Add(NJC_TestPerformanceTargetFormat.Csv, new NJC_TestPerformanceDescriptorWriterCsv());
_writers.Add(NJC_TestPerformanceTargetFormat.Html, new NJC_TestPerformanceDescriptorWriterHTML());
}
Every writer extends the base class NJC_TestPerformanceDescriptorWriter
public abstract class NJC_TestPerformanceDescriptorWriter
{
///
/// Write to the destination target
///
public abstract void Write(NJC_TestPerformanceDescriptor descriptor);
public abstract String FileExtension
{
get;
}
}
Here there is a representation of the inheritance tree of the supported/mentioned writers 
As already mentioned, the available output formats are 3 (one for each writer extended from NJC_TestPerformanceDescriptorWriter) and they can be generated only into the file system (OutputTargetKind = NJC_TestPerformanceTargetKind.FileSystem).
The output file names are composed using the pattern
<output_folder>\<file_name>.<file_extension>
where:
<output_folder> = NJC_TestPerformanceRecorder.OutputTarget value, or the test assembly folder if no value was specified
<file_name> = A name based on the format specified in NJC_TestPerformanceRecorder.OutputTargetIdentificationFormat. The available values are MethodName, ClassNameAndMethodName, ClassFullNameAndMethodName.
<file_extension> = A file extension based on the NJC_TestPerformanceRecorder.OutputTargetFormat flag values- "csv" for
NJC_TestPerformanceTargetFormat.Csv - "xml" for
NJC_TestPerformanceTargetFormat.Xml - "html" for
NJC_TestPerformanceTargetFormat.Html. For this working mode is generated also an "xml" file, that's used as an incremental repository for the NJC_TestPerformanceTargetWriteMode.Append "mode" of the NJC_TestPerformanceTargetFormat.Html option.
Including the writers classes into the global diagram, the resulting "big picture" is as follows:
|