Firebenchmarks - NUnit driven performance testing

HOME

Feeds & tools

Sponsors & Partners

Output samples
Here below there are some samples generated by the AddIn.
by sa (created on the 05/07/2009 02:05)
Table of contents
CSV file
XML file
HTML file
CSV file
The first row contains the columns header, the following rows contains the test executions data.
start;startTicks;end;endTicks;elapsed;elapsedTicks
lun, 06 apr 2009 13:20:49 GMT;633746208497456374;lun, 06 apr 2009 13:20:50 GMT;633746208500267866;00:00:00.2811492;2811492
lun, 06 apr 2009 13:21:14 GMT;633746208741743790;lun, 06 apr 2009 13:21:14 GMT;633746208744555282;00:00:00.2811492;2811492
lun, 06 apr 2009 13:21:20 GMT;633746208806408106;lun, 06 apr 2009 13:21:20 GMT;633746208809688180;00:00:00.3280074;3280074
lun, 06 apr 2009 13:21:27 GMT;633746208878569734;lun, 06 apr 2009 13:21:28 GMT;633746208881693614;00:00:00.3123880;3123880
XML file
The node testExecutionHistorycontains a set of testExecutionchildren nodes.
Every child node represent a test execution.
The testExecutionnodes contains dates and time spans in both in RFC1123 format (see http://www.isi.edu/in-notes/rfc1123.txt or http://www.faqs.org/rfcs/rfc1123.html ) and ticks count (where a tick is one hundred nanoseconds or one ten-millionth of a second, as stated in MSDN documentation )
<?xml version="1.0" encoding="UTF-8"?>
<testExecutionHistory 
ns="NinjaCross.Classes.Nunit.TestPerformance" 
className="NJC_TestPerformanceRecorderAttributeTest" 
methodName="Test_Success_WriteModeAppend_TargetKindFileSystem_TargetFormatCsvXml" 
testName="Test 1" 
testDescription="Test description">
    <testExecution>
        <start>lun, 06 apr 2009 13:20:50 GMT</start>
        <startTicks>633746208500424060</startTicks>
        <end>lun, 06 apr 2009 13:20:50 GMT</end>
        <endTicks>633746208504328910</endTicks>
        <elapsed>00:00:00.3904850</elapsed>
        <elapsedTicks>3904850</elapsedTicks>
    </testExecution>
    <testExecution>
        <start>lun, 06 apr 2009 13:21:14 GMT</start>
        <startTicks>633746208744711476</startTicks>
        <end>lun, 06 apr 2009 13:21:14 GMT</end>
        <endTicks>633746208748460132</endTicks>
        <elapsed>00:00:00.3748656</elapsed>
        <elapsedTicks>3748656</elapsedTicks>
    </testExecution>
    <testExecution>
        <start>lun, 06 apr 2009 13:21:20 GMT</start>
        <startTicks>633746208809844374</startTicks>
        <end>lun, 06 apr 2009 13:21:21 GMT</end>
        <endTicks>633746208812655866</endTicks>
        <elapsed>00:00:00.2811492</elapsed>
        <elapsedTicks>2811492</elapsedTicks>
    </testExecution>
    <testExecution>
        <start>lun, 06 apr 2009 13:21:28 GMT</start>
        <startTicks>633746208881849808</startTicks>
        <end>lun, 06 apr 2009 13:21:28 GMT</end>
        <endTicks>633746208885442270</endTicks>
        <elapsed>00:00:00.3592462</elapsed>
        <elapsedTicks>3592462</elapsedTicks>
    </testExecution>
</testExecutionHistory>
HTML file
This is the most important and interesting report format.
It contains every available information related to the test executions in terms of raw data and in terms of data analysis/trend.
Using a grid-based graphical/data representation of the execution time, coupled with a line-chart generated with Google Chart API is possible to visually evaluate the performances trends of the tracked method along it's development lifetime.
Here there are some explanations about the visual features:
Test identification and info
The "name" field inside the "Test identification and info" appears only if the NJC_TestPerformanceRecorderAttribute.TestName is given in the attribute declaration
The "description" field inside the "Test identification and info" appears only if the NJC_TestPerformanceRecorderAttribute.TestDescription is given in the attribute declaration
If the XSL stylesheet detect that there's at least one record with elapsedTicks==0, the report shows a warning message that states that the benchmark is not useful, since the timer cannot measure meaningful time spans.
Test stats and tests performances tracking boxes
The red bars represents the executions with worst performances.
The green bars represents the executions with best performances.
The yellow bars represents the executions with performances that are not classified as "best" nor "worst".
Test performances evolution
If the XSL stylesheet detect that there's only one record for a given test, the Google lines chart is not rendered into the report since it would not be meaningfull.
The lines chart shows also min, max and average values for execution times, so the working range and characteristic behaviour are clearly visible and ponderable.
The x axis represents the tests execution dimension. At the origin there is the first execution date and on the right side there is the last execution date. The dates are shown in RFC1123 format
The y axis represent the execution time expressed in seconds.
Both axis auto-size themselves to contain the last 100 executions of the test