THE PROJECT

A short introduction to the project

What is Firebenchmarks ? FireBenchmarks is a NUnit Addin able to record execution time of unit tests and generate XML, CSV, HTML performances reports with charts and history tracking. Its' main purpose is to enable a developer or a team that works using XP and agile methodologies, to increase the software quality integrating performances metrics and analysis into the unit testing environment controlling and monitoring the evolution of a system in terms of algorithmic complexity and system load. driving to a deeper comprehension of code executions flows Here you can find the license conditions . History and background FireBenchmarks was originally developed by Federico "NinjaCross" Coletto, and is now freely distributed by Dogma Solutions. The first version of the code and the related tutorial has been published on The Code Project. This project has seen its first stubs during the development of the CMS engine WebBones that needed, and still currently needs, for constant and accurate performance benchmarking. WebBones' contents database is totally file-system based into xml files, and its whole architecture has become very complex due to the mandatory flexibility of the plugins subsystem. Due to these two factors, the performances mainly depends on: the overall algorithmic complexity the xml IO timings. The algorithmic complexity can be estimated (since it's deterministic by definition) but estimating it in a complex workflow is a pain. The xml IO timings are even more difficult to forecast. These two factors combined together made the performances optimization an hell, and at some point we realized that theory was not enough: We needed for real-case benchmarks. Prerequisites formalization On the features side, to tweak performances and evaluate possible optimization strategies is needed a tool able to: measure the execution timings of every high-level and mid-level functionality in the system every time I make a meaningful modifications in the code base keep a chronological tracking of every measurement taken, and visually compare the measurements in a way that makes me able to determine if a certain modification improves the system performances or not. Going more in deep with the prerequisites analysis, other features spontaneously emerge on the surface, and the final prerequisites asset needs the tool to be also: money and license free fully customizable in terms of benchmarks visual layout fully customizable in terms of benchmarks parametrization not intrusive in native code or unit tests code "smart" enough to automatically suggest ways to improve the code.  The solution Since all the project code is covered by NUnit tests, we decided to search for a tool able to integrate into NUnit and execute the performances benchmarks in the easy-to-manage shape of unit tests . Unfortunately, no one of the tools we tested satisfied all the requirements we needed, so we decided to develop a tool on my own based on the extensibility features embedded in NUnit . After some documentations and samples digging, we decided to develop our new Addin so it would: be driven by the smallest possible chunk of code into the test method to benchmark. We decided to follow the general NUnit extension strategy, developing a brand new attribute ( NJC_TestPerformanceRecorderAttribute ) able to instruct the Addin without affecting the code inside the test and so without touching its execution flow. track the test execution info with a descriptor ( NJC_TestPerformanceDescriptor ) initialized through the parameters declared in the tracking attribute NJC_TestPerformanceRecorderAttribute   connect to the NUnit infrastructure through the EventListener interface  Makes the architecture extensible enough to make us able to generate useful data reports reusable in external environments. The current implementation supports 3 types of output report formats:  Csv:  Write a CSV file, where every row is a single benchmark with the description values written in is columns  Xml: Write an XML file, where every child node of the root is a single benchmark Html: Writes both an XML file and an HTML file generated by the transformation of the XML file with the embedded. The HTML report contains graphs and visual aids that helps reading the benchmarks results and taking further decisions about tweaks to adopt.