|
 I can't compile the sources ! |
Since the addin is based on NUnit, it's functionalities are based on NUnit specific implementation, so you must be sure that into your system there is the correct NUnit version to successfully compile and execute it.
To know the needed NUnit version, refer to the downloads page.
In the same way, you also must be sure that the source codes solution has the correct NUnit reference before building the assembly.
|
|
| I compiled the sources, but I have a post-build error ! |
The sources of this project have a post-build event that copies the output DLL into the default NUnit installation folder (default is "C:\Programmi\NUnit 2.5\bin\net-2.0\addins" for the 2.5 version).
If your NUnit installation is on another path, just change it and recompile the project.
If you want to manually manage the AddIn installation (i.e. to don't interfere with already running instances), just remove the post-build event. |
|
I can execute a test using the NJC_TestPerformanceRecorderAttribute but I can't find the generated reports ! |
Fist of all, you must be sure that the downloaded versions of the codes/binaries of the addin are compatible with the version of NUnit installed into your system.
To check the compatibility of the addin code version versus the NUnit version, refer to the downloads page.
From version 1.1.0.0, FireBenchmarks automatically check the NUnit assembly version, and shows a message if it's not compatible.
If you are sure that you version is compatible with the NUnit version installed into your system, consider the following assertions.
- First of all, ensure that the FireBenchmarks DLL is correctly installed into the
bin\net-2.0\addins folder, like described into the install guide. Then, check in the "Tools->Addins" window if the FireBenchmarks DLL appears in the list of loaded addins.
If it's not there, consider that FireBenchmarks cannot work on NUnit releases different from the one that was targeted at build time, so if you want to target an unsupported version, you have to download the sources and rebuild them against your local NUNit.
- The
NJC_TestPerformanceRecorderAttribute
attribute allows to specify a target directory where to generate the output reports of the executed tests.
The property to use to specify the output location is
OutputTarget
.
By default it's null, so if you don't set it explicitly, the target output folder will be the same as the test's assembly folder.
Examples:
- OutputTarget = "C:\\MyProjects\\MyTests\\PerformanceRepository", // the output will be generated into the "C:\MyProjects\MyTests\PerformanceRepository" folder
- OutputTarget = "", // the output will be generated into the test's assembly folder and is equivalent to don't specify the property at all
- OutputTarget = null, // the output will be generated into the test's assembly folder and is equivalent to don't specify the property at all
|
|
I can compile and execute a test using the NJC_TestPerformanceRecorderAttribute, but I can't find the generated reports ! |
| Same answer of the previous question |
|
| I want to make the addin work with parametrized tests using attributes like Sequential, Range, Repeat, Random |
This question is more complex as it seems at a first look, so it deserves for an adequate explanation.
The expressed need is understandable at an high level analisys, but this feature is not supported, and it probably will not be in the future (by design) due to these motivations:
- The spirit of the FireBenchmarks addin is to keep
simple and crystal clear
the formulation of benchmarking tests.
A test, to be meaningfull,
must
have a very specific, reproducible and consistent execution context.
If it hasn't these features, it's pretty useless because it can't leads the developers to a concrete analysis of performance benchmarks.
This lead to the next point...
- Since the FireBenchmarks addin isn't human, it can't be "intelligent" in the way a human can be, so it can't understand the
semantic
of the execution context originated by the modifications applied to the test by the parametrization attributes.
This means that it's nearly impossibile to automatically create an output report suitable to represent the mentioned execution context.
Example:
Imagine to execute and benchmark a parametrized test that accepts multiple value params in many (hundreds, thousands) combinations and/or sequences.
- How many output report variants should be generated ?
- How should be the recorded timings aggregated and correlated to the params combinations ?
- Would be those final reports really meaningfull for the purpose of the benchmarking ?
The response to all these questions is
"realistically, no"
.
- Even considering the development of this kind of feature, there is a technical difficulty to override.
FireBenchmarks works in a really generic an simple way, using the EventListener interface to know when a test starts and ends.
The NUnit core fires only one time the
EventListener.TestStarted
and
EventListener.TestFinished
methods for each test, even if its body is executed more times using one or more parametrization attributes.
This means that FireBenchmarks never knows about the way the test is executed, and under whitch criteria/params.
Sure, this could be accomplished revolutioning the internal architecture of the addin, so it would operate at a different level of the NUnit system, but the previous motivations suggests that this would be a great effort with really small (or null) income.
|
|
| I upgraded NUnit, and now FireBenchmarks doesn't work anymore ! |
First of all, ensure that the FireBenchmarks DLL is correctly installed into the bin\net-2.0\addins folder, like described into the install guide.
Then, check in the Tools->Addins window if the FireBenchmarks DLL appears in the list of loaded addins.
If it's not there, consider that FireBenchmarks cannot work on NUnit releases different from the one that was targeted at build time, so if you want to target an unsupported version, you have to download the sources and rebuild them against your local NUNit. |
|
| No reports are generated running the tests using TestDriven.NET, Resharper or another hosting application/runner ! |
First of all, check that the hosting application/runner is using the correct NUNit version as stated into the install prerequisites.
If the prerequisites are matched and it still doesn't generate reports, maybe it's an issue related to the application/runner architecture. The runner is not NUnit, it just use its core, so maybe it doesn't load the addins.
In some cases this issue can be resolved using
this workaround
|