The obvious issue with executing the most up to date unit test assemblies is some new features in 3.6.1 won't work on a 3.6.0 database. e.g. assuming a new calendar table was included in a feature in 3.6.1, any tests which required this table would fail if run on databases less than 3.6.1. A calendar class may typically exist in a common assembly which contains many other types which work in previous versions. There are essentially three types of modiciations we could be making to unit test assemblies in each new release
- Addition of schema which wouldn't work in previous versions e.g. using new table / sps / UDFs
- Updates to existing unit tests. As we adapt existing code we may have to change tests in some cases slightly, although this should be quite rare
- Addition of new tests for old schemas e.g. we may find some additional scenarios which haven't previously been tested.
2 and 3 obviously don't pose any issues however when executing unit tests against old databases we clearly need to conditionally execute the tests in 1 only if the schema is valid. A solution to this could be to apply an optional attribute against any test method called something like ApplicationVersion which would take the version number of the schema the unit test can be executed on. The attribute could be applied on a class or method. If applied on a class all tests within the class would require this version number unless any test methods overrode this by specifying a different number. e.g. a CalendarTest class could be defined like this
[TestFixture]
[ApplicationVersion(3,6,1,0)]
public class CalendarTest
However I thought whilst the attribute and use itself seemed quite generic, the actual implementation to retrieve the version number would not be. There are two ways in this case NUnit could be modified to incorporate this
- Use a plug in framework within NUnit. The plug in could have one method which allows the method to determine whether an actual method is executed or simply ignored. A Bottom line plug in could simply have knowledge about the custom attribute, read it using reflection at runtime, and then verify the version number against that stored in the database and decide whether or not to ignore the method.
- Have specific knowledge of the ApplicationVersion attribute and perhaps allow a simple piece of configured embedded C# script which could evaluate what the application number would be at runtime e.g. for us we could read it from the data store from the a table which contains the schema version.
If tests were ignored, the ignore reason could be added automatically e.g. "Test method ignored as version executing against is 3.6.0 and this method requires 3.6.1".
This feature would certainly be useful for us as we have a large number of clients using old versions and only a few at a time initially using the most up to date versions. I'm sure the solution would be useful to others too, thoughts?
3 comments:
NUnit has generally tried to avoid any sorts of hooks that are specific to a certain type of application - such as database testing. That's why, for example, Roy Osherove's Rollback facility was written as an extension to NUnit, rather than being contributed to the core code.
I can see several more generic ways to do this:
1) Depending on how you run these tests, you may be able to use a CategoryAttribute to select the tests to run. That only works if you run the entire suite of tests against a single database schema at once, rather than switching schemas as you run. [ If you can use this, it's free. ]
2) Using the PropertyAttribute to set a particular property like ApplicationVersion on the test. Your tests (or setup) would contain code to reflect on the properties and decide whether or not to run. That code could be in a base class and NUnit could even supply some helpers. [If you write the reflection code, you can use this with the current NUnit 2.4 Beta]
3) As a parameterized test case, with the test code simply skipping the test if it didn't like the parameter. [Parameterized tests are a proposed feature, not yet implemented.]
4) As an extension to the PlatformAttribute - some way of allowing you to extend the set of strings that are recognized by Platform. [This isn't even proposed - I just thought of it.]
If I were doing it, I'd look at the options in the order I listed them.
Charlie
The category and property attribute seem very useful however I don't think they would work for us here i.e. you could imagine we'd have a number of tests like so
[Test, Category("3.6.1.0")]
void A();
[Test, Category("3.6.2.0")]
void B();
[Test, Category("3.6.3.0")]
void C();
[Test]
void D();
I've shown a table of the expected tests we'd want run for each app version.
DB Version Tests to execute
3.6.0 D
3.6.1 A, D
3.6.2 A, B, D
3.6.3 A, B, C, D
I.e. if the category attribute for a version number is not specified we'd like to execute the test regardless of the version executed on. If it states 3.6.2.0 then any test cases less than or equal to this version should be included AND any test cases without an explicit version specified e.g. A, B and D here.
I can think of one way we could achieve this quite easily in the existing code e.g.
[Test, Category("3.6.2.0")]
void B()
{
VerfiyVersion();
}
The VerifyVersion function can simply reflect on the current method to find the version number in the attribute and then look up the version in the current database the test is executing, and determine whether to execute or ignore the test using methods such as Assert.Ignore. However if we did this by adding the VerifyVersion call we might as well add a stronger typed attribute which could convery more meaning e.g.
[Test, ApplicationVersion("3.6.2.0")]
void B()
{
VerfiyVersion();
}
We could then extend the attribute in the future e.g. to support only running the tests when executed between specific ranges of versions e.g. 3.6.2.0 - 3.6.4.0..
[Test, ApplicationVersion("3.6.2.0", "3.6.4.0")]
void B()
{
VerifyVersion();
}
It does strike me that this test execution based on application version seems quite a useful addition to incorporate into the nunit framework though. With a customisation, extension point to allow the user to define how to return the application version.
To use Category, you could do wone of two things...
1) Specify a single category on each test, as shown in your example, but select multiple categories at runtime. So, for testing version 3.6.2, you would select categories 3.6.0, 3.6.1 and 3.6.2.
2) Specify multiple categories on each test, like this...
[Test, Category("3.6.1"), Category("3.6.2"]
void B();
Category is pretty flexible and has the advantage that it makes no assumptions as to the semantics of the category.
I still think you should look at PropertyAttribute. It too is very generic, but we plan to provide a way to derive new Attrributes from PropertyAttribute, where the name of the new attribute matches the property. So...
[Property( Name="ApplicationVersion", Value="3.6.1")]
could be replaced with
[ApplicationVersion( "3.6.1")]
But note that this doesn't work yet because we identify properties by their name, not by type. Anyway, our properties are currently sealed.
Maybe we need to do something similar with CategoryAttribute to let you define categories that implement slightly different semantics. After all, this seems to be a Category: an attribute applied for the purpose of deciding whether to run the test.
BTW, I wonder if you could think of a better name for your proposed attribute? It seems to say "run this using version X of the software" but NUnit doesn't actually have any control over what version of the software you are using. However, others may read it differently, so I suggest asking a number of people.
All in all, I think you're on to something here, but I'd still like to find the more general concept of which it's an instance, so we can avoid proliferating concepts needed to understand NUnit.
Charlie
Post a Comment