The obvious issue with executing the most up to date unit test assemblies is some new features in 3.6.1 won't work on a 3.6.0 database. e.g. assuming a new calendar table was included in a feature in 3.6.1, any tests which required this table would fail if run on databases less than 3.6.1. A calendar class may typically exist in a common assembly which contains many other types which work in previous versions. There are essentially three types of modiciations we could be making to unit test assemblies in each new release
- Addition of schema which wouldn't work in previous versions e.g. using new table / sps / UDFs
- Updates to existing unit tests. As we adapt existing code we may have to change tests in some cases slightly, although this should be quite rare
- Addition of new tests for old schemas e.g. we may find some additional scenarios which haven't previously been tested.
2 and 3 obviously don't pose any issues however when executing unit tests against old databases we clearly need to conditionally execute the tests in 1 only if the schema is valid. A solution to this could be to apply an optional attribute against any test method called something like ApplicationVersion which would take the version number of the schema the unit test can be executed on. The attribute could be applied on a class or method. If applied on a class all tests within the class would require this version number unless any test methods overrode this by specifying a different number. e.g. a CalendarTest class could be defined like this
public class CalendarTest
However I thought whilst the attribute and use itself seemed quite generic, the actual implementation to retrieve the version number would not be. There are two ways in this case NUnit could be modified to incorporate this
- Use a plug in framework within NUnit. The plug in could have one method which allows the method to determine whether an actual method is executed or simply ignored. A Bottom line plug in could simply have knowledge about the custom attribute, read it using reflection at runtime, and then verify the version number against that stored in the database and decide whether or not to ignore the method.
- Have specific knowledge of the ApplicationVersion attribute and perhaps allow a simple piece of configured embedded C# script which could evaluate what the application number would be at runtime e.g. for us we could read it from the data store from the a table which contains the schema version.
If tests were ignored, the ignore reason could be added automatically e.g. "Test method ignored as version executing against is 3.6.0 and this method requires 3.6.1".
This feature would certainly be useful for us as we have a large number of clients using old versions and only a few at a time initially using the most up to date versions. I'm sure the solution would be useful to others too, thoughts?