Window Workflow Foundation - A new way to develop workflow

I’ve recently reviewed the new workflow framework called Windows Workflow Foundation ( WWF ) which is part of the .NET framework 3.0 runtime. Using the framework successfully within our applications will require a completely different thought process on traditional methods e.g. where we typically put most of the workflow behaviour directly within the code.

I started to develop a generic workflow framework implementation of core business object services a few years back, however even with considerable effort, the implementation lacks many of the features provided by WWF. Given that most business applications go some way to make them flexible by developing a generic workflow framework, use of WWF should allow most teams to concentrate more on the back end business logic rather than trying to do what WWF does very well.

I would highly recommend reading Bruce Bukovics excellent WWF book http://www.apress.com/book/bookDisplay.html?bID=10213 which provides coverage of all the important features in WWF very clearly.

I have summarised the main points of WWF below

General Points

1. Multiple types of workflow – It handles both sequential and state machine workflows, although typically P2P / Banking applications which Bottomline develops use state machine workflows because the processes don't define a fixed flow of control within the workflow.
2. Scalability - The ability of a workflow to persist itself could allow multiple servers to be workflow runtime hosts and therefore a workflow could be loaded and executed on a different server than the one that started it.
3. Workflow tracking - This is one of the most powerful features of WWF. However whilst it provides rich support for tracking the workflow through the different stages in the lifecycle, it also tracks rules. Rules evaluation records each rule or rule set as it is evaluated. This could also help to easily explain to users why the workflow has progressed in a particular way. You can even go further and use custom tracking profiles to track the values of the entities at particular points in time when for example a rule was evaluated. This provides a complete tracking picture for rules evaluation
4. Hosting of workflow designers - The tools to host the designer tools within your own applications are exposed and available as part of the WWF framework. This allows you to create even simple designer interfaces very easily e.g. allowing a user to create a number of different states within a state machine and chain them together to describe how each state transitions to the next and the event which cause the transition.
5. Parallel execution of activities within a workflow – Whilst activities cannot execute truly concurrently within a workflow the framework does support execution of two parallel branches of activities in that the first activity of the first branch will execute, when this finishes, the first activity in the second branch executes and so on
6. Exception handling – Provides support for handling exceptions in the workflow e.g. if activities which are part of the workflow do not complete as expected, different activities can be called and activities cleaned up correctly. Related to this is compensation which allows an activity to be undone rather than rolled back, when typical long running transactional flows could not possibly maintain a database transaction for a long period of time.

Rules

1. The rules and workflow data can be specified in an XML based format which is compiled by the WWF into code making it very quick to execute at runtime. The rules files can either be created manually or modified through using a rules designer.
2. Very feature rich rules can be created e.g. if you have access to the invoice entity you could say if Invoice.LineItems.Count > 1 then HasMultipleLineItems, and then write control of flow in your workflow using this rule.
3. Support for forward chaining of rules also means that you don't need to be so concerned with the order or priority you specify rules in. If a rule further down in a list of rules causes fields to change for which previous rules relied upon, this will cause those previous rules to be re-evaluated. Although you can override this for specific rules and specify a priority and indicate you don’t want any forward chaining to occur

Persistence of workflow

1. Objects associated with the workflow are serialized and de-serialised seamlessly. E.g. an invoice workflow object may require references to the invoice itself, and perhaps related purchase orders.
2. Handles the persistence of the workflows in the database or any other data store as required. Given the complex workflows you can create this support coming out of the box is a big plus. Not only is the workflow itself persisted i.e. the current state machine workflow , but all the associated rules and rule sets that the workflow contains
3. Workflow versioning / Dynamic workflow updates - Versioning is also handled well i.e. as you evolve your workflows and add new activities, create new or modify existing rules. Multiple entities can therefore exist which are associated with older and newer workflows. Although conversely with very long running workflows e.g. when a process will take a considerable time to complete you may want to change an existing workflow, this is also supported through dynamic updates.

Integration

1. Publishing of workflows – Workflows can be published as web services, which provide very powerful integration features to third party systems. This enables applications to call into the workflow and take appropriate actions e.g. approve an invoice.
2. Workflow chaining - Allows you to chain workflows together i.e. call one entirely new workflow from within an activity of another workflow.
3. Activities within workflows can call into web services, which provide easy integration points into many different ERP systems.

State machine workflows

1. Support complex state machines - Support for recursive composition of states which effectively means states can contains other states i.e. any events defined by the outer state are available while the workflow is currently in any of the inner states. This allows more elegant state machines to be designed which don’t contain redundant activities e.g. for an invoice a Cancel activity could be defined in a single state rather than all possible states from which the invoice could be cancelled.
2. Visibility of permissible events - The framework provides support within state machine workflows to easily see which events can be called when the workflow is in a specific state. We can easily see how this would be useful e.g. for invoices to determine which action buttons to display such as "Approve" could be driven on whether the "Approve" event is available. This would also facilitate external components knowing the events which they could call through a web service interface.

Using WWF should allow you more time to concentrate on the development of the business logic and also reduce the risks that are inherent in building a generic workflow framework.

Continuous Integration – Using integration queues

One of the long standing issues we had with our continuous integration project was the lack of any concurrency control when multiple projects were executing. Although there was some support for this through custom plug ins to ccnet it wasn’t integrated within the core code well i.e. you couldn’t easily see through cctray which projects were executing and which projects were pending, waiting on other projects to complete.

I managed to successfully integrate this new release with our project today which means for example when our unit test project is building, the core NET build project can no longer execute, which previously could cause all components to be removed whilst tests were running L. We have quite a few ccnet projects with a few dependencies so this is very useful. I’ve included the project dependencies in the diagram below

The dependencies shown in the diagram are not true UML dependencies, they simply show the project dependencies which trigger builds to occur i.e. if the Sprinter3.6.4 database project performs a new successful build, it will trigger the web app project to build.

If the diagram showed true UML dependencies then there would be a dependency between the NET assemblies project and the Sprinter3.6.4 database because the NET components depend on the schema of the database. However the NET code doesn’t need to be re built if the database changes, only component or web application unit tests should be re-executed.

Debugging - Exception stack traces are not always detailed enough

We often use the Microsoft Exception Management Application Block ( MEMAB )to publish unhandled exceptions in our applications. This provides a detailed log including all inner exceptions contained within the outer most exception. For each exception the message and stack trace are output. For the inner most exception the stack trace usually contains the actual method which caused the exception.

Although most well written applications should contain small reusable methods of perhaps no more than 20 lines, this is often not the case. If methods only contain a few lines, even with a stack trace it is sometimes very difficult or impossible to pin point the exact line which is causing the exception to occur. To find out the exact line causing the error the best course of action would be to take a dump. You can find out how to do this for a .NET application in my article http://msdn.microsoft.com/msdnmag/issues/05/07/Debugging under “Managed First-Chance Exception”. Once you have a dump you can usually find the line of code on the production server without even using debugging symbols using the techniques highlighted below.

Assuming you have a System.NullReferenceException thrown from a .NET component you should trap access violation exceptions ( av code ) and take a full dump on this type of exception. E.g. assuming the MEMAB logs the following exception info

Exception Type: System.NullReferenceException
Message: Object reference not set to an instance of an object.
TargetSite: Void Tranmit.Common.Interfaces.IXmlSerialisable.LoadFromNode(System.Xml.XmlNode, System.Collections.Hashtable)
We can find out the exact location in LoadFromNode by loading the full dump file in windbg then executing

We can find out the exact location in LoadFromNode by loading the full dump file in windbg then executing

0:000> !u @eip
Will print '>>> ' at address: 04a34312
Normal JIT generated code
[DEFAULT] [hasThis] Void Tranmit.Sprinter.AccountCodeCollection.Tranmit.Common.Interfaces.IXmlSerialisable.LoadFromNode(Class System.Xml.XmlNode,Class System.Collections.Hashtable)
Begin 04a34168, size 25b
04a34168 55 push ebp
04a34169 8bec mov ebp,esp
04a342de ff15381ea204 call dword ptr [04a21e38] (Tranmit.Sprinter.AccountCode.ExtractIdentityFromNodeAsString)
04a342e4 8945d8 mov [ebp-0x28],eax
04a342e7 b9241fa204 mov ecx,0x4a21f24 (MT: Tranmit.Sprinter.HistoryItem)
04a342ec e827ddf8fb call 009c2018mscorwks.pdb not exist
Use alternate method which may not work.
04a342f1 8bf0 mov esi,eax
04a342f3 8b45e8 mov eax,[ebp-0x18]
04a342f6 8945c8 mov [ebp-0x38],eax
04a342f9 8b157cc31202 mov edx,[0212c37c] ("The level {0} account code {1} could not be found")
04a342ff 8955cc mov [ebp-0x34],edx
04a34302 b978afba79 mov ecx,0x79baaf78 (MT: System.Int32)
04a34307 e80cddf8fb call 009c2018mscorwks.pdb not exist
Use alternate method which may not work.

04a3430c 8945d0 mov [ebp-0x30],eax
04a3430f 8b4ddc mov ecx,[ebp-0x24]
>>> 04a34312 8b01 mov eax,[ecx]
04a34314 8b400c mov eax,[eax+0xc]
04a34317 8b803c040000 mov eax,[eax+0x43c]

Reviewing the source code we can see

String accountCodeIdentity = Tranmit.Sprinter.AccountCode.ExtractIdentityFromNodeAsString(accountCodeNode);
historyItems.Add(new HistoryItem(this,
String.Format("The level {0} account code {1} could not be found", accountCode.Level, accountCodeIdentity),
DocumentType.SupplierInvoiceLineSplit));

Although we have no symbols loaded on the production server we can be pretty sure that the line

04a34302 b978afba79 mov ecx,0x79baaf78 (MT: System.Int32)

Refers to the references in the String.Format method call to the Int32 property Level referenced on the accountCode property. Reviewing the code surrounding this line we can see why the account code object would be equal to a null reference and proceed to diagnose the issue further.

Code Reviewing - Improving our process

We have implemented a code review process over the past 3 years which seems to work reasonably well. This helps considerably to reduce the number of defects QA and ultimately the customer will see. However it’s only recently that I’ve started to analyse our current process with a view to improve it.

One of my biggest concerns is that we find a lot of non functional defects and not many functional defects in code review. Whilst the non functional defects are important in terms of helping with code maintainability, functional defects should be the top priority on any code review as these defects usually manifest themselves as issues which would be visible to a customer.

I have recently read a copy of “Best Kept secrets of peer code review” which is freely available from smart bear software at http://smartbearsoftware.com. It provides useful techniques which can be used to improve your code review process and are based on real life case studies. Although of course the book does lean towards using smart bears code collaborator product to help with the code review process it does have a lot of useful content to warrant reading it even if you’re not likely to purchase these tools.

Some points I picked up from the book which I’m going to look at in improving our process are

1. Initial quick scan – The first initial scan can be very important in helping find defects throughout the code. As there is a negative correlation in the time the reviewers takes on the first scan versus the detected speed. The more time the reviewer takes on their first scan, the faster the reviewer will find defects. If you don’t perform a preliminary scan your first issues identified may focus on the wrong areas. With a quick scan you can identify areas with the highest possible issues and then address each one with a good change you’ve selected the right code to study.
2. Review for at most one hour at a time – The importance of reviewing in short time periods shouldn’t be underestimated. We should try and avoid large reviews and try and split them up into smaller reviews where possible. E.g. you could certainly split up reviews of unit test codes versus core code. Although of course this isn’t an excuse to submit code with known defects for review.
3. Omissions are hardest defects to find – It’s very easy to find issues for code you’re reviewing, it’s much harder to find defects in code which isn’t there. Having a review check list illeviates this issue some what to remind the reviewer to look for code which may not be present. Defects that aren’t associated with any checklist item should be scanned periodically. Usually there are categorical trends in your defects; turn each type of defect into a checklist item that would cause the reviewer to find it.
4. Defect density and author preparation – Providing some preparatory comments before code is submitted for review does seem to have an impact on defect density. The key point here is that by annotating certain areas of code, developers are forced to do some kind of self review of their code which obviously increases the quality. Although there is a danger that the developer by preparing could disable the reviewer’s ability for criticism i.e. he’s not thinking afresh. However usually by developers preparing these kind of comments makes them think through their logic and hence write better code.
5. Leverage the feedback you’re getting - Improve your ability and make yourself a better developer by reviewing the points you commonly receive. Code review points should be learned so you reduce the number of defects over time. A useful exercise could be to put your comments in your own personal check list which you visit when ever performing a code review. Any points which aren’t covered on the global check list should be added
6. Developer and reviewer joint responsibility – Defects found in code review are good. A defect found in review is a defect which never gets to QA or the customer. Developer and reviewer are a team and together plan to produce excellent code in all respects. The back and forth of coding and finding defects is not one developer punishing another, it’s simply a process in which two people can develop software of far greater quality than either could do single-handedly. reviewer finds in the developers code the better, this should be thought excellent summary
7. Checklist – Use check lists as an aid to identify common defects. The most effective way to build and maintain the checklist is to match defects found during review to the associated checklist item. Items that turn up many defects should be kept. Defects that aren’t associated with any checklist item should be scanned periodically. Usually there are categorical trends in your defects; turn each type of defect into a checklist item that would cause the developer to find them next time.

Design - Transitioning from conceptual domain to physical data model

When creating the design for new requirements we often design a domain model of key entities and then when were happy with it we migrate this domain model into a logical and then physical data model. Although we should go to the logical model and then transition to the physical data model we often go straight to a physical data model.

We currently perform this process manually however Enterprise Architect ( UML modelling tool ) has got great support for transforming a domain model into a data model using model driven architecture ( MDA ) templates. It simply reads the entities and the multiplicity relationships between them and generates an appropriate data model. Handling many to many relationships isn't handled with the built in templates however you can easily customise the transformation template using an editor e.g. you can specify

%if attTag:"columnLength" != ""%
length=%qt%%attTag:"columnLength"%%qt%
%elseIf classTag:"columnLength" != ""%
length=%qt%%classTag:"columnLength"%%qt%
%endIf%

to use tagged values to denote the length of a character string in the domain model. Although you shouldn't be too concerned with lengths of strings in the domain model, if you want to specify this information in a tagged value you can at this point, and then ensure it's carried over to the data model e.g. to create a varchar(100) field for a UserName attribute of a User entity. Sparx support are currently developing these templates further and the latest version of the template handles many to many relationships by creating a link table in the format JoinTo however you could customise this to fit your own database standards.

It would be great if through automation we could eventually create a data access layer from the conceptual model using the following steps

1. Create the conceptual domain model. A business analyst would ideally create the first draft
2. Add attributes as required to turn this model into a logical data model
3. Transform the logical data model into a physical data model

We actually use LLBLGen ( object relational mapping tool ) to generate a data access / business object layer from the database so in this way we could continue this process and generate the LLBLGen classes by following some additional steps

1. Create the database from the physical data model.
2. Create the LLBLGen classes from the database directly.

Both of these steps can actually be automated through the automation ability within Enterprise Architect and LLBLGen can be automated too to forward engineer the generated code. In this way we could go from a logical model to a generated database and then to an auto generated business object / data access layer automatically.

Unit tests - Web applications

We have had our unit test .NET assemblies up and running through Cruise Control for nearly a year now. This has worked very well in catching bugs early introduced by developers hastily checking in code without ensuring that ALL unit tests execute successfully. We use a main nant target which executes about a dozen testing assembly and any failures send an email to the development group so the whole team can see the user/s responsible for the checkins which broke the build. We name and shame them and this usually gets them, me quite a few times :(, to fix the build promptly.

However over the last year I noticed 90% of the defects we introduced related to the web application e.g. common script errors or simple page errors which should have been caught however because

1. The quality of testing performed by each developer was minimal
2. No developer ever regression tested many other potentially effected areas

We usually had a large number of issues which QA found come each major release. It was then a battle to fix them AND fix any other defects found from the new changes in the release. We barely had enough bandwidth to fix defects from the new changes let alone issues introduced in other areas. Since then I have had a strong motivation to enable a cruise control project which checks for changes in the web application and runs a unit test assembly with a selection of tests in the web app.

The unit testing of the web application requires a new class I’ve created called InternetExplorerWrapper which provides a number of interesting features

1. Managed access to the unmanaged internet explorer object to manipulate the object model of page contents when pages are navigated to. E.g. forms and their input fields, frames, anchors, images among many other elements.
2. Support to create unit test assertion with methods such as AssertFormFieldClassName which assert the class name of a specific form field e.g. used in test which specify a different class for fields which fail validation.
3. Internal auto reset event objects to ensure methods such as ClickAnchor only return back to the unit test code when the anchor has been clicked and the new page navigated too. The document complete event is used in this case, however this is made tricky because in some cases 3 or 4 document complete events maybe fired together because IE is loading multiple documents for a given request.
4. Methods to assist with viewing and manipulating the IE object model e.g. outputting the contents HTMLDocument object to the trace window. In this way you can see the elements you can access through your unit tests at specific points in time. Also other methods such as TraceDocsLoaded show you how many events are fired for a given request to help determine how many document complete events to wait for in requests such as ClickAnchor ( c)

There were quite a few details which were problematic e.g. the creation of pop up windows e.g. these need to be wrapped by catching the NewWindow event, then create a new IE wrapper object around this and storing in a hash table within the original requested page. Client code can then access the new windows using methods off the main IE wrapper object such as GetNewIEWindow.

Has anyone else had experience creating unit tests for web applications, I’d be interested to hear your thoughts. The framework should work regardless of the platform used to build the application e.g. ASP / ASP.NET and JSP.

Unit tests - Pattern for testing localisable code

Currently we have a few unit tests, although this is sure to increase, which test code should work in any locale ( culture in .NET terms ). To test this we currently have an implementation such as

public AppVersionTest()
{
// Set the current locale to French Thread.CurrentThread.CurrentCulture = new CultureInfo("fr-FR");

Assert.AreEqual(3.6, Tranmit.Sprinter.Configuration.Database.TBConfig.ApplicationVersion, "Failed to read the configuration ApplicationVersion");
}

We did this as we actually had a bug when the code was running on a french machine. I thought a better pattern would actually tests the code in all defined cultures e.g. changing the implementation to

// Obtain the current culture
CultureInfo oldCurrentCulture = Thread.CurrentThread.CurrentCulture;
try
{
foreach(CultureInfo cInfo in CultureInfo.GetCultures(CultureTypes.AllCultures)
{
Thread.CurrentThread.CurrentCulture = cInfo
// Perform test code
}
}
finally
{
// Restore the previous culture
Thread.CurrentThread.CurrentCulture = oldCurrentCulture;
}

Any code which is locale independant should certainly work in any culture e.g. if monetary code was being tested the culture would allow the code to find thousand / decimal separators values, so this seems a sensible implementation. However this certainly strikes me as a pattern which NUnit could implement for test methods decorated with an attribute such as TestAllCultures which would reduce this to

Test, TestAllCultures]
public AppVersionTest()
{
// Test code
}

This would certainly reduce a lot of boilerplate code required for each test method and I'm sure useful to many other developers who have to write code which works across all cultures.

Unit tests - Pattern for time changing test methods

Some of our recent unit tests change the system time within the execution of the test method. e.g. testing components which escalate purchase orders to new users if the approvers don't respond within a configured time period. We test this by setting up configuration, such as the period of time which must elapse before escalation occurs, and then changing the system time to add this time period and call the component again. The assertion is usually a simple one e.g. a new user such as the approvers manager should now in the approval route for the order.

We currently use a simple method of restoring the system time when the test method is executed by resetting it to the value captured when the test method is started. If the test method takes 2 seconds to elapsed and we have perhaps 10 test methods we can easily see that we lose 20 seconds of time every time the test fixture is called.

I've thought of a better way to do this using a class such as RestoreTimeChangingCode. The class should be created as the first object which holds the following values

a) StartTickCount
b) StartTime

The StartTickCount value is obtained using Environment.TickCount and StartTime from DateTime.Now. The Restore method should be called when the test method has finished executing, preferrably in a finally block so it's called even if an exception is raised. The Restore method simply adds the number of ticks which have elapsed since the object was created, which represents the elapsed time the test method took to execute, and adds this to the start time, and sets this as the current time.

Whilst this class is useful it would be great if the NUnit framework provided perhaps an attribute called TestChangesSystemTime which could be added in addition to the standard Test attribute. The NUnit framework could then time the method and restore the time appropriately. It could also use a more accurate timer such as the performance counter timer.

The TestChangesSystemTime attribute, in addition to reducing the amount of repeatable code you have to write in each test method, would also provide the ability to reflect on a testing assembly to see whether any test methods change the system time. This would be useful if changing the system time was a concern on any server e.g. we would not want to execute this test assembly on our source control server to avoid clients inadvertently checking in / checking out files with the incorrect date / time.

Unit tests - Testing performance

We currently use a single performance unit test assembly which has all performance related times e.g. testing X invoices can be created in under Y seconds. If the tests fail it's usually because either

1. Database changes have been applied such as indexes / triggers or modfications to stored procedures, to slow the time to perform the database access code
2. Business logic has been changed e.g. when saving the invoice additional validation has been added which slows the time to create an invoice.

There are two important things to note about our implementation

1. The expected time periods for the test methods are obtained from the configuration file. This is required because of the differences in expected time from machine to machine. If the expected time is hard coded, the code would need to change when the test is executed on different machines. The key value is read from the configuration file using reflection to obtain the method name the test is executing in, e.g. a configuration key of SP_INVOICE_PERFM_1-MaxTime maybe used for a test method called SP_INVOICE_PERFM_1. The configuration file with all the expected timings can be managed in your source control repository and checked in when the unit test assembly is. We usually check in the configuration file with the timings working on our cruise control build machine. If anyone checks in some bad database code, or some ridiculous index which slows the whole process down, we soon know about it when the next build breaks because of the failing tests.
2. The actual time period tested is not the duration of the test method, it really starts and stops at specific points in the method e.g. an invoice creation performance test is fine to test the whole method because all the method does is created a number of invoices which should be complete in X seconds. However a second test which verifies X invoices can be updated in Y seconds firstly needs to create the X invoices, and only then start the timer when just before the first update occurs, and stops when the last invoice has been updated. And of course all the invoices have to be deleted when the test finishes which shouldn't be included in the expected time. The SetUp and TearDown methods can't be used here because the unit test class such as InvoicePerformanceTest has many different methods with different set up and tear down implementations, and SetUp and TearDown are common methods called by all test methods in a text fixture. On a side note perhaps it would be useful to extend NUnit one day to support multiple set up and tear down attributed methods which apply to specific test methods.

In terms of implementing this generically in NUnit an attribute such as TimedTest with an optional attribute to specify whether the time period is obtained from a configuration file using a similar scheme we've devised. Also to cope with b), the framework could have a Start and Stop timing event which the test method could raise to indicate when the timing should start and when it should stop. This behaviour could also be specified using another parameter in the TimedTest constructor, e.g. if set to false then the start and stop would be implicit as soon as the test method started and finished.

Unit tests - Executing over multiple application versions

The main application I design and develop usually contains additional database schema with each new version e.g. a recent 3.6.1 release will no doubt have a few new tables / new stored procedures e.t.c. Currently all our automated unit tests execute against a single database using the latest schema. When we complete and release 3.6.1 the cruise control build will only execute tests against this schema. However I would like to be able to execute our unit tests against old versions at least one back if not more e.g. so we can execute tests against 3.6.0 and 3.6.1. We do put effort into the assemblies to ensure backwards compatibility so it would be very handy to test this within our continuous integration process. Even in cases where we haven’t added any specific backwards compatibility code it would be nice to know existing tests still worked on old databases as we will still apply database patches to them.

The obvious issue with executing the most up to date unit test assemblies is some new features in 3.6.1 won't work on a 3.6.0 database. e.g. assuming a new calendar table was included in a feature in 3.6.1, any tests which required this table would fail if run on databases less than 3.6.1. A calendar class may typically exist in a common assembly which contains many other types which work in previous versions. There are essentially three types of modiciations we could be making to unit test assemblies in each new release

1. Addition of schema which wouldn't work in previous versions e.g. using new table / sps / UDFs
2. Updates to existing unit tests. As we adapt existing code we may have to change tests in some cases slightly, although this should be quite rare
3. Addition of new tests for old schemas e.g. we may find some additional scenarios which haven't previously been tested.

2 and 3 obviously don't pose any issues however when executing unit tests against old databases we clearly need to conditionally execute the tests in 1 only if the schema is valid. A solution to this could be to apply an optional attribute against any test method called something like ApplicationVersion which would take the version number of the schema the unit test can be executed on. The attribute could be applied on a class or method. If applied on a class all tests within the class would require this version number unless any test methods overrode this by specifying a different number. e.g. a CalendarTest class could be defined like this

[TestFixture]
[ApplicationVersion(3,6,1,0)]
public class CalendarTest

However I thought whilst the attribute and use itself seemed quite generic, the actual implementation to retrieve the version number would not be. There are two ways in this case NUnit could be modified to incorporate this

1. Use a plug in framework within NUnit. The plug in could have one method which allows the method to determine whether an actual method is executed or simply ignored. A Bottom line plug in could simply have knowledge about the custom attribute, read it using reflection at runtime, and then verify the version number against that stored in the database and decide whether or not to ignore the method.
2. Have specific knowledge of the ApplicationVersion attribute and perhaps allow a simple piece of configured embedded C# script which could evaluate what the application number would be at runtime e.g. for us we could read it from the data store from the a table which contains the schema version.

If tests were ignored, the ignore reason could be added automatically e.g. "Test method ignored as version executing against is 3.6.0 and this method requires 3.6.1".

This feature would certainly be useful for us as we have a large number of clients using old versions and only a few at a time initially using the most up to date versions. I'm sure the solution would be useful to others too, thoughts?

Continuous Integration of database scripts

I've recently got Cruise Control working on our development server to get latest from source safe, build our .NET assemblies and execute all unit tests. I'm extremely pleased with this and has really proved it's worth recently to show complete visibility to the team of failed builds. Any broken builds which do occur get turned around very quickly; I would strongly encourage anyone who hasn't already integrated cruise control into their build process to do so now.

I have also recently taken this further by integrating database script changes into the process. In source safe we keep all the creation scripts for tables / stored procedures / UDFs e.t.c. When any changes are detected on the VSS database the scripts are applied using a few nant tasks and then the unit tests are re executed. This has quickly caught a few issues recently when stored procedures weren't in sync with the .NET code e.g. wrong data types or too many parameters in the DB compared to the client code.

Like most development teams we have to maintain a few versions of our application and upgrade databases from old version to new versions. The latest database script are rerunnable against old versions of the schema i.e. it will add columns / stored procedures if their not already there, Ideally I would like to execute the database scripts against a number of different versions of the database and then execute the unit tests against the newly updated database to ensure the tests still passed. This would find issues for clients upgrading very quickly. I'd be interested in hearing from anyone else who has tried to automatically integrate database changes within their continous integration process.

Unit tests - Changing app.config values at runtime

It would be useful for a number of unit tests to change the configuration values used in the app.config file of a unit test DLL. Actually changing the value in the file is straightforward enough i.e.

1. Get access to the app.config file full path from the process using AppDomain.CurrentDomain.SetupInformation.ConfigurationFile
2. Load the file into an XmlDocument
3. Extract the relevant node using code such as XmlNode node = xmlDocument.SelectSingleNode(@"/configuration/appSettings/add[@key='" + configurationName + "']");
4. Set the value of the “value” attribute using node.Attributes.GetNamedItem("value").Value = configurationValue;

However by default changes to the configuration file won’t be reflected at runtime i.e. code such as

String fromEmailAddress = System.Configuration.ConfigurationSettings.AppSettings["EmailFromAddress"];

will not reflect the new value if EmailFromAddress is changed in the app.config file. The AppSettings class caches the values retrieved within the AppDomain and never refreshes them. ASP.NET as a CLR host actually does refresh the settings when the web.config file is changed however by default the standard CLR host won't.

Since there is no way to force the settings to be re loaded the simplest approach would be to ensure that all clients use the existing façade defined in Tranmit.Common.ConfigurationSupport. The GetValue method looks like this

public static Object GetValue( String key,
Type typeOfObject,
Object defaultValue)
{
// Assign default value to object if no key found
Object value = defaultValue;

System.Configuration.AppSettingsReader appSettingsReader = new System.Configuration.AppSettingsReader();

// Check value is present first to avoid exception
// being thrown from AppSettingsReader.GetValue
if(ConfigurationSettings.AppSettings.Get(key) != null)
{
value = appSettingsReader.GetValue(key, typeOfObject);
}

return value;
} // GetValue

This is fine for production code however when executing unit tests we need to make sure the value is loaded directly from the app.config file rather than through the AppSettings or AppSettingsReader class. We could do this by

• Set a value within the AppDomain to indicate a unit test is being executed e.g. AppDomain.CurrentDomain.SetData(“UnitTestExecuting”,true);
• Modifying the existing GetValue method to check the app domain value UnitTestExecuting. If this is enabled then extract the configuration value using the steps 1 to 3 above directly from the file. Otherwise use the existing code which uses the AppSettings class

There should only be a slight performance impact for unit test code which is fine, but production code should execute the same. This will obviously require that client code uses the GetValue method rather than the classes in the ConfigurationSettings namespace directly.

VBScript plug ins for ASP applications

Recently one of the developers in my team was developing some custom code in a COM table rendering object we've built for an ASP web application. The object essentially takes source SQL with added meta data and creates a HTML table which is returned into the calling ASP application, and then written out using Response.Write.

The design required some very custom presentation within one of the table cells. The table rendering object is quite generic ( knowing little about specific tables in the database schema ) and the custom presentation task required work on very specific tables. Custom work is essentially associated with each table cell using meta data tags such as "SubordinateList(%UserID%)". Assume the table object is rendering a list of users and if a user is a manager displaying a comma separate list of subordinate user names.

Although this example is contrived (as our real example would require a lot more explanation) the design issue is the same. The obvious easy approach would be to simply add handling within the table rendering object in a switch statement to match on the start of the tag "SubordinateList" and then perform some custom processing in SQL to find the list of sub ordinates. The list would then be converted into a comma separate string which would be added to the table cell. This business logic ideally should not be contained within a generic table rendering engine.

You may have seen plug in design patterns which support the extensiblilty we are trying to achieve here. Typically this would require the creation of a well defined interface e.g. IValueResolver which is defined and called in the table rendering object. The implementation of different IValueResolver objects could be created in one or many plug in components. The IValueResolver interface could only require one method such as GetValue which takes a row, column and table data.

Although ASP, using VB script, doesn't support strongly typed interfaces we can still use the late bound IDispatch support in VBScript to implement this design. We could create a VB script class on the ASP page such as

Class SubordinateListValueResolver
Dim UserColumn

Public Function GetValue(row, col, tableData)
' Work to retrieve the list of subordinates from the user specified
' in the column UserColumn in the row from the tableData data source.
End Function
End Class

The plug in class could then be instantied on the ASP page and passed to the table rendering object using code such as

Dim oValueResolver
Set oValueResolver = New SubordinateListValueResolver
oValueResolver.UserColumn = 4 ' Means the 4th column in the result
oTableRenderer.AddValueResolverPlugIn(oValueResolver, "SubordinateList")

The table rendering object can then determine whether any plug ins exist using the meta data tag passed when the data source SQL is specified. e.g. if a matching plug in object is found for the tag SubordinateList the VBScript object of type SubordinateListValueResolver will be returned. The GetValue method can then be called directly passing the row, column and table data from within the table rendering object. i.e.

oCellResolvedValue = plugin.GetValue(row, col, tableData)

Although designs which use well defined interfaces and contain the plug in code in compiled languages are preferrable to this approach, it does demonstrate how easy a simple plug in mechanism can be developed for a script based application.

Remote debugging - Release build optimisations

I recently had to diagnose an issue which required remote debugging the application as no exception was raised and therefore no dump could be obtained and analysed. The issue was caused by an error code deep down in the call stack being returned and ignored a few levels up the call stack :(

Whilst this was trivial to debug using the windbg client and dbgsvr on the server ( both part of the debugging tools package ) I found in some cases, particularly where the error code was being returned, that local variables couldn't be seen. The message <Memory access error> was displayed in the locals window for each variable.

Our standard process is to generate debugging symbols with each release build by simply selecting the generate debugging symbols option (http://msdn.microsoft.com/library/en-us/dnvc60/html/gendepdebug.asp). To accurately see variables in both the watch and locals windows you need to make sure you disable optimisations i.e. through Project Settings / C++ / Optimisations - Set this to Disable ( Debug ).

Obviously this should only be disabled for ad hoc builds as you will most likely want to enable all optimisations in your standard builds you deploy. It's hard enough trying to get our applications to be as performant as possible without disabling optimisiations which the compiler gives us automatically :)

Database filtering of documents - performance issues

Our application has a requirement to filter the documents a user can see based on various criteria e.g. security / approval rights. A user may only wish to see documents which they can approve or have approved. This would seem to be a common pattern for a lot of business applications.

To implement this our business objects create a list of document identifiers which are then appended to a SQL statement which selects the data and filters the documents returned. We used to use the IN clause to filter the documents returned using a pattern such as


SELECT * FROM PurchaseOrders WHERE PurchaseOrderID IN ( CommaSeperatedListOfIDs )

where the list of order ids was generated from the business object. After usage of the application quickly increased we encountered a stack overflow exception issue http://support.microsoft.com/kb/q288095. To resolve this we changed our approach to use temporary tables i.e. insert all the IDs from the business object into a temporary table and then use this to filter the main query e.g.

SELECT * FROM PurchaseOrders WHERE PurchaseOrderID IN ( SELECT ID FROM UniqueTempTableName )

The IDs are inserted into the temporary table using batch INSERTS e.g. in batches of 20 statements at a time. However when upwards of 10000 document identifiers of inserted into the table, it can take over 5 seconds to do this. E.g. an ASP screen which presents a list of invoices takes 15 seconds of which 11 seconds is attributed to the time to insert into the temporary table. Over the last day I've been exploring ways to minimise this.

Extended stored procedure
My first thoughts were creating an extended stored procedure which would create an in memory result set using a comma separated list of identifiers. The extended stored procedure would be called from a UDF which would return a table variable which could be used in a select statement e.g.

CREATE FUNCTION dbo.Tfn_IDs(@listIdentifiers text)
RETURNS @idTable TABLE (id int )
AS
BEGIN
INSERT INTO @idTable
EXEC master..xp_IDContainer(@listIdentifiers)
RETURN
END

However unfortunately EXEC cannot be used as the source for an INSERT INTO statement.

OPENXML
I then realised I maybe able to generate an XML representation of the identifiers using the extended stored procedure e.g. in the format

<values>
<value id="84">
<value id="85">
</values>

and then use the XML in a SQL OPENXML statement using

INSERT INTO @IDTable
SELECT ID
FROM OPENXML (@hDoc,'/Values/Value')
WITH (ID int)

UDF Only
I then went back to an approach without using an extended stored procedure and only using a UDF. The UDF could parse the comma separated list of values and insert into a table variable which would be returned. I won't outline the solution in detail but it basically encompasses using CHARINDEX to find the start and end positions of each identifiers using the comma as the delimmiter. It would be interesting to compare the timings of this and the OPENXML approach. The XML approach could be slower because of the more verbose structure of the data and the cost of creating the XML internally in SQL Server and performing XQuery expressions however it maybe offset by the improvement in speed of using a single INSERT INTO statement rather than multiple INSERT INTO statements for each identifier found and parsed in the UDF only approach.

However after increasing the size of the input to this solution it quickly became apparent that CHARINDEX truncates text data passed to varchar(8000) and so won't work with long text types which contains 10000 identifiers with an average of 3 characters each.

Back to a UDF + extended stored procedure
To overcome the issues faced in the UDF only solution an extended stored procedure would probably be the logical place to put all the parsing logic of the comma separated string into values used for an insert statement. However instead of using the resultset of an EXEC call as source to an INSERT INTO statement, which weve shown isn't possible, we could use an output parameter to get each identifier. Psuedo TSQL could look like

@sessionToken = xp_IDContainer_ParseValue('1,2,3')
WHILE xp_IDContainer_EndOfResults(@sessionToken) <> 0
BEGIN
@ID = xp_IDContainer_GetNextID(@sessionToken)
Insert id into table variable
END

Has anyone solved this problem using the approaches I've highlighted or other approaches which are successful with large volumes of data.

Debugging - Data grid exceptions

I recently noticed an issue in one of our .NET controls which contains a Data grid which was causing values editted to be reverted back to their original values without any events being raised or any exceptions being raised to the client.

However if you enable all CLR exceptions to break into the debugger you will see an exception e.g. in this case "1,444.66 is not a valid value for Decimal". You can then get the call stack at this point and determine where the code is failing.

The data grid seems to handle quite a few exceptions silently which can cause confusion sometimes. If you are experiencing any odd behaviour with the Data Grid I strongly recommend you enable all exceptions to break into the debugger. To do this from Visual Studio .NET perform the following

1. Select the Debug menu
2. Select Exceptions
3. Select Common Language Runtime Exception.
4. Change "When the exception is thrown" to "Break into the debugger". This will apply the setting to all exception types unless you've overridden this.
   

Visual Basic Production Debugging

A Visual Basic (VB) error is usually raised by either an explicit call to Err.Raise or by calling another COM object which raises an error. In both cases VB provides similar error handling to VB Script through the On Error Resume Next and On Error Goto statements. In VBScript production debugging I discussed how to generate dumps and obtain error information when VB Script errors occurred. Obtaining the same information when a VB error is raised is raised takes a bit more work.

You may be using VB components from within an ASP web application to perform a lot of business logic and may find some errors are being raised but ignored by either the components or the ASP script. Trapping these errors in a production environment is therefore very useful.

The main difference between VBScript and VB errors are VB errors are raised as true native exceptions (exception code c000008f ). To trap VB script errors break points had to be placed within the scripting runtime module. In this post I will walk you through the commands you need to execute on a live or full dump.After we’ve gone through the commands we’ll put together an adplus script to capture this information which you can use on production servers.

When the dump is open switch to the VB thread - you can use ~*kb to see which threads have MSVBVM60. e.g.

0:000> ~20s

Determine the index of the EBTHREAD object in thread local storage (TLS).

0:020> dd msvbvm60!g_itlsebthread l1
6aae0ec0 0000001e

Find the location of this thread's TLS based on the thread environment block (TEB). If you want to dump the Err object for other VB threads just get the TEB for those threads.

0:020> ~. 20
Id: 758.5bc Suspend: 1 Teb: 7ffa8000 Unfrozen
0:020> ?7ffa8000 +0xe10 Evaluate expression: 2147126800 = 7ffa8e10

Get the location of our EBTHREAD object based on the index into TLS

0:020> ? 7ffa8e10 + 0000001e * 0x4
Evaluate expression: 2147126920 = 7ffa8e88
0:020> dd 7ffa8e88 l1
7ffa8e88 0977ef58

Get the address of the EXCEPINFOA member variable based on its offset into EBTHREAD

0:020> ?0977ef58+0x78
Evaluate expression: 158855120 = 0977efd0

Now dump the EXCEPINFO data

0:020> dd 0977efd0 l8
0977efd0 00000000 250613ec 250314da 00000000
0977efe0 00000000 00000000 6aaa8bcf 80004005

At this point you have all of the information in the VB Error object. In this particular instance we have the bstrSource, bstrDescription, the pfnDeferredFillIn, and the scode. The pfnDeferredFillIn is simply a function in VB that is called as needed to get more information about the error object (ie. the description). This is called when the VB component tries to get the data (ie. access the Err.Description property) which is why the information may not be filled in sometimes.

For this dump we can see the error source

0:020> du 250613ec
250613ec "Microsoft OLE DB Provider for SQ"
250613ec "L Server"

and the description

0:020> du 250314da
250314da "[DBNETLIB][ConnectionOpen (Conne"
250314da "ct()).]SQL Server does not exist"
250314da " or access denied."

and the error number

0:020> ? 80004005
Evaluate expression: -2147467259 = 80004005

As a full dump is required to access this information you can typically run into performance issues on the server if many exceptions are being generated due to the size of the files. It would be far more effective if we could use an adplus script to extract the error source, description, and number from a process at runtime. This avoids the need to create dump files and adds all the relevant information in the adplus log file. The script is shown below

<ADPlus>
<Settings>
<RunMode> CRASH </RunMode>
</Settings>
<Exceptions>
<NewException>
<Code> c000008f </Code>
<Name> VBError </Name>
</NewException>
<Config>
<Code>c000008f</Code>
<CustomActions1>
.echo VB error occurred.;
.echo ;
r $t1 = poi($teb+0xe10+poi(MSVBVM60!g_itlsebthread)*4)+0x78;
.echo Source ;
.if poi($t1 + 0x4)=0 { .echo Unknown } .else { du poi($t1 + 0x4) };
.echo Description ;
.if poi($t1 + 0x8)=0 { .echo Unknown } .else { du poi($t1 + 0x8) };
.echo Error number ;
?poi($t1 + 0x1c);
</CustomActions1>
</Config>
</Exceptions>
</ADPlus>

The script uses some useful commands such as $teb to extract the TEB memory address and .if / .else to allow commands to be conditionally executed. It also sets a pseudo register $t1 to improve the performance of the error extraction, otherwise this data would have to be evaluated three times.

For some VB errors where the description is not filled in you can open VB6 and in the immediate window just type 'error ’. If you don’t have access to debugging symbols on the server you can use the same techniques I discussed in Setting breakpoints without symbols. In this case you should find the memory address of the symbol MSVBVM60!g_itlsebthread and replace it with the memory address e.g. MSVBVM60 + 0x12345678.

Setting breakpoints when production servers have no access to symbols

In some cases a production server may not

1. Have access to the internet to use the Microsoft symbol server.
2. Access to symbol files using a UNC path.

In both cases it is impossible to set break points using symbolic names in either a live debug session or when using adplus. This is clearly a problem for the adplus configuration script I discussed in VB Script production debugging.

If symbols aren’t available there is another solution by adding break points set using a memory address location. In the previous blog the adplus config file was setting a breakpoint on the function vbscript!CScriptRuntime::RecordErrorContext. To find out the memory address for the breakpoint we need to determine what the offset to vbscript!CScriptRuntime::RecordErrorContext is in vbscript.dll.

First we need to take a dump of the process on the production server and load it into windbg on a machine which does have access to a symbol store. Open the dump and enter the x command to find the address of the function

0:031> x vbscript!CScriptRuntime::RecordErrorContext
734753cf vbscript!CScriptRuntime::RecordErrorContext =

Then we need to find the module address for vbscript.

0:031> lmm vbscript
start end module name
73460000 734c5000 vbscript

Subtract the module start address from the symbol address

0:031> ?734753cf - 73460000
Evaluate expression: 86991 = 000153cf

Now we have our offset we can use it to set a break point either during a live debug or in the adplus config file. The dump I used in the example above was using 5.6.0.8515 of vbscript.dll. The Address element of the adplus configuration file should be modified from

vbscript!CScriptRuntime::RecordErrorContext

to

vbscript + 0x000153cf

Effective Production Debugging MSDN Article

Following my recent production debugging MSDN article (http://msdn.microsoft.com/msdnmag/issues/05/07/Debugging/default.aspx.
I've written a number of posts on techniques I wanted to cover in the article but couldn't as I ran out of space.

I've already written on topics such as creating dumps through terminal services and VB Script debugging however If there's anything you'd especially like to see let me know and I'll try and cover it.

VBScript Production Debugging

An error typically occurs in an ASP application running VB Script when either

1. An On Error Resume Next statement is called prior to executing some script. A number of method calls are then made which can potentially raise errors. After each statement is completed the Err.Number property is usually checked and action taken such as logging the error and providing user friendly messages.
2. No On Error Resume Next statement is called prior to executing some script . In this case any statements that follow will halt the processing of the script and return a runtime application error to the user.

Whilst the first case is the preferred approach to handle errors gracefully most developers typically do not check most statements for errors. If Err.Number is not checked errors can be silently ignored by the server side script, and then consequently cause other errors which are logged. It is therefore essential to be able to log information on the original error, namely the source, description and number.

Unlike languages such as C++ / VB or .NET exceptions aren’t raised to the debugger from scripting engines such as VBScript. Consequently we can’t use adplus to capture a dump file when a specific type of exception occurs. However when an error is raised a number of functions are called within the scripting runtime.

• vbscript!COleScript::OnScriptError. This is called when a script occurs in the 2nd case above
• vbscript!CScriptRuntime::RecordErrorContext. This is called to record an error in the scripting runtime when On Error Resume Next has been called.

Although we can’t use adplus to create a dump file on an exception (because none is raised) we can use it to create a dump when a specific break point is reached. Opening the dump file shows the following call stack

0:023> kvn
# ChildEBP RetAddr Args to Child
00 0148f4a8 733183bf 0003aa70 000b5f48 00000000 vbscript!CScriptRuntime::RecordErrorContext
01 0148f4c4 73318320 0003aa70 0148f5cc 00000010 vbscript!CSession::RecordExcepInfoAndClear
02 0148f5b8 73304b01 0003a9c0 000b5f48 60030001 vbscript!InvokeDispatch

If we dump the first argument plus 8 we can see the EXCEPINFO structure which is passed to this method. EXCEPINFO is the automation structure which defines an error i.e.

struct EXCEPINFOA, 9 elements, 0x20 bytes
+0x000 wCode : Uint2B
+0x002 EXC_unused1 : UChar
+0x004 bstrSource : Ptr32 to Void
+0x008 bstrDescription : Ptr32 to Void
+0x00c bstrHelpFile : Ptr32 to Void
+0x010 dwHelpContext : Uint4B
+0x014 EXC_unused2 : UChar
+0x018 pfnDeferredFillIn : Ptr32 to Void
+0x01c scode : Uint4B

Reviewing the dump we see

0:023> dd 0003aa70 + 8
0003aa78 00000000 000b85bc 00121254 00000000
0003aa88 000f43e8 00000000 00000000 800a0063
0003aa98 00000000 00000000 00000000 00000000
0003aaa8 00000000 00000000 000a0004 0008012a

To find the source and description we can use the du command i.e.

0:023> du 000b85bc
000b85bc "MyVBClass"
0:023> du 00121254
00121254 "Failed to get supplier invoice, "
00121294 "Reason=Failed to open connection"
001212d4 ", reason=Object required"

However to get this information a full dump needs to be created. If many script errors occur this maybe unworkable in a production environment. A more elegant approach involves using a custom action in an adplus config file to extract the source, description and number and add to the log file adplus produces. i.e.

<ADPlus>
<Settings>
<RunMode>CRASH</RunMode>
<ProcessID><Put process id in here></ProcessID>
</Settings>
<Breakpoints>
<NewBP>
<Address>vbscript!CScriptRuntime::RecordErrorContext</Address>
<CustomActions>.echo -----;.echo VB Script On Error Resume Next error occurred.;.echo;.echo Source:;du poi(poi(esp+4)+c);.echo Description:;du
poi(poi(esp+4)+10);.echo Code:;?poi(poi(esp+4)+24);g</CustomActions>
<Actions>Log;Stack;</Actions>
<ReturnAction>G</ReturnAction>
</NewBP>
<NewBP>
<Address>vbscript!COleScript::OnScriptError </Address>
<CustomActions>.echo -----;.echo VB Script On Error Resume Next error occurred.;.echo;.echo Source:;du poi(poi(esp+4)+c);.echo Description:;du
poi(poi(esp+4)+10);.echo Code:;?poi(poi(esp+4)+24);g</CustomActions>
<Actions>Log;Stack;</Actions>
<ReturnAction>G</ReturnAction>
</NewBP>
</Breakpoints>
</ADPlus>

The most interesting line in the config file is

.echo Source:;du poi(poi(esp+4)+c);.echo Description:;du poi(poi(esp+4)+10);.echo Code:;?poi(poi(esp+4)+24);

The source and description are output as strings using the du command. From the parameter poi(poi(esp+4)+c), esp+4 is the stack pointer + 1 DWORD which equals the first parameter to the function were breaking on ( RecordErrorContext or OnScriptError ). This is dereferenced using the poi operator from which we then add 0xC to get to the offset of the Source member in the EXCEPINFO structure. The Source member is just a pointer to a string so poi is used again to dereference it. Description is handled the same and the error code is simply output using the ? evaluate expression command.

When a VB Script exception occurs you get all the relevant information provided in the log file along with any other exceptions which are raised i.e.

VB Script On Error Resume Next error occurred.

Source:
0017edf4 "MyVBClass"
Description:
0017eeac "Failed to get supplier invoice, "
0017eeec "Reason=Failed to open connection"
0017ef2c ", reason=Object required"
Code:
Evaluate expression: -2146828189 = 800a0063