A site devoted to discussing techniques that promote quality and ethical practices in software development.

Showing posts with label Unit Testing. Show all posts
Showing posts with label Unit Testing. Show all posts

Wednesday, May 11, 2011

Caveat in NUnit testing Matlab NE Builder produced assembly

Matlab NE Builder produced assembly when loaded needs to extract the compiled scripts into a cache area which is normally in the executable directory.

When such an assembly is being tested in NUnit, with Shadow Copying enabled (the default setting), can generate a file path that exceeds the maximum length of file path name of Windows.

When this happens, it can generate an exception saying that it cannot create an instance of MCR. Check the Text output tab of NUnit console for any message saying something like this:
Failed to remove
 Verify file ownership and access permissions.
Where is a very long path name.

If you still want to use Shadow Copying support in NUnit, go to NUnit's console's settings dialog and specify a short directory as a Shadow Copy Cache, which is usually your "%Temp%\nunit20\ShadowCopyCache". Alternately turn off the Shadow Copy can help to alleviate this problem.

Sunday, March 13, 2011

Definition of a "Bad Design" in software

I came across this succinct definition of what constitutes a "Bad Design" in software:
  1. It is hard to change because every change affects too many other parts of the system. (Rigidity)
  2. When you make a change, unexpected parts of the system break. (Fragility)
  3. It is hard to reuse in another application because it cannot be disentangled from the current application. (Immobility).
Surprisingly, software having these traits are very difficult or even impossible to be unit tested. Hence unit test is often used to a gauge of these bad traits. Don't believe, try it.

Sunday, November 14, 2010

Beware when using AppDomain.BaseDirectory in NUnit

In my previous post I accused Uri() exhibiting some quirky behavior,
The reason is that depending on whether the base uri for file protocol is your current execution directory, that constructor will drop the last component of your base uri if the current directory is not your base uri  when forming the final uri.
This is indeed a wrong accusation. System.Uri() is functioning correctly in accordance to RFC 3986. For example
Assert.IsTrue( (new Uri( new Uri(@"C:\A\B\C"), "test.txt")).LocalPath,
               @"C:\A\B\test.txt" );
Assert.IsTrue( (new Uri( new Uri(@"C:\A\B\C\"), "test.txt")).LocalPath,
               @"C:\A\B\C\test.txt" );

In normal application, like a console application or WinForm application, the following result is true:
Debug.Assert( AppDomain.CurrentDomain.BaseDirectory.EndWith( @"\" ) );

But when executing the above statement in a TestFixture class in NUnit, the assert fails. This is because when NUnit defines the AppDomainSetup.ApplicationBase, it fails to include a trailing '\'. NUnit simply uses Environment.CurrentDirectory, which does not have a ending '\'. Incidentally, the sample code makes the mistake of assuming the Environment.CurrentDirectory ends with a '\' resulting in forming an incorrect path to the application configuration file.

Normally, this discrepancy does not matter if you use System.IO.Path.Combine() to form path names. However, in the case of when log4net is used inside NUnit it becomes an issue when it uses Uri( Uri, string ) to form the Uri to the log4net configuration file with the value from AppDomain.CurrentDomain.BaseDirectory. The end result is a wrong Uri to the configuration file. To overcome this problem in NUnit, it is therefore wise to specify a full path name for the configuration file in the NUnit's project's configuration file as recommended.

After all these experiments, I am none the wiser what is the official convention for AppDomain.BaseDirectory and Environment.CurrentDirectory? At least in Uri, it is unambiguously defined in the RFC. Perhaps someone reading this from Microsoft can care to comment on this to let the world know if there is such a convention at all.

Saturday, November 13, 2010

Log4Net configuration issue when using with NUnit - Solutions & Explanation

Log4Net is a widely used logging framework and an application has to configure it in order for your instructions, such as where to log your message to, what to ignore, etc, are carried out. However, there are several quirks that one has to be aware, particularly when using it with NUnit. This post uses NUnit version 2.5.7 and log4net version 1.2.10 and hence the comments below are relevant to this version combination.

However, when used in NUnit, report has been filed that it does not generate any log file. In other word, the combination fails to find the required configuration file. The report accused NUnit being the culprit but I can assure readers that it is not entirely. This triggers a number of unnecessary work around.

In this post I will give you a number of solutions first and I will go into detail explanation of what causes this quirky behavior. It is neither party's fault but exposing an issue with how log4net loads the configuration, which is not a fault either.

If you are not interested in the technical discussion, just skip the second section.

Solutions

Solution 1 - No need to write code

I prefer this solution because it fits perfectly into "The Most Beautiful Code I Never Wrote".

In order to use NUnit with your test assemblies, which may or may not contain log4net logging calls, that call into your deploy assemblies, the ones being tested, containing log4net calls, do the following steps:
1) Create a NUnit project config file which usually is in the same directory as your NUnit project file, if not already present.
2) In the config file define the key (in red):
<configuration>
  <appSettings>
    <add key="log4net.Config" 
   value="E:\Libraries Tests\Log4NetDemo\Unit Tests\SimpleLog4net-UnitTest.config"/>
  </appSettings>
</configuration>
The value of that key defines the log4net configuration file that will be used to initialize the default repository.

That is all to it! Nice and simple.

Now when you run your NUnit, all your log4net calls will follow the instructions specified in that configuration file.

It is recommended to use full path name in specifying the configuration file due to some quirky behavior of System.Uri( String, String ) call. This will be disclosed in the Technical Discussion section.

Solution 2 - Use work around

While work around generally works but not very scalable, particularly when you have a collection of test fixture classes and/or assemblies. You need to ensure no matter which test assemblies are called that the configuration file still is used. Here is the recipe of a better work around:
1) Create a class NUnitLog4NetLogManager which has a method called GetLogger( Type type ) and package that inside an helper assembly:
static public class NUnitLog4NetLogManager {
  public static ILog GetLogger( Type type ) {
    return GetLogger( Assembly.GetCallingAssembly(), type );
  }
  // ...
} 

2) NUnitLog4NetLogManager.GetLogger() will use the given assembly object to attempt to retrieve any custom attribute of type log4net.Config.XmlConfiguratorAttribute type. Once found it will then use it to retrieve the config file and watch state. With these pieces of information it will form the full pathname of the configuration file using AppDomain.CurrentDomain.BaseDirectory, if the config file is not a full path.

It then form the System.IO.FileInfo with which it then calls XmlConfigurator.Configure() or XmlConfigurator.ConfigureAndWatch() to load the configuration into log4net.

3) Once step 2 is completed, it then calls LoggerManager.GetLogger(Assembly, Type) to grab the logger and return it to the caller.

This technique allows user to use familiar way of specifying the preferred configuration file like this:
// AssemblyInfo.cs
[assembly: log4net.Config.XmlConfigurator(ConfigFile="MyTest.config")]

and to create the logger in a TextFixture class like this:
// MySimpleTest.cs
[TestFixture]
public class MySimpleTest {
  static ILog log = NUnitLog4NetLogManager.GetLogger( typeof(MySimpleTest) );
  [Test]
  public void SimpleTest() {
  // ....
  }
} 

This has advantage over the work around in that any class included into this assembly can make use of the familiar syntax to get a logger and in doing so indirectly load up the configuration file.

Technical Discussion

Typically, an assembly can include an assembly-level Log4Net custom attribute, XmlConfiguratorAttribute, to specify the configuration information. When a situation that needs to load up the configuration file, log4net internal logic will look for an instance of this custom attribute in a given assembly for the configuration file, etc. This class is located in src\Config\XmlConfiguratorAttribute.cs.

In theory, each assembly can use this to specify different configuration file. log4net does not process or configure its system until the first reference to create a logger or an appender. But in practice the first logger created determines the configuration file used and subsequent calls to create different loggers will not load the config file specified in that assembly.

There is really no need to have separate configuration files as one can control all loggers in one configuration file.

Bear this piece of information in mind will help to understand the issue at hand.

When NUnit is launched and before running your test assemblies, code inside NUnit causes NUnit.Core.Log4NetCapture.StartCapture() to execute. This is an override of its parent abstract class TextCapture. The result is that it causes a the log4net.Appender.TextWriteAppender to be created.

When this happens, some how it triggers a call to log4net.Config.BasicConfigurator() for this appender. This in turn causes the creation of Logger Repository for the assembly nunit.core, which in turns starts looking for custom attributes for repository.

Since these custom attributes are rarely used and nothing is found, it then look for an instance of custom attribute log4net.Config.ConfiguratorAttribute or log4net.Config.XmlConfiguratorAttribute in nunit.core to configure this default repository. Note that it is not your assembly that is used to create the logger repository; it is nunit.core.

If no such custom attribute is found it will then look for a value for the key. If a value is returned, it will then use it to configure this default repository (see code DefaultRepositorySelector.ConfigureRepository(Assembly, ILoggerRepository) in src\Core\DefaultRepositorySelector.cs). The logic will eventually then call XmlConfigurator.Configure() to load and configure the default repository.

If nothing is found, it will simply leave this default repository with its default initialized state.

After all this is completed, NUnit will begin to execute the first TestFixture class it can find, when that happens, assuming the logger is created in a static object initialization style, NUnit will call into log4net to create a logger identified by the given type or string. log4net will grab the calling assembly to begin creation of the logger.

Instead of looking for the custom attribute that you have injected into the AssemblyInfo.cs file, for example, to specify the log4net config file, log4net now has a default repository and it will no need to go through the motion to locate those custom attributes.

The result causes people to believe NUnit or log4net is responsible for not finding their configuration file. This is because they do not realize NUnit is the culprit that causes the creation of a default repository with default initialization state. It is not that log4net can't find your log4net config file, no matter how convenient you have placed it. It is simply that by then log4net does not need to do that.

log4net will then create a logger and stashes it into the default repository, which happens to have the default initialized state, that your logger will use when call upon. It then returns this logger back to your code.

When you call the logger (ILogger) method to log messages, it has found that it is not initialized and hence it will not generate the expected log file, if you use FileAppender for example leading to the submission of the bug report on NUnit.

In reality, perhaps NUnit in version 2.5.7 happens by accident trips the load once issue of log4net as described above and hence when it comes to your test assembly, it simply ignores your directive placed inside the config file.

Using Solution 1 causes the creation of the default repository to use that file to initialize the default repository thereby ensuring the logger will obey your instructions.

With respect to using my solution 1, I mentioned that there is a quirky issue with System.Uri(Uri, string) that makes me to recommend the use of full path name for the value in the key. The reason is that depending on whether the base uri for file protocol is your current execution directory, that constructor will drop the last component of your base uri if the current directory is not your base uri  when forming the final uri. The following code fragment illustrates the issue
// Current directory in MyTestAssembly is 
String curDir = "C:\A\Test\MyTestAssembly\bin\debug";
Debug.Assert( Environment.CurrentDirector==curDir );
Uri x = new Uri( new Uri("C:\A\Test\MyTestAssembly"), "hello.txt" );
Debug.Assert( x.LocalPath == "C:\A\Test\hello.txt" );

In the log4net, it uses AppDomain.CurrentDomain.BaseDirectory as the base Uri and this may or may not be the current directory and this uncertainty is very hard to tell since log4net does not throw any exception when the config file does not exist. For this reason, it is best just to use a full path name for your configuration file and this causes Uri(Uri, string) to ignore the base uri.

The work around specifies a full path to the custom log4net config file to construct the FileInfo object to XmlConfigurator.Configure(). This is not necessary because at that point in time, your current directory is the target directory of your test assembly and FileInfo() will always use current directory to fully qualify the supplied file name if it is not an absolute path name.



[1] "Beautiful Code - Leading Programmers Explain How They Think" Chapter 3 "The Most Beautiful Code I never wrote".

Wednesday, December 2, 2009

Treat String containing '\0' with care

Recently I had to decode a sequence of octets representing the UCS2 of some data and the sequence contained the C-Style string's null terminator. The sequence of octets were represented in C# as Byte[] naturally and Encoding.Unicode.GetString() was used to decode the sequence into a Unicode string.

The decoding process did not trim off the terminating '\0'. So it produced a Unicode string like this "Hello World\0".

During testing, several interesting things were discovered that could have significant impact on your code and the validity of your Unit Tests:
1) If you write out the string to console like this:
String s = "Hello World\0"; 
Console.WriteLine( "This is the Unicode String = '{0}'", s );

Your output will not contain the trailing '.

2) If you use write out several lines like this:
String s = "Hello World\0"; 
Console.WriteLine( "This is the Unicode String = '{0}'", s );
Console.WriteLine( "See anything strange?" );

It will produce a line like this:
This is the Unicode String = 'Hello WorldSee anything strange?

The Console.WriteLine() removes the ending ' and the newline.

3) Make sure you use the correct version of String.Compare() and use the correct comparison type as specified by the StringComparison enumeration:
String s0 = "Hello World"; 
String s1 = s0 + "\0";

Assert.IsFale( s0.EndsWith( "\0" ); // Just to check 
Assert.IsTrue( s1.EndsWith( "\0" ); // Just to check
Assert.IsTrue( s0.Equals( s1 )==false ); 
Assert.IsTrue( String.Compare( s1, s0 )!=0 );     // Fail but should Pass
Assert.IsTrue( String.Compare( s1, s0, false )!=0 ); // Fail but should Pass
Assert.IsTrue( String.Compare( s1, s0, StringComparison.Ordinal )!=0 );
Assert.IsTrue( String.Compare( s1, s0, 
   StringComparison.InvariantCulture )!=0 ); // Fail but should Pass

The lines in red are incorrectly processed.

Be careful when using NUnit's Assert.AreEqual(). In version 2.4.x, the AreEqual() incorrectly handles this case resulting in passing the comparison. This mistake has been fixed in ver 2.5. MSTest's AreEqual() correctly handles this case.

This illustrates the need to exercise care when comparing strings.

Sunday, May 3, 2009

Designing API - Bad API can really hurts

An article titled "API Design Matters" is a must read article for any one in charge of designing API and has this to say:
Good APIs are a joy to use. They work without friction and almost disappear from sight: the right call for a particular job is available at just the right time, can be found and memorized easily, is well documented, has an interface that is intuitive to use, and deals correctly with boundary conditions.

So, why are there so many bad APIs around? The prime reason is that, for every way to design an API correctly, there are usually dozens of ways to design it incorrectly. Simply put, it is very easy to create a bad API and rather difficult to create a good one. Even minor and quite innocent design flaws have a tendency to get magnified out of all proportion because APIs are provided once, but are called many times. If a design flaw results in awkward or inefficient code, the resulting problems show up at every point the API is called.
The article went on to provide a list of guidelines. It also offers the following top advice:
APIs should be designed from the perspective of the caller. When a programmer is given the job of creating an API, he or she is usually immediately in problem-solving mode: What data structures and algorithms do I need for the job, and what input and output parameters are necessary to get it done? It's all downhill from there: the implementer is focused on solving the problem, and the concerns of the caller are quickly forgotten.
[...]
This alternative definition requires the implementer to think about the problem in terms of the caller.
This recommendation is not dissimilar to the advice "Your users is not you" or another saying is "there are more users than developers".

The best way to deal with this as alluded in the article is to use TDD (Test Driven Design) to combat this. Let the customer, as suggested by the paper, to write the Unit Test code.

A good API reads correctly and sensibly in the user's code and not in the implementer code. Far too many 'designer' starts the API design from an implementation point of view concerning too earlier with the implementation details, which can lead to a break down in abstraction. APIs are the visible interfaces to these abstractions.

In component technologies that heavily relying on interfaces, it is important the consultants or designers begin the process on the interface.

Using UML diagram is a good starting point as you can quickly play out the scenario to see the effect. At the same time it can document the requirements (Activity/Use case diagrams), how it is to be used (sequence diagrams), or error states (State Diagrams). Then at a later stage, have executable documentation in the form of Unit Test code.

The second most import point in my mind is to strive for 'miminalist' approach:
The fewer types, functions, and parameters an API uses, the easier it is to learn, remember, and use correctly. This minimalism is important. Many APIs end up as a kitchen sink of convenience functions that can be composed of other, more fundamental functions.
This topic was given a detail treatment by Scott Meyers, "When it comes to encapsulation, sometimes less is more.". Or the YAGNI principle.

I once worked for a company that prides itself in designing kitchen-sink API, regardless if it is COM or pure C-style interface. Methods are thrown into any place it can stash regardless if they make sense. Management only interested in getting things out quickly and applying bums-on-seats resource allocation algorithm and the result speaks voluminously of the foolishness of this myopic style, for a long time.

The article touches on the issue confronting the immutability, a term not used by alluded to, which can cause "
crud that, eventually, does more damage than the good it ever did by remaining backward compatible.". When that happens, it is better to start again!

The paper correctly lays blame on education, particularly many developers are not interested or have the patience and time to learn the art and are only interested in getting something 'working' in the IDE. Experience is also overlooked when selecting someone to the task, as identified by the paper.

The example the author used to illustrate a bad API design touches on a topic that .Net/Java should have considered. The author complains about the destruction of the input parameter when using the Select() method and the API method was not dutifully documented.

In C++, you have const member function and const parameters to allow compiler to catch any misuse and to tell user of the consequence of calling the method. But this is sadly not available in .Net/Java. While you have out and ref qualifier but they serve a different purpose.

In .Net/Java, one cannot be sure what would happen to a parameter when you make a call; the method could change the state of the object, as illustrated in the example, it could take a copy of the data (cloning it), or it could hold onto the reference of the object. The latter allows the holding object to change the state of an object it has a reference to and that changes can then be reflected in some other part which has also holds a reference to this object.

All these unknowns require explicit documentation that not too many developers read and can make the API harder to use. My advice is if in doubt, use the "Learning Test" test pattern, which also has an effect of detecting changes in API's behaviour over time.

CLR does not have any support to make sure a method does not alter the state of an object; VB has a language-level construct but it is just a smoke and mirror.

On the topic of cloneable, this can be a double-edged sword. In defense of .Net, the general recommendation is not to implement ICloneable. One such problem is what if an inner class does not support ICloneable and the other is that the caller has no idea if a deep or shallow copy is used. In many way this is the flip side of C++ in which copying an object is a natural behavior.


"Test-Driven Development - by Example" by Kent Beck, Addison Wesley, 2004, Chapter 26, "Learning Test"

Wednesday, April 15, 2009

Google's Unit Testing framework for C++

This is a good site on unit testing in C++ made available by Google. A companion blog discussing C++ Unit Testing issues can be found here.

Thursday, May 17, 2007

Using NUnit without installation

I have been using NUnit 2.2.6 for quite some time now and want to upgrade. The fact that I have to remove my NUnit 2.2.6 and install the new version and then to choose which version, .Net 1 or .Net 2, sort of put me off.

Then I came across this discussion thread on the NUnit forum with a lot of recommendations from Charlie Poole, the key developer/architect of NUnit that completely dispelled my concern and worry. It shows how to 'install' NUnit without installation and supporting both versions.

Essentially, you can use NUnit without installation and you can also have NUnit 2.4.1 for .Net 1.1 and .Net 2 co-existing on your machine without conflict.

This is how to do it:
  1. Download from NUnit project on SourceForge the version you want. Let say NUnit 2.4.1 for .Net 1.1. Instead of downloading the msi version, take the ZIP file (not the source). In this case, it will be NUnit-2.4.1-net-1.1.zip
  2. Create a distinct directory for this file, say NUnit-2.4.1-1.1 and unzip the content to it preserving the folder structure.
  3. Create a short-cut on the desktop to bin\NUnit.exe and label it accordingly.
  4. Then go to the Visual Studio Command prompt and in the bin directory run the GacUtil /I to install the nunit.framework.dll to the GAC.
  5. Repeat steps 1-4 for NUnit 2.4.1 for .Net 2 making sure that this version is placeed in a separate directory.
Because the nunit.framework.dll is in the GAC, as Charlie recommends, VS will not perform local copy.

If you have to develop the project at different places, say at work and at home, where the locations of the NUnit are different, then you are recommended to define the location of the NUnit's bin directory to VS project using the "Reference Path" setting.

Reference Path settings are not saved with the csproj file but is in a user file, csproj.user, and this is not source controlled either. Hence moving the csproj around will not require one to change the csproj file.

Now on my work station I have NUnit 2.2.6 so that old projects continue to work and NUnit 2.4.1 for .Net 1 and .Net 2. The only trouble with this arrangement is that your project source control does not contain a version of the nuit.framework.dll.

This blog message explains why there are two versions of NUnit. In fact, one can simply lauch NUnit.exe with the /framework switch instead of changing the .config file.

Blog Archive