A site devoted to discussing techniques that promote quality and ethical practices in software development.

Friday, March 16, 2007

Nearly become another victim of Delphi.Net's feature

Ever since my discovery of Borland's infamous bastardisation of CLR, I have become totally distrustful of the IL code generated by Delphi.Net compiler. My paranoia is not unfounded. Now I will not accept anything from the Delphi.Net compiler without checking them with Lutz Reflector.

The other day as I was developing our own Form Creation code that replaced the need to have compile time construct demanded by

TApplication.CreateForm()

I nearly became a victim of the infamous Delphi's feature disguised as static linkage in .Net.

I got D2006's wizard to create a VCL.Net application project so that it can become a runner using my Delphi class to perform the routine

Application.CreateForm()

where the parameters came from either the configuration file, command line or some other means removing the need to use the name of the type etc. by a call like this:

FormLoader.CreateForm( assemblyName,
unitName,
formName,
formVariableName );
So when I removed the customarily generated start up form as it would be supplied in another package, that action took the reference of Delphi.Vcl and others from the list of referenced assemblies.

When I added a unit to the project and began to use Vcl artefacts in that unit, I did not change the project's references list. The uses statement needed to successfully compile the unit, forces these Vcl types and their supporting types to be transferred into this executable.

Thanksfully my distrust of Borland's output insisting on being validated by Lutz Reflector paid off.

So my next challenge is to develop an automated tool to catch this kind of stupid act.

The recommendation to anyone using Delphi.Net, check your output with Lutz Reflector before each successful compilation! It is a wicked 'feature' only its creator would love it.

An enhanced version of ChDir (or CD) command

Thanks for Raymond Chen's recent blog message on changing directory to UNC path name, I have now discovered that the pushd is actually an enhanced version of XP's cd or Chdir command.

Often people wants to change to a directory on another drive and often people discovers that nothing happens because the person has forgotten to include the /d switch like this:

C:\Windows>cd /d d:\temp

Instead of doing this clumsy way, why not simply replace your cd command with pushd, like this:

C:\Windows>pushd d:\temp

This command essentially performs this command:
pushd . & cd /d d:\temp

The added bonus of using pushd to change directory is that you can use popd to return to your original folder.

Great idea and is extremely useful to create user friendly batch file.

Tuesday, March 13, 2007

These are truly WOW

I am not here to give Microsoft Vista free advertisement but to report two truly deservingly labelled wow materials.

One when wow is used in a superlative sense and this is the concentration of brain power in MSR. Microsoft truly has the most powerful computational machine on earth.

The other worth being labelled wow material is being used in a surprise and rather pitiful sense. It's ironical and laughable that Microsoft's OneCare came in dead last. Perhaps if Microsoft spends as much time on activation scheme and DRM as on their AV, they may not come in last. I guess that's a sure way to ensure no one will pirate 0neCare, more effective than WGA or any activation scheme. While it's a pitifully bad AV, it turns out to be a very "competent" e-mail cleaner ;-)

WOW

Sunday, March 11, 2007

Bridging the two cultures - unmanaged COM and .Net RCW

Microsoft and many literatures on COM Interop programming always seem to portrait an almost seamless integration between unmanaged COM and .Net worlds. In most situations, they are right. However, bridging these two cultures need extra attention than when you are using just one.

I am going to describe two very common scenarios that will cause the unwary a lot of problems.

Using a COM object model in .Net can result in memory leak like behaviour

Many unmanaged COM solutions utilise a staircase like object model to publish the behaviour. You may have a top level object called Application and within it one has Document objects and within Document object you may have Text, etc.

It is very common to find program accessing these sub objects with a dotted expression like this:

application.Document["blah"].Text.Line[2]

When you use them in say VB6 or C++/ATL clients, these environments obey the COM object life-cycle. That is as soon as the ref count for a COM object has dropped to 0, the framework deletes it. The memory/resources are reclaimed deterministically.

Now if you program a COM server with an object model like this in .Net using the RCW that the framework provides you, you will unwittingly be delaying the release of these sub-objects. If your .Net program is a long living kind of program, e.g. 24X7, you will observe behaviour akin to memory leak on the COM Server.

If you examine the .Net equivalent dotted expression in ILDasm, you will be seeing something like this (pseudo-code):

application.Document["blah"] -> temp1
temp1.Text -> temp2
temp2.Line[2] -> temp3

temp1, temp2 and temp3 signify some internally generated intermediate .Net objects of relevant types.

Since they are just RCW wrapping up the corresponding COM objects, we are confronting a situation of bridging the two cultures. On the RCW side the object's life-cycles are totally controlled by the garbage collector while on the COM side, the COM object is controlled by the reference count.

Once these temp1, temp2, and temp3 are of no use they become candidates for garbage collection but that action may not come for a while. In the mean time because they have not been collected, they are holding reference counts on the corresponding COM object preventing them to die releasing the resources they are holding on. These cause the COM server's memory usage to increase as compared with the behaviour when using an unmanaged client.

If you increase memory demand on the .Net side to a point that triggers a garbage collection cycle, you will see an abrupt fall in memory on the COM server and this corresponds to the sudden surge of a batch of COM objects being released at the same time.

So it is not a leak in a strictest sense but it does have that kind of behaviour. If you are confronted with this kind of situation, you need to be confident about it and trust the garbage collector. It will do the job.

If those resources that are being held unnecessary long are not just unmanaged memory but other more precious types, such as database connections or sockets, then you will have a problem because the starvation of resource could be transmitted from unmanaged world to the managed world.

To avoid this kind of situation, you have to restructure your client code to avoid using dotted expression to take control of execution to avoid the generation of those internally generated intermediate objects. In this way, you can control the life cycles of those temporary objects. Often this can result in faster execution performance as well as less demand on the memory.

I have encountered this kind of behaviour in a project I recently been involved with and it is hard to convince the other party that there is nothing wrong.

Passing unmanaged COM object to a managed COM object - side effects

I have encountered a situation like this and it has taken me a week to identify the cause of this problem. Before I explain the architecture to you, the impact on this operation is to prevent an unmanaged COM local server from terminating.

To understand this problem you need to be aware of the architecture that causes this problem.

I have an ATL/C++ local server which hosts a com component let's call this AtlComServer with an interface IAtlComServer. This component embeds a .Net COM component and this component has an interface called IMyDotNetComServer.

When the client calls IAtlComServer.CreateInstance(), it will create an instance of the .Net COM component and call its Setup(), passing to it an object that implements ICallback.

The C++ code in IAtlComServer.CreateInstance() looks something like this (with all error handling removed for brevity):

CComPtr spCallback = CreateInternalObject();
CComVariant v ( spCallback );
hr = spDotNetCom->Setup( ......, v );

spDotNetCom is a variable of type CComPtr<IMyDotNetComServer> previously created.

For the moment consider MyDotNetComServer.Setup() does nothing, including holding onto the ICallback, how can such harmless piece of code can prevent the ATL server from shutting down?

Even if the client take extra care to dispose the AtlComServer with Marshal.ReleaseComObject(), this action only deletes AtlComServer object promptly, but the server still stubbornly refuses to terminate. Why?

To understand this one needs to understand the behaviour of a COM local server. Whenever a COM object is created by this server, that object increments the server's lock count. This is to prevent another com object from initiating a server termination process while there are still living COM objects hosted on this process.

So when an internal (ole non-createable) object is being created by CreateInternalObject(), that object places a server lock on the AtlComServer process. When that is passed across to IMyDotNetComServer.Setup() via the CCW of the .Net Com component, .Net creates a RCW to represent the source com object by incrementing a ref count on the unmanaged COM component.

Even when IMyDotNetComServer.Setup() does nothing with this COM object, like saving it in a member field, that call causes a .Net RCW object to be created and its life-cycle is then managed by the garbage collector. So when the function returns, the RCW object becomes a candidate for garbage collection and that may not come for a long while.

In the mean time, the unmanaged COM object is being held up and since its ref count is not zero, the server lock count is not zero. Because the server lock count is not zero, the local server does not initiate a server shutdown process.

You can follow this kind of action in more detail if you enable the ATL to trace interfaces access by defining the macro _ATL_DEBUG_INTERFACES.

To correct this problem, you have to take action to release the COM object held by the RCW promptly as follows:

public class MyDotNetComServer : IMyDotNetComServer {
public void Setup( ....., object callback ) {
Marshal.ReleaseComObject( callback );
}
}

With this, you will see the server terminates in accordance to the server termination policy.

These two scenarios highlight the danger of programming and using COM in two different cultures without fully aware of the underlying issues.

In the first one, the RCW is holding up the COM object. In the second scenario, the RCW is holding up the COM object which resulted in preventing the server to terminate. The second scenario starts from a CCW and progressing to the generation of an RCW, which ends up holding up the COM server.

Monday, March 5, 2007

Wasting a whole week because of some "Borland feature"

Late last year, by accident I discovered this "Borland feature" whereby Delphi.Net compiler will suck the type and all its necessary supporting types from one assembly and dump them into the assembly you are referencing it, if you do not declare the depending assembly in the project.

One respondent to my blog said:
This is a common mistake that many people make, so don't feel bad. Let me explain. When you referenced the Borland.Vcl.Forms unit and did *not* reference the Borland.Vcl.dll assembly, you linked into your application a new and distinct copy of Borland.Vcl.Forms and all the types therein.

This same exact scenario is possible using C# simply by including the same class into to two different assemblies or in an assembly and an application.
This sounds too illogical as one cannot compare source file binding as in C# to one that transfer chunks of IL code from one assembly to another in complete disregard to the ECMA-CLI specification in relation to the definition of type.

Perhaps Borland/CodeGear can post some example how I could achieve this without source file in C#.

Anyway, I then outlined a number of scenarios that this "Borland feature" will get developers into deep trouble.

I have unfortunately (but fortunately for Borland) that I missed a few. My colleague, a Delphi programming veteran of more than 10 years, fell into one that I did not reported and it took 2 developers one whole week to unravel this mystery.

In the end, the trusted Type.AssemblyQualifiedName confirmed the minefield.

The scenario was compounded by another Type-dropping bug. The .Net architecture we tried to realise involved a number of assemblies (some are packages while others are executable assemblies being use as if they are dll assemblies).

This Type-dropping bug for some reason causes the types to be rammed into the assembly referencing the types via the Delphi Metadata as oppose to the IL metadata. Lutz Reflector confirmed the bug had discarded the expected types.

Because Borland's relying on Delphi metadata as opposed to IL metadata, Delphi.Net compiled and produced assemblies. But at run time, CLR ruled the world and the assemblies thus produced bite the dust.

All without informing the user of this wonderful feature during compilation. This stealthiness is mainly my complaint as Borland is free to add whatever it sees fit to violate ECMA creating .Net incompatible assemblies. It has lots of warning about platform depending features being used.

But it must also allow and in fact encourage the user to choose to comply with ECMA so that what one produces will play well in the .Net world. Their compiler does not have a switch to turn this feature on or off.

This kind of stupidity, which benefits very few, caused 2 senior developers' one whole week and I am sure there are many other victims like me. We knocked up a C#/VS2003 solution that mimicked the architecture in less than 15 minutes to confirm the architecture was sound.

Of course VS2003/C# does not have this Type-dropping bug. I could as well use C# Builder to perform this validation check. It is not a VS verse C# Builder contest.

One of the alleged benefit of this "Borland feature" is to allow developer to choose how to deploy the application - one executable and a bunch of DLL's or all in one.

While I do not dispute this alleged benefit in Delphi.W32 land (there is no standard of any sort in that area), doing this in CLI is not only wrong, in clear violation of ECMA standard, but also totally unnecessary.

The reason is that you could either use ILMerge or Borland's own version DILMerge. It is better for Borland to discard this dangerous practice, to promote the use of these tools and to encourage Delphi developers to write proper .Net assemblies.

Bug in Delphi.Net in handling class function of Char

Every time, when I tried to do some .Net programming in Delphi, I often run into compiler and/or runtime bug by Delphi.Net (D2006) and it is getting to the annoying point. I must have missed something. Should I simply program Delphi as if .Net does not exist? If I do that would I have better experience? Weird!

I did not discover this bug but I was asked why one needs an object instance to use Char's class procedure or function (in C#, they are known as static function or in UML, they are the classified function).

At first, I did not know the answer thinking that Delphi.Net may be using TChar or something like that. On deeper investigation, there is no such type in Delphi.Net.

So I cooked up a sample application so that I can examine the IL code it generates - very important when dealing with Delphi.Net output. As previously blogged, while Borland can spin its own flavour, but ultimately at run time, it has to obey the CLR.

System.Char in .Net is a structure containing mostly static methods and by definition these methods can be accessed without the need to use an object instance.

The following code fragment illustrates the compiler bug:

00000001 var
00000002 c : System.Char;
00000003 cc : Char;
00000004 b : system.Boolean;
00000005 begin
00000006 c := 'a';
00000007 cc := 'b';
00000008
00000009 Console.WriteLine( 'Is it a Char? ',
00000010 System.Char.IsLetter( c ).ToString() );
00000011 Console.WriteLine( 'Type of cc is {0}',
00000012 cc.GetType().AssemblyQualifiedName );
00000013 Console.WriteLine( 'Type of Char is {0}',
00000014 Typeof(Char).AssemblyQualifiedName );
00000015 Console.WriteLine( 'Char is {0}', cc.ToString() );
00000016 // b := Char.IsLetter( 'b' );
00000017 end;
Here is the output from the above code fragment:

Result:
Is it a Char?
Type of cc is System.Char, mscorlib, Version=1.0.5000.0, Culture=neutral,
PublicKeyToken=b77a5c561934e089
Type of Char is System.Char, mscorlib, Version=1.0.5000.0, Culture=neutral,
PublicKeyToken=b77a5c561934e089
Char is b
If you uncomment Line 16 above, the code fragment will not compile complaining that it needs a object reference. Lines 11 & 13 clearly shows that Delphi.Net's Char is simply System.Char and hence it should allow the syntax in Line 16 to be used.

Insisting on having a object reference just to use static method is just plain wrong!

Another tip for using Delphi.Net, when in doubt, always print out the Type.AssemlyQualifiedName as Delphi.Net has a habit of giving you this Borland Special Treatment.

As my colleague said, Char is one of the most basic thing in any programming language, why can Borland handle this without error? I am not a Borland employee and I do not have an answer for that other than to tell him that it is another Delphi.Net compiler bug.

Sunday, February 25, 2007

Road map from Windows to Linux

Sick of being restricted of what you can and cannot be done with Vista and sick of having to buy special hardware to run Vista, perhaps this road map from Windows to Linux may be just what you need.

The draconian restriction of not allowing the cheaper version of Vista to be installed in virtual machines is cleanly mad.

Blog Archive