A site devoted to discussing techniques that promote quality and ethical practices in software development.

Monday, March 19, 2007

Delphi.Net generating .Net incompatible call convention for array of parameter

Delphi.Net in Delphi 2006 generates IL code that render the .Net incompatible call convention for methods with array of any type parameter.

For example, if one implements a method like this:

type MyDemo = class( System.Object, IDoSomething )
public
procedure PlayWithData( data : array of System.String );
// ....
end;
where the interface IDoSomething is specified in a different assembly in C# as follows:

public interface IDoSomething {
void PlayWithData( String[] data );
}

In Delphi.W32 language the data parameter thus specified is a non-var parameter and hence it tries to implement the C++'s "const object" semantics preventing the callee from changing the content. That it is the address of object is passed by value and the content is then protected from bring modified.

In C++, this protection is enforced via const methods but in Delphi there is no way to specify a const member functions/procedures. So it is kind of half-baked adoption. In C++, the compiler ensures that only const member functions can be used to access const objects.

In .Net world there is no support for const object in parameter passing. If the parameter is not specified and if the object is a reference object the ECMA standard says (12.4.1.5.1):
For built-in types (integers, floats, etc.) the caller copies the value onto the stack before the call. For objects the object reference (type O) is pushed on the stack.
Therefore, if someone passes an "array of String" to the PlayWithData(), .Net permits the callee to change any element in the array. The only thing .Net does not permit under this condition is for the function to replace the object reference on the stack upon returning from the function.

In fact, if the parameter of type Foo, the callee can change the state of Foo with any methods the callee can access. There is nothing the caller can do to prevent callee from changing it other than passing a copy to the callee.

OK. Now comes the crunch time. In Delphi.Net, the compiler generates code in clear violation of this call convention. If one examines the code generated in PlayWithData(), the first thing Delphi does is to emit code (in pseudo-code style) like this:

data := (data.Clone as String[]);

In other words, before anyone is allowed to deal with the array, the function's code is actually dealing with a cloned version of the array. Hence anything you do to the array element will be discarded by the function upon function exit.

Using a cloned version inside the function clearly will upset the expected behaviour of any .Net assembly because of the lack of const control in the parameter passing. Whether it is good or bad is not the debate in this blog message. Delphi.Net emitting code like that inside the callee will produce incompatible assembly because there is a published ECMA standard governing the convention.

It may be perfectly logical to do that in Delphi.32 where there is no standard at all. But now the world is different.

In fact, this standard actually forces C++ code to disobey the ISO C++ standard when code is compiled with /clr switch when it comes to the treatment of virtual function call convention in the Constructor and Destructor.

Hence it is wise for Delphi to toe the CLI line rather than creating incompatible assemblies.

This means that C# implementation of IDoSomething can result in different behaviour of that using Delphi.Net. This clearly violates ECMA standard introducing incompatibility.

Not only that, such parameter passing convention can result is memory and CPU wastage because of cost of cloning an array can be very high if the array is an array of some objects with deep object model. Remember ICloneable.Clone() is a deep copy function. Not very smart!

For example implementations a swap interface in Delphi and C# will resulted in totally different behaviour. In Delphi.Net, a swap method will require by-ref parameter passing, while C# can be done with a by-value passing.

One should not use By-ref parameter passing convention for this usage because (12.4.1.5.2):
By-ref parameters are the equivalent of C++ reference parameters or PASCAL var parameters: instead of passing as an argument the value of a variable, field, or array element, its address is passed instead; and any assignment to the corresponding parameter actually modifies the corresponding caller’s variable, field, or array
element....

Passing a value by reference requires that the value have a home (see §12.1.6.1) and it is the address of this home that is passed.
In essence, home is defined as (12.1.6.1):
The home of a data value is where it is stored for possible reuse.
Since the function is not asked to redefine a new array of string. It merely allowing it to change any or all of the elements in-place as permitted in ECMA standard and hence the use of pass by-value is correct.

At the moment, I have not explored this parameter passing conflict between Delphi.Net/Pascal and that in ECMA-CLI with other parameter types. But this will be reported in another blog once I have done sufficient investigation.

For the moment, be careful when you are implementing interfaces specified by others and for users of assembly to beware of using assemblies developed in Delphi.Net language. Delphi.Net does not follow the ECMA standard and this can cause integration problem.

Could this be another example of bastardisation of CLR?

This issue is actually more thorny to Borland than their special treatment of assembly accessible member because in that situation Borland uses the Delphi Compiler to restrict Delphi's private member to behave like CLR private when accessed outside the unit but with an IL assembly access qualifier. In that situation, one needs to use their compiler to compile Pascal code and hence they can control this kind of strange behaviour - until they support netmodule.

But in this situation it involves calls to and from any .Net assembly out of the control of Borland's compiler!!

Friday, March 16, 2007

Nearly become another victim of Delphi.Net's feature

Ever since my discovery of Borland's infamous bastardisation of CLR, I have become totally distrustful of the IL code generated by Delphi.Net compiler. My paranoia is not unfounded. Now I will not accept anything from the Delphi.Net compiler without checking them with Lutz Reflector.

The other day as I was developing our own Form Creation code that replaced the need to have compile time construct demanded by

TApplication.CreateForm()

I nearly became a victim of the infamous Delphi's feature disguised as static linkage in .Net.

I got D2006's wizard to create a VCL.Net application project so that it can become a runner using my Delphi class to perform the routine

Application.CreateForm()

where the parameters came from either the configuration file, command line or some other means removing the need to use the name of the type etc. by a call like this:

FormLoader.CreateForm( assemblyName,
unitName,
formName,
formVariableName );
So when I removed the customarily generated start up form as it would be supplied in another package, that action took the reference of Delphi.Vcl and others from the list of referenced assemblies.

When I added a unit to the project and began to use Vcl artefacts in that unit, I did not change the project's references list. The uses statement needed to successfully compile the unit, forces these Vcl types and their supporting types to be transferred into this executable.

Thanksfully my distrust of Borland's output insisting on being validated by Lutz Reflector paid off.

So my next challenge is to develop an automated tool to catch this kind of stupid act.

The recommendation to anyone using Delphi.Net, check your output with Lutz Reflector before each successful compilation! It is a wicked 'feature' only its creator would love it.

An enhanced version of ChDir (or CD) command

Thanks for Raymond Chen's recent blog message on changing directory to UNC path name, I have now discovered that the pushd is actually an enhanced version of XP's cd or Chdir command.

Often people wants to change to a directory on another drive and often people discovers that nothing happens because the person has forgotten to include the /d switch like this:

C:\Windows>cd /d d:\temp

Instead of doing this clumsy way, why not simply replace your cd command with pushd, like this:

C:\Windows>pushd d:\temp

This command essentially performs this command:
pushd . & cd /d d:\temp

The added bonus of using pushd to change directory is that you can use popd to return to your original folder.

Great idea and is extremely useful to create user friendly batch file.

Tuesday, March 13, 2007

These are truly WOW

I am not here to give Microsoft Vista free advertisement but to report two truly deservingly labelled wow materials.

One when wow is used in a superlative sense and this is the concentration of brain power in MSR. Microsoft truly has the most powerful computational machine on earth.

The other worth being labelled wow material is being used in a surprise and rather pitiful sense. It's ironical and laughable that Microsoft's OneCare came in dead last. Perhaps if Microsoft spends as much time on activation scheme and DRM as on their AV, they may not come in last. I guess that's a sure way to ensure no one will pirate 0neCare, more effective than WGA or any activation scheme. While it's a pitifully bad AV, it turns out to be a very "competent" e-mail cleaner ;-)

WOW

Sunday, March 11, 2007

Bridging the two cultures - unmanaged COM and .Net RCW

Microsoft and many literatures on COM Interop programming always seem to portrait an almost seamless integration between unmanaged COM and .Net worlds. In most situations, they are right. However, bridging these two cultures need extra attention than when you are using just one.

I am going to describe two very common scenarios that will cause the unwary a lot of problems.

Using a COM object model in .Net can result in memory leak like behaviour

Many unmanaged COM solutions utilise a staircase like object model to publish the behaviour. You may have a top level object called Application and within it one has Document objects and within Document object you may have Text, etc.

It is very common to find program accessing these sub objects with a dotted expression like this:

application.Document["blah"].Text.Line[2]

When you use them in say VB6 or C++/ATL clients, these environments obey the COM object life-cycle. That is as soon as the ref count for a COM object has dropped to 0, the framework deletes it. The memory/resources are reclaimed deterministically.

Now if you program a COM server with an object model like this in .Net using the RCW that the framework provides you, you will unwittingly be delaying the release of these sub-objects. If your .Net program is a long living kind of program, e.g. 24X7, you will observe behaviour akin to memory leak on the COM Server.

If you examine the .Net equivalent dotted expression in ILDasm, you will be seeing something like this (pseudo-code):

application.Document["blah"] -> temp1
temp1.Text -> temp2
temp2.Line[2] -> temp3

temp1, temp2 and temp3 signify some internally generated intermediate .Net objects of relevant types.

Since they are just RCW wrapping up the corresponding COM objects, we are confronting a situation of bridging the two cultures. On the RCW side the object's life-cycles are totally controlled by the garbage collector while on the COM side, the COM object is controlled by the reference count.

Once these temp1, temp2, and temp3 are of no use they become candidates for garbage collection but that action may not come for a while. In the mean time because they have not been collected, they are holding reference counts on the corresponding COM object preventing them to die releasing the resources they are holding on. These cause the COM server's memory usage to increase as compared with the behaviour when using an unmanaged client.

If you increase memory demand on the .Net side to a point that triggers a garbage collection cycle, you will see an abrupt fall in memory on the COM server and this corresponds to the sudden surge of a batch of COM objects being released at the same time.

So it is not a leak in a strictest sense but it does have that kind of behaviour. If you are confronted with this kind of situation, you need to be confident about it and trust the garbage collector. It will do the job.

If those resources that are being held unnecessary long are not just unmanaged memory but other more precious types, such as database connections or sockets, then you will have a problem because the starvation of resource could be transmitted from unmanaged world to the managed world.

To avoid this kind of situation, you have to restructure your client code to avoid using dotted expression to take control of execution to avoid the generation of those internally generated intermediate objects. In this way, you can control the life cycles of those temporary objects. Often this can result in faster execution performance as well as less demand on the memory.

I have encountered this kind of behaviour in a project I recently been involved with and it is hard to convince the other party that there is nothing wrong.

Passing unmanaged COM object to a managed COM object - side effects

I have encountered a situation like this and it has taken me a week to identify the cause of this problem. Before I explain the architecture to you, the impact on this operation is to prevent an unmanaged COM local server from terminating.

To understand this problem you need to be aware of the architecture that causes this problem.

I have an ATL/C++ local server which hosts a com component let's call this AtlComServer with an interface IAtlComServer. This component embeds a .Net COM component and this component has an interface called IMyDotNetComServer.

When the client calls IAtlComServer.CreateInstance(), it will create an instance of the .Net COM component and call its Setup(), passing to it an object that implements ICallback.

The C++ code in IAtlComServer.CreateInstance() looks something like this (with all error handling removed for brevity):

CComPtr spCallback = CreateInternalObject();
CComVariant v ( spCallback );
hr = spDotNetCom->Setup( ......, v );

spDotNetCom is a variable of type CComPtr<IMyDotNetComServer> previously created.

For the moment consider MyDotNetComServer.Setup() does nothing, including holding onto the ICallback, how can such harmless piece of code can prevent the ATL server from shutting down?

Even if the client take extra care to dispose the AtlComServer with Marshal.ReleaseComObject(), this action only deletes AtlComServer object promptly, but the server still stubbornly refuses to terminate. Why?

To understand this one needs to understand the behaviour of a COM local server. Whenever a COM object is created by this server, that object increments the server's lock count. This is to prevent another com object from initiating a server termination process while there are still living COM objects hosted on this process.

So when an internal (ole non-createable) object is being created by CreateInternalObject(), that object places a server lock on the AtlComServer process. When that is passed across to IMyDotNetComServer.Setup() via the CCW of the .Net Com component, .Net creates a RCW to represent the source com object by incrementing a ref count on the unmanaged COM component.

Even when IMyDotNetComServer.Setup() does nothing with this COM object, like saving it in a member field, that call causes a .Net RCW object to be created and its life-cycle is then managed by the garbage collector. So when the function returns, the RCW object becomes a candidate for garbage collection and that may not come for a long while.

In the mean time, the unmanaged COM object is being held up and since its ref count is not zero, the server lock count is not zero. Because the server lock count is not zero, the local server does not initiate a server shutdown process.

You can follow this kind of action in more detail if you enable the ATL to trace interfaces access by defining the macro _ATL_DEBUG_INTERFACES.

To correct this problem, you have to take action to release the COM object held by the RCW promptly as follows:

public class MyDotNetComServer : IMyDotNetComServer {
public void Setup( ....., object callback ) {
Marshal.ReleaseComObject( callback );
}
}

With this, you will see the server terminates in accordance to the server termination policy.

These two scenarios highlight the danger of programming and using COM in two different cultures without fully aware of the underlying issues.

In the first one, the RCW is holding up the COM object. In the second scenario, the RCW is holding up the COM object which resulted in preventing the server to terminate. The second scenario starts from a CCW and progressing to the generation of an RCW, which ends up holding up the COM server.

Monday, March 5, 2007

Wasting a whole week because of some "Borland feature"

Late last year, by accident I discovered this "Borland feature" whereby Delphi.Net compiler will suck the type and all its necessary supporting types from one assembly and dump them into the assembly you are referencing it, if you do not declare the depending assembly in the project.

One respondent to my blog said:
This is a common mistake that many people make, so don't feel bad. Let me explain. When you referenced the Borland.Vcl.Forms unit and did *not* reference the Borland.Vcl.dll assembly, you linked into your application a new and distinct copy of Borland.Vcl.Forms and all the types therein.

This same exact scenario is possible using C# simply by including the same class into to two different assemblies or in an assembly and an application.
This sounds too illogical as one cannot compare source file binding as in C# to one that transfer chunks of IL code from one assembly to another in complete disregard to the ECMA-CLI specification in relation to the definition of type.

Perhaps Borland/CodeGear can post some example how I could achieve this without source file in C#.

Anyway, I then outlined a number of scenarios that this "Borland feature" will get developers into deep trouble.

I have unfortunately (but fortunately for Borland) that I missed a few. My colleague, a Delphi programming veteran of more than 10 years, fell into one that I did not reported and it took 2 developers one whole week to unravel this mystery.

In the end, the trusted Type.AssemblyQualifiedName confirmed the minefield.

The scenario was compounded by another Type-dropping bug. The .Net architecture we tried to realise involved a number of assemblies (some are packages while others are executable assemblies being use as if they are dll assemblies).

This Type-dropping bug for some reason causes the types to be rammed into the assembly referencing the types via the Delphi Metadata as oppose to the IL metadata. Lutz Reflector confirmed the bug had discarded the expected types.

Because Borland's relying on Delphi metadata as opposed to IL metadata, Delphi.Net compiled and produced assemblies. But at run time, CLR ruled the world and the assemblies thus produced bite the dust.

All without informing the user of this wonderful feature during compilation. This stealthiness is mainly my complaint as Borland is free to add whatever it sees fit to violate ECMA creating .Net incompatible assemblies. It has lots of warning about platform depending features being used.

But it must also allow and in fact encourage the user to choose to comply with ECMA so that what one produces will play well in the .Net world. Their compiler does not have a switch to turn this feature on or off.

This kind of stupidity, which benefits very few, caused 2 senior developers' one whole week and I am sure there are many other victims like me. We knocked up a C#/VS2003 solution that mimicked the architecture in less than 15 minutes to confirm the architecture was sound.

Of course VS2003/C# does not have this Type-dropping bug. I could as well use C# Builder to perform this validation check. It is not a VS verse C# Builder contest.

One of the alleged benefit of this "Borland feature" is to allow developer to choose how to deploy the application - one executable and a bunch of DLL's or all in one.

While I do not dispute this alleged benefit in Delphi.W32 land (there is no standard of any sort in that area), doing this in CLI is not only wrong, in clear violation of ECMA standard, but also totally unnecessary.

The reason is that you could either use ILMerge or Borland's own version DILMerge. It is better for Borland to discard this dangerous practice, to promote the use of these tools and to encourage Delphi developers to write proper .Net assemblies.

Bug in Delphi.Net in handling class function of Char

Every time, when I tried to do some .Net programming in Delphi, I often run into compiler and/or runtime bug by Delphi.Net (D2006) and it is getting to the annoying point. I must have missed something. Should I simply program Delphi as if .Net does not exist? If I do that would I have better experience? Weird!

I did not discover this bug but I was asked why one needs an object instance to use Char's class procedure or function (in C#, they are known as static function or in UML, they are the classified function).

At first, I did not know the answer thinking that Delphi.Net may be using TChar or something like that. On deeper investigation, there is no such type in Delphi.Net.

So I cooked up a sample application so that I can examine the IL code it generates - very important when dealing with Delphi.Net output. As previously blogged, while Borland can spin its own flavour, but ultimately at run time, it has to obey the CLR.

System.Char in .Net is a structure containing mostly static methods and by definition these methods can be accessed without the need to use an object instance.

The following code fragment illustrates the compiler bug:

00000001 var
00000002 c : System.Char;
00000003 cc : Char;
00000004 b : system.Boolean;
00000005 begin
00000006 c := 'a';
00000007 cc := 'b';
00000008
00000009 Console.WriteLine( 'Is it a Char? ',
00000010 System.Char.IsLetter( c ).ToString() );
00000011 Console.WriteLine( 'Type of cc is {0}',
00000012 cc.GetType().AssemblyQualifiedName );
00000013 Console.WriteLine( 'Type of Char is {0}',
00000014 Typeof(Char).AssemblyQualifiedName );
00000015 Console.WriteLine( 'Char is {0}', cc.ToString() );
00000016 // b := Char.IsLetter( 'b' );
00000017 end;
Here is the output from the above code fragment:

Result:
Is it a Char?
Type of cc is System.Char, mscorlib, Version=1.0.5000.0, Culture=neutral,
PublicKeyToken=b77a5c561934e089
Type of Char is System.Char, mscorlib, Version=1.0.5000.0, Culture=neutral,
PublicKeyToken=b77a5c561934e089
Char is b
If you uncomment Line 16 above, the code fragment will not compile complaining that it needs a object reference. Lines 11 & 13 clearly shows that Delphi.Net's Char is simply System.Char and hence it should allow the syntax in Line 16 to be used.

Insisting on having a object reference just to use static method is just plain wrong!

Another tip for using Delphi.Net, when in doubt, always print out the Type.AssemlyQualifiedName as Delphi.Net has a habit of giving you this Borland Special Treatment.

As my colleague said, Char is one of the most basic thing in any programming language, why can Borland handle this without error? I am not a Borland employee and I do not have an answer for that other than to tell him that it is another Delphi.Net compiler bug.

Blog Archive