OFX Encoding bug - This time from CBA

Further to my discover of a financial institution failing to generate OFX standard compliant OFX file, while I was helping a friend to download bank statements from the Commonwealth Bank of Australia's Netbank facility, I noticed something not quite right.

One would think CBA being one of the largest banks in Australia would have hired the most competent developers to code their download statement in OFX format. Well it seems their developers have failed the test of standard software development practice producing ridiculous results, just like others.

In CBA's case, it seems that they have not read section 3.2.8.2 of OFX 1.02 specification for representing datetime. 

In 4 files out of 13 I downloaded, in a space of a day, the elements <SONRS><DTSERVER>, <LEDGERBAL><DTASOF> and <AVAILBAL><DTASOF> containing ridiculous dates and here are the figures:

File-01:
20120937021837
20120937021837
20120937021837

File-02:
20120945030545
20120945030545
20120945030545

File-03:
20120950015650
20120950015650
20120950015650

File-04:
20120948021348
20120948021348
20120948021348

According to the standard, section 3.2.8.2, which says:

Tags specified as type date or datetime and generally starting with the letters “DT” accept a fully formatted date-time-timezone string. For example, “19961005132200.124[-5:EST]” represents October 5, 1996, at 1:22 and 124 milliseconds p.m., in Eastern Standard Time. This is the same as 6:22 p.m. Greenwich Mean Time (GMT).

Date and datetime also accept values with fields omitted from the right. They assume the following defaults if a field is missing:

Note that times zones are specified by an offset and optionally, a time zone name. The offset defines the time zone.

Take care when specifying an ending date without a time. If the last transaction returned for a bank statement download was Jan 5 1996 10:46 a.m. and if the was given as just Jan 5, the transactions on Jan 5 would be resent. If results are available only daily, then just using dates and not times will work correctly.

NOTE: Open Financial Exchange does not require servers or clients to use the full precision specified. However, they are REQUIRED to accept any of these forms without complaint.

Some services extend the general notion of a date by adding special values, such as “TODAY.” These special values are called “smart dates.” Specific requests indicate when to use these extra values, and list the tag as having a special data type.

So how can any developer possibly produce 37, 45, 50 and 48 days in September? Do they actually test their code at all? I wonder if these people has actually read the standard before coding?

Thankfully GnuCash does not use these elements otherwise it will prevent the download data from being imported into GnuCash. 

Microsoft's System.Xml.Serialization.XmlSerializer generates uncompilable code for some xs:double default values

Consider the following valid schema:

<xs:schema id="MyDemoSchema"
    targetNamespace="http://tempuri.org/MyDemoSchema.xsd"
    elementFormDefault="qualified"
    xmlns="http://tempuri.org/MyDemoSchema.xsd"
    xmlns:mstns="http://tempuri.org/MyDemoSchema.xsd"
    xmlns:xs="http://www.w3.org/2001/XMLSchema"
>
  <xs:complexType name="Foo">
    <xs:sequence>
      <xs:element name="Name" type="xs:string" minOccurs="0" maxOccurs="1" />
      <xs:element name="SomeNumber" type="xs:double" 
                  minOccurs="0" maxOccurs="1" 
                  default="NaN" />
    </xs:sequence>
  </xs:complexType>

  <xs:element name="Foo" type="Foo" />
</xs:schema>

The minute you execute the following piece of code:

Foo f = new Foo();
XmlSerializer ser = new XmlSerializer( typeof(Foo) ); // ---- Error CS0103 ---

will generate a CS0103 compilation error. This is caused by the CLR runtime in which it will at that moment generates a temporary serialization assembly that actually performs the serialization and deserialization process.

It is the code generation process that fails to handle W3C acceptable xs:double values such as NaN, INF and -INF producing uncompilable code.

In brief, but will provide a more detail expose below, is that it generates code like this:

   if( x != NaN ) { < --- causes CS0103
     // do something 
   }

Which naturally does not compile and it generates the same error message as when the XmlSerializer is instantiated. You can't even use the logical operator to test if the value of System.Double is Double.NaN.

This is the exact code fragment of the generated code causing the CS0103:

if (((global::System.Double)o.@SomeNumber) != NaN) {
    WriteElementStringRaw(@"SomeNumber", @"http://tempuri.org/MyDemoSchema.xsd", 
    System.Xml.XmlConvert.ToString((global::System.Double)((global::System.Double)o.@SomeNumber)));
}

There is no way this line can compile. It has to be corrected like this:

if ( !System.Double.IsNaN ((global::System.Double)o.@SomeNumber)  ) {
    WriteElementStringRaw(@"SomeNumber", @"http://tempuri.org/MyDemoSchema.xsd", 
    System.Xml.XmlConvert.ToString((global::System.Double)((global::System.Double)o.@SomeNumber)));
}

Detail Discussion and work around

To see how this manifested into a runtime error, you can use sgen.exe together with the /keep switch to look at what happen. The catch-22 is that if your schema's default value is NaN, sgen.exe will fail to generate the serialization dll because the generation process generates the above mentioned uncompilable code and you can't see the generated code.

So in order to see what is happening, you replace the default value with a double that you can recognised. In my experiment, I used -5.55.

Once your schema is modified, xsd.exe is used to generate C# classes that are bundled into an assembly, you can then use sgen.exe with the /keep switch to generate the serialization dll together with the source file. It also generates a file with the extension .cmdline which is essentially the response file for CSC.

Using our default value as search string, you can quickly locate the potentially offending piece of code. Below are the steps to prove that code generated by the serialization process is faulty and how to rectify it.

Now correct the line where it checks for -5.55 to using System.Double.IsNaN() as shown above.

The next step is to rebuild the serialization dll and to do that you need to modify the schema to change the default value back to NaN.

Then you use XSD to generate the class and rebuild your assembly.

Next you modify the cmdline file as follows:

• remove the setting where it references System.Xml.dll to avoid duplicate reference compilation error. 
• remove the /D: _DYNAMIC_XMLSERIALIZER_COMPILATION 
• to aid experimentation change the /debug- to /debug.

Check the response file for the location of the CS file. You should execute this response file in that directory. For example if the source file is like this "OutTestDir\5fm3rmet.0.cs", you should run your CSC from the parent directory of "OutTestDir".

Next is to compile and generate the serialization dll which by now should have the offending line corrected using Double.IsNaN().

Next you modify the project for the dll that once called XmlSerializer constructor by

• adding a reference to the serialization dll. 
• then replace following piece of code where you instantiate an instance of XmlSerializer:

Foo f = new Foo();
StringBuilder buffer = new StringBuilder();
using( XmlWriter writer = XmlWriter.Create( buffer, null ) )
{
   XmlSerializer ser = new XmlSerializer( typeof(Foo) ); // <-- to be replaced
   ser.Serialize( writer, f );
}

With this:

Foo f = new Foo();
StringBuilder buffer = new StringBuilder();
using( XmlWriter writer = XmlWriter.Create( buffer, null ) )
{
   Microsoft.Xml.Serialization.GeneratedAssembly.FooSerializer ser = 
       new Microsoft.Xml.Serialization.GeneratedAssembly.FooSerializer();
   // XmlSerializer ser = new XmlSerializer( typeof(Foo) ); // <-- to be replaced
   ser.Serialize( writer, f );
}

If you have the PDB, you can easily step through the code to convince yourself that your fix indeed does work.

Let's hope Microsoft will fix this bug which is in .Net 4. May be .Net 5 has fixed this?

Android 'smartphone' cannot create bi-weekly events

Further to my reporting on this issue on my Samsung Galaxy Note, I went to dig a bit deeper to see whose fault it is - Samsung or its underlying operating system Android.

Just to recap, here are the only options offered to me when I try to create a repeatable events in Galaxy Note:

What is so difficult with 'Bi-weekly' - 2 x 7 days interval?

It turns out that this defect appears to be caused by the underlying operating system - Android - and is found in HTC, Samsung Nexus, Samsung S3 and Samsung Galaxy Note. The only thing that is common amount them is the Android operating system.

Anyone with an Android phone can try their hands to see if they can create an event that repeats every 2 weeks on the Calendar/Appointment manager that is shipped with their machine.

It begs the question why these developers and product managers that oversee their development could be so ignorant of this basic business requirement. Have they not used any other calendar programs in their life? Or are they so arrogant that they just want to be different for the sake of being different?

Perhaps they are so insular from living their entire life in Google Calendar, which could do this, that they do not understand that there are users out there that do not live like them? If that is the case, it is worth them to remember this simple phase: "There are more users than developers"

Well Apple has that covered in iOS, which is more business friendly. Perhaps Apple should run a TV advertisement much like the Mac Vs PC version to highlight Android phone's failure to support this basic business requirement.