In search of a Backup Utility for Windows 7

While people reading this title may be wondering if I have missed the much touted "Backup and Restore" features in Windows 7 and wondering why I need to search for (better) Backup Utility?

Let me describes the deficiency in Windows 7's "Backup and Restore" utility before I will describe my search result.

Microsoft has succeeded in taking a highly capable program called NTBackup and destroying it for the sake of some eye-candy screen. The Win7 backup is so slooooow that any ZIP program will beat it hands down. Not only that, the eye-candy stuff lacks any really useful progress information. It does not even bother to tell you how many files it has picked up, which file is processing, and the total size of the files being backup (until the whole thing is finished) or expected time to take, given that is a best-guess. Apart from some pretty looking screens, the user-interface is totally unintuitive and functionality lacking.

If you have not seen an industrial strength real back up utility, I suggest you fire up a copy of NTBackup in XP Pro or installed it in Home Edition.

One of the basic needs of a back up utility is to be able to add the back up information to what is already there like keeping revision so that you can go back several generations to restore the data.

Not only that a backup utility is as good as its restoration capability. It has to be able to restore the data accurately and precisely including restoring the original ACLS for the NTFS files and folders. Failure to do that can cause major problem and security risk. Imagine you are backing up the profile areas for a machine and has to do a disaster recovery only to discover that the resulting ACLS are all wrong!

A good backup utility also should ensure that the person running this must be a backup operator or one with the SeBackupPrivilege, to perform backup and SeRestorePrivilege, to perform restoration.

The other features commonly found in industrial strength backup utility is to be able to perform incremental backup so to reduce the backup volume. This is important when you are doing a daily backup.

Armed with these demands, I evaluated the following free backup programs: NTBackup, FBackup, and Comodo Backup.

Sadly only NTBackup managed to perform flawlessly managing to replicate the ACLS perfectly. Here is a screen shot of the source (left hand window) and restored folder's ACLS (right hand window):

I restored the material to a different directory to check the restoration process. As a consequence, NTBackup is being used as a benchmark against which the other utilities are compared.

FBackup4 Version 4.4 Build 207
This is a very simple to use free 'backup' utility. The good part of this one is its ability to manage several backup instructions as jobs in the program. It automatically increments the backup volume file name and the backup volume is in ZIP format. It is relatively fast.

However, it is a naive implementation of a backup utility. At best, it can only be classified as a ZIP program with a purpose-built user-interface.

Here is the screen shot of restoring the user's profile to an alternate location. Once again the left hand window shows the security settings of the original top level user profile folder and the right hand window shows that for the restored materials:

This clears shows the restored materials have fewer privileges than the original materials failing being a competent back up utility. This is a simulation of profile restoration.

Comodo Backup Version 2.2.127000.12
The user-interface is prettier than FBackup but it is also more confusing and functional lacking. With its user-interface, a user cannot workout how to get the program to remember the backup instructions so that one can reuse them again. However there are several ways this program that this program can do that, albeit not as intuitive as that for FBackup.

During the composition of the backup instruction using the wizard, you can use the Schedule definition to remember your instructions even if you do not want to do scheduled backup. You simply change the type to manual backup. Strange logic.

The other way is to export your backup instructions to a file which contains the command-line arguments corresponding to your instructions. With this file you can then use the /script command-line directive to supply the script file when launching the backup utility.

This utility has some slick facility allowing you to define the backup volume file format - such as including date, time etc. They call them macros and they are not available in FBackup. It also has facility for you to define the level of compression you want during backup.

Performance of this program is very good. However it suffers the same problem as in FBackup when it fails to reproduce precisely the ACLS as shown below:

Conclusion
These backup utilities are not really backup tools but specialized ZIP programs. It is not just because they are using ZIP format but because they left out the ACLS what they need to use to restore them

While NTBackup is available official in XP Pro and optionally in XP Home Edition, Microsoft has provided a cut-down version of this tool for Vista/Windows 7, called "Windows NT Backup - Restore Utility", so that user can restore from NTBackup volumes.

However, NTBackup has been known to run fine in Vista and in Windows 7 (and here). Since the 'installation' of NTBackup is so low impact, I will definitely give that a try. The big remaining question is: At what time in the future that Windows development will render our trusty friend not operable?

Is it wise to break convention - the wildcard convention

I have been a long term user of 7Za.exe, the command line version 7-ZIP, and recently have been using it to back up my Subversion repositories. Like many that use this kind of programs, one is too trusting to believe that it will honor the same convention as other CMD commands to archive all files, with the exception of those files in used, when one issues a command line like this:

7za a -tzip Test.zip MyWork\*.* -r

To many long time users of Windows Dos Prompt, this is the standard way to instruct a program to process all files; the Del, Attrib, Copy, XCopy, Cacls, Dir, RoboCopy, and Rar all conform to this convention and hence one naturally assumes that 7Za.exe would naturally honor this.

Not so, and I learned this the painful way. According to the help file of 7-Zip:

7-Zip doesn't follow the archaic rule by which *.* means any file. 7-Zip treats *.* as matching the name of any file that has an extension. To process all files, you must use a * wildcard.

Well, while it is admirable for someone to make a partial stand of this 'archaic rule', it is dangerous to use the same syntax and then silently producing a different result set. It is like switching the active and neutral wires in the electric wiring just because someone took exception to the color coding convention of the wire.

The fact remains that before 7-zip comes along to Windows, that convention has already been established and entrenched, way before it was mistakenly claimed by 7-Zip as introduced in Win 95. There are millions of users accustomed to this convention whether it is logical or illogical; this has become their second nature; it is like arguing whether it is logical or illogical to drive on the left hand side or right hand side of the road; a lone group of dissenting drivers taking a stand can not only play havoc on our roads but producing fatalities.

A convention has been established and people driving on public road has therefore to conform to it, like it or not. In US, all vehicles driving in a mine drives on the opposite side to those on the public road. The change of convention was explicitly stated, for reason sound in mining operations, and drivers are deliberately made to go through a change over section.

7Za however, did not do such a thing. It took *.* to mean 'all files must have extensions' in contrast to the Windows convention which means 'all files with or without extensions', which is understood by millions or trillions.

It is not a dispute of 7Za for being correct. It is raised here for its dangerous and irresponsible practice of flaunting a convention while using the same syntax.

For example,

7za a -tzip Test.zip MyWork\*.* -r

Should pick up all files in a Subversion repository regardless if the files contain extensions of not; many Subversion files do not have extensions. Rar with this command picks up all files honoring the Windows convention:

Rar a -r Test.rar MyWork\*.*

This is not only wise but also responsible realizing the consequence for failing to and taking a stand only brings at best hollow victory and wrath from users at the worst; such is the case with 7za now.

According to the Windows API and treatments of wildcard characters, there is no way to specify collecting files only with extensions. For example using Dir as an example:

Dir *.* /s /b

and

Dir * /s /b

produce the same result: a list of files with or without extensions. The second form is 7Za's way of specifying any file with or without extension but accepting the first form and producing a totally different result set.

Dir *. /s /b

produces a list of file without extensions. But there is no wildcard syntax to say 'file only with extension'.

As a result of 7-ZIP hollow stand against a convention entrenched in more users of Windows than 7za, it has successfully dislodged people's trust on this program. Archiver should behave much like a copy/xcopy commands; changing the operation with the same syntax is extremely dangerous and developer should not toy with this kind ideological stand in an important tool.

My ill-placed trust on 7za has caused me losing the collection of my Subversion repositories. It is an expensive loss and 7Za's ideological stand against an illogical convention is equally illogical resulting in real loss; what is 7-ZIP hoping to achieve? It has hardly won any friend!

The result is a total distrust of this tool. While I can use Subversion's command to ascertain the integrity of the restored repository, other archives produced by 7Za lack such detection and hence there remains an unknown number of imperfect archives.

I don't discourage 7za to take this kind of admirable stand but it should be selected by users. The default should always follow the convention of the OS in which it is deployed into. 7Za has already established this kind of overriding switches/options and its admirable attempt to correct the convention should only be selected when user makes the choice.

If that is a Unix convention, then either build 7Za to be a Posix conforming program that runs in Windows' Posix subsystem, in which case 7Za can even use case-sensitive file names or have a switch to turn on Unix convention.

This should be a real-life example of the danger of developers failing to conform to entrenched convention, no matter how 'archaic' or illogical that is. The first occupancy rule applies here!

For me, 7Za is now being banished to the recycled-bin as it is too dangerous to use tools that do not conform to convention. It will not earn my recommendation for sure.

JVC's MOD file is actually a MPEG-2 file.

Recently, I have acquired a JVC hard disk digital video camera and it creates files with .MOD extensions.

So I am interested to find out what kind of files they are.

Searching the internet, I found this great tool called GSpot. Not only does it tells you what format (video + audio) plus a plethora of other technical data, it also informs you if you have the codec to decode it. This tool quickly identifies the .MOD file is actually a MPEG-2 file and that my machine does not have the codec for it.

My friend recommends me to use the K-Lite codec suite. Once the suitable codec is installed, renaming the .MOD to .MPG allows me to play it on Windows media player.

The world without Visual SourceSafe

I am not here to bash Visual SourceSafe as I still believe that it is a great tool for simple development shop but to explore the world after leaving VSS. What are the possibilities on offer?

The primary aims are to keep as much cash in my pocket rather than putting them in some vendor's pocket and that it has to scale up.

What can scale up better than source control system that are routinely used by Internet Open-Source communities? Hence there are CVS and Subversion. I have used CVS and not at all impressed with it and since Subversion (SVN) is a replacement of CVS, I will concentrate on it instead.

The other on offer is VSTS with TFS from Microsoft. Since my MSDN subscription can let me use this and that it is a fairly common toolset, I will also explore this one too. The fact that it is becoming like a giant octopus reaching out and trying to be a tool for everything worries me. Others such as ClearCase, I can't afford and my dislike of it has subsided considerably after my discovery of TFS as revealed below.

There are enough centralised materials on TFS that need not be repeated but Subversion route is more interesting and is free!

One of the things that stunned me when looking at VSTS/TFS as a VCS is that it has taken away the facility of keyword substitution of things like $Revision:$, $Date:$ or $Log:$ (keywords available in VSS) in the source file to brand it. They are god-send when you have a few copies of them lying around.

In these days of ubiquitous USB memory drives, the chance of having a few copies of the same file is extremely high. Without any form of identification, you will be spending hours resolving the differences.

To me, this is a philosophy that someone is trying to ram that into the customer's throat and is a classic example of "Why Software Sucks...". The fact is that the repository has all these pieces of information and the software refuses to allow the users to use them as they see fit. Their developers should be reminded that "You are not the user" principle.

This is not just VSTS/TFS dogmatic approach in pursue of their philosophy but out of the box, ClearCase does not support this too. But at least in ClearCase, one can add a script that is called when one checks in a file to extract those information from the repository and injecting them into the source file. Microsoft, I hope you are listening.

It is also interesting to compare the philosophy used in Subversion in managing the versions.

• It is very similar to that used in TFS.
• It treats a set of files/folders as a tree rather than file by file as in CVS.
  
• It is also atomic - meaning that if one file in a set fails to check in, the whole set will not get in.
• The revision number applies to the entire tree not really to a file.

OK. I have left the best part last. Even though their versioning models are almost identical, SVN allows users to perform keyword substitution to inject revision number using $Revision: $, check in date using $Data: $, and the customary $Id: $ is also there too. For more information please see "Keyword Substitution" in Chapter 3 of the book, which is free. The only thing missing is to inject the change history into the file. For that I can live with it.

Good to see SVN developers that are considerate and not pushing one's philosophy down one's throat.

This alone wins me over immediately.

For Windows users, one of the disadvantage with early CVS was that there was no GUI client. SVN has fixed all that. The best is the Windows Explorer plug-in called TortoiseSVN.

For those that live day in day out inside Visual Studio, help is also available in the form of an add-in.

Finally, the Windows version of SVN with an installer can be downloaded from here free.

Armed with all the materials, I am off to explore the world without VSS. Stay tune.

Grep the tab - driving me mad

Most of the people would have heard of this great Unix tool called Grep. This is the authoritative site on the Grep and can be downloaded from here for Windows. This downloads the whole set of GNU tools, which are very good.

There appears to have a number of slight variants of Grep around, kind of like Linux/Unix and that the GNU Grep does not support the ability to find the TAB (0x9) character. For example, if you have a line like this:

<tab>Hello World

in a file file.txt and if you use this syntax

grep -E "\t+Hello" file.txt

You will get nothing. The reason is that according to GNU Grep, \t is not an acceptable special character. It simply looks for \ follows by t repeating one or more time followed by Hello, contrary to standard Regular Expression syntax. So grep does not use regular expression at all.

To search for white spaces, GNU Grep has this syntax: [:blank:] which indicates a space or tab. But it will not be just looking for TAB.

All is not lost! There is a variant of grep that supports the Perl Regular Expression mode. It is selected by -P or --Perl switch. You can download it from here for Windows version.

With this you can use this syntax to echo the above line:
grep -P "\t+Hello" file.txt

The only disadvantage of this version of grep is that it is not a single file program. It now requires the following DLL: libiconv2.dll, libintl3.dll, pcre3.dll and in addition, it requires MSVCP60.DLL.

It is nice to see this version of grep.exe also has the PE version information so that you can tell what version of software you have. It never amazes me why Unix/Linux never has anything like this. It is almost impossible to tell the vintage of the number of grep.exe in my machine and all of them have different MD5.