Disk Cloning and Imaging Software speeds

[mta]

Some recent threads have talked about using disk cloning/imaging software for backup/restore purposes.

In particular, the time taken to perform those tasks.

 

So I thought, with the view of spreading the word of their benefits, I'd start a topic to get feedback on those speeds so others can compare.

 

I use Macrium Reflect, and the stats are;

 

Backups

full disk image takes 00:05:21 for 22GiBs.   (internal SSD going to an internal HDD)

full disk clone takes 00:07:20 for the same 22GiBs.  (same SSD going to an external, USB3 connected, 7200rpm, hard drive)

 

Restore

sadly the Macrium logs for the Restore, last needed about 6 months ago, are gone (probably thanks to CC ) but the times were very similar to the backup, less than 10minutes certainly.

 

My setup is a 120GiB SSD for system/program files and a 500GiB HDD for personal data.

The SSD is imaged weekly and cloned monthly.  The HDD is backed up daily with just the normal 'what files have changed' backup software to a NAS.

Keeping your system files and personal files separate is the key here.

 

So for me, spending 5 minutes a week to snapshot the system drive is time well spent.

It gives you that revert to last know good configuration safety net which system restore points don't offer.

 

Macrium also can do Incremental backups, so I'd be interested if anyone has timings for those.

[Andavari]

Macrium Reflect by far is the fastest at creating a backup of any disk imaging software I've ever used, and trusting it having had to use it to restore Windows speaks volumes - a software I can trust. Couple that with it works very well, and has very strong compression at a high rate of compression speed. At maximum compression it does a full image for me in under 6 minutes when saving to another internal hard disk. It's still fast when saving to another disk partition (which isn't another hard drive it's split from C:\). It takes about 3x longer when I create a secondary full disk image backup for paranoia's sake on my 16GB USB Thumb Drive.

 

I don't know what they use for compression but if they offered their compression technology as a file archiver with a GUI and be somewhat similar in function to 7-Zip, WinZip, WinRAR, etc., they'd perhaps beat them all on compression and de-compression speed.

 

My ancient way of doing a so-called backup all of which is automated is using a lengthy batch script file to copy files to a folder, and then use 7-Zip to compress that folder using 7z Normal Mode which is about the same speed as Macrium Reflect. However that ancient way of doing a backup doesn't backup everything, can't backup in-use files (antivirus, etc.,) and it can't restore the computer from a boot CD.

[login123]

Very informative, thank you.   

Mta, what OS is that?   Is Macrium free or paid?

[hazelnut]

I am using Macrium paid on Windows 7 and Windows 10.

 

It all works fine.

 

I tend to do a full image every month with incrementals inbetween.

 

Because it is paid version I can set it up to have a boot option to either do a restore or boot into Windows as well as doing it from a disk.

 

Not had a failed restore yet

[Andavari]

Not had a failed restore yet

 

The same for me. Although I use an older free version of it, and I always make sure I verify the backup immediately after creating it. So it takes under 6 minutes for me to create the backup, and about another 3 minutes to verify it.

 

I'm so paranoid about the Full Backup integrity that I get the file hash (MD5, SHA-1, SHA-256, etc.,) of the backup image and save that in the folder with the disk image, and I'll always verify in Windows that the file hash matches before restoring - even though I also have Macrium Reflect verify before restoring.

[mta]

Very informative, thank you. 

Mta, what OS is that?   Is Macrium free or paid?

 

Was Windows 7 when I was using v5 Free, then 8 & 8.1 and now with w10 it's the paid v6.

I had to go paid (not that I minded) when upgrading from v5 as one of the changes Macrium made to the program was the ability to clone across two different disk types.

I think that was available in the free v5 but they removed it in v6, plus the new v6 free seemed to only be on a 30 day trial, I didn't stick around to see if it got hobbled after the trial period, so bought it.

[login123]

. . .

Because it is paid version I can set it up to have a boot option to either do a restore or boot into Windows as well as doing it from a disk.

 

Not had a failed restore yet

 

Thank you, Hazelnut.  Music to my ears. 

[bartholomewking]

Hello all, I read your post with interest Mta. I was quite astonished by the disc image backup speeds you were achieving.

My machine is quite old now, but serviceable. I thought I would do an experiment to hopefully add something to the debate.

I downloaded the Macrium Reflect product (free) from their website. I had just performed a backup using Paragon Backup and Recovery Free version that I generally use.

 

I did two Image Backups using Macrium to an external USB 2.0 IDE disc. The first was with the default medium compressed data, and the other with the maximum compressed data.

The comparison with Paragon is not entirely fair as the Paragon image backup was performed with the same source disc, a little earlier granted, but to an internal SATA drive. This backup one would have expected to be much quicker given this advantage. This was necessary because of the availability of backup space. The results however speak for themselves. The backup I took with Paragon was with their maximum compressed data configuration.

 

After a disappointing start with Macrium, largely caused by user error:

(I forgot to remove the backup drive from the list of drives to be imaged. This immediately resulted in the error: “MFT corrupt - Error code = 6. Please run 'chkdsk /r'” from Macrium. Sadly this resulted in me running chkdsk on both drives to no avail, a pretty long winded process, and a reformat of the backup drive. It was after all this I realised what I had done wrong. In minor mitigation, the error is somewhat misleading. However, having said that, I throw up my hands and exclaim “it’s a fair cop!” to the deities overlooking data backup. )

 

System Data:

Intel Pentium D 930 dual processor clocked at 3.45GHz.

4 GB installed RAM (clocked at 235MHz). ASRock Motherboard with RS400 Chipset.

Source Disc IDE Drive system disk ATA UDMA (mode 6)

Destination disc for Macrium test: 320 GB USB 2.0

Destination Disc for Paragon test: internal 500 GB SATA ULi M1573 SATA Controller.

Windows XP 32bit SP3.

 

The Results:

Macrium Reflect System Drive Image backups.

Macrium Reflect Free Edition (32-bit) Verision 6.1 Build: 871

Total Drive Data to image: 137,145,954,304 (127 GB)

 

Full Backup (Medium Compression):

Target Disc (IDE USB 2.0)

Size of Backup on disc:

110,036,877,288 (102 GB)

Backup Process Duration: 1hr 30min

I/O Performance: Read 524.2 Mb/s - Write 171.3 Mb/s (reported by Macrium)

 

Full Backup (Full Compression):

Size of Backup on disc:

108,551,979008 (101 GB)

Backup Process Duration: 2hr 38min

I/O Performance: Read 412.3 Mb/s - Write 167.3 Mb/s (reported by Macrium)

 

Paragon Backup and Recovery Free Version 10.1.21.638 (09.05.14) (32bit)

Full Backup (Full Compression):

Started 23:57 finished 04.08 am (4hrs 11min)

Size of Backup on disc: 99,180,044,288 bytes (92.3GB)

 

So, it looks like Macrium Reflect is much quicker, but produces a marginally larger disc image. The differences in speed are just not comparable.  Macrium is the best part of twice as fast.

 

The Macrium interface is less cluttered. Things are more obvious, although it did catch me out as indicated above.

I don’t have figures for a Paragon backup uncompressed, or indeed their version of medium compression.

 

A Few questions:

I cannot yet find a way to restore individual files or folders/subfolders from a disc image. Is this not available in the Free version of Macrium?  It is in the free Paragon product. [Edit2: yes you can! see p200 of the pdf user manual]

 

One of my hard drives has a red bar indicated in Macrium I think indicating how much data is on the disc. I had a look at the manual but didn’t find the information. The fact that it is marked in red concerned me slightly, or does this just mean that the disc is near capacity (425/465.78GB)?

 

My firefox browsing history was no longer available after the Macrium backup, have others had this problem or is it just a coincidence?

[edit1: i found the file places.sqlite.corrupt in my firefox profile file alongside places.sqlite. I replaced the latter with the former and my history returned ( at least to last night). This .corrupt file was timestamped at the time the backup was running. Perhaps this was as a result of the backup going on at the same time. I just had a quick web-browse during the backup!]

 

I hope this information proves useful.

[mta]

@bart,

Good testing as usual.

I'd be interested if you ever did a non-compression test to hear about the timings.

For me, I don't care how big the end product is, I just want it done ASAP.

 

Yep, the red bar is not for 'errors' but for remaining space.

I think when it hits 10% left it goes red.

 

I've never had browser issues, or any issue in fact, after running a MR backup, so I reckon coincidence would be the best excuse.  further usage would prove (or not) that theory.

Any chance Firefox was running when you did your cloning?  (ahhh, just re-read your post - that indeed would be my guess as to your issue)

 

I've only ever used MR for complete system dumps and restores so can't answer your 'can it restore individual files?' question, but considering it does incremental backups, I'd be surprised if it couldn't.

Others on this forum who use MR do do incrementals, so maybe they will chime in with a response.

[bartholomewking]

Yup MTA, it does beg the question whether a backup is a backup if you're doing something while it's running! In my Unix days, you'd put the system into single user mode so nothing else could be done. I'm not sure MR would run in "safe mode", which I guess is the nearest equivalent. I found out how to restore an individual file, so I opened the documents and settings\firefox\profiles folder in the backup image and there was no places.sqlite.corrupt on the backup. It's got to be a pretty minimal chance the thing that happened, and as a matter of fact I tend to do a backup then go to bed, or go out, or write a lengthy book, so it's not usually an issue. To be honest, without some research I can't envision how an image backup can be a true image of a quiescent machine if it isn't, quiescent that is. I'm not really sure what the VSS service does.  I will boot up the PE disc and see what you can do in that. I only installed the thing last night but well impressed so far. I've used the Paragon product(s) for quite a while, and I suspect they come from a Unixy background. After all, "it was hard to write, so it ought to be hard to use". That's a little unfair, but the interface does take a bit of getting used to. Some of their other product free versions are also very useful for disc related tasks. For XP there was when I last looked a copy of "Paragon Drive Copy Free Edition for XP" which became available after XP support was withdrawn.

 

I must admit, apart from my first ( and quite alarming ) glitch, I think it looks  like MR is a great product. It also has apparently backed up my Linux partition. I haven't tried to restore anything yet (apart from one file) so i remain a little sceptical. I will try an incremental when there has been time for an increment to have happened. And yes you can restore individual files/folders, it talks about incrementals in that respect as well in the PDF manual ( in my old age i am starting to become an advocate of RTFM ), however I tend to read it after I've got a bit familiar with whatever the program is and botched a couple of things. I'll maybe do a test of the backing up the Linux partition ( it's the smallest ) with and without compression and see what happens. That should be a reasonably valid test of speed. I suspect the overhead of the compression is the thing that slows it down. The CPU was nearly maxing out when the full compression version was running and it took about 1 and a half times as long,  with little reduction in eventual file size over 'medium' compression. I should imagine that processor is the rate controlling step, especially if you are using SSD USB 3.0 and so forth.

 

[Edit: just having had another look at the throughput data above, that would indicate to me that the CPU is the rate controlling step, at least on my machine. If space is not a consideration, and speed is the issue, I would also suggest trying an archive with no compression. Of course, if you have a processor capable of modeling the weather systems of gas giants that might not be the case.]

[bartholomewking]

MTA,

OK, well I booted into the Macrium PE windows environment and somewhat unsurprisingly to me not all my disc drives were visible. As I have mentioned before in other threads, the machine is old, and that's why it's still XP ....  Drivers!

 

I used the option "find unsupported devices", and it found the Ethernet Card driver, by manually pointing it to look in the system32 directory on my system disc ( rather helpfully labeled D: )

 

However, as I say, my machine is old, so this should not apply to many others. I will also look again in the Macrium package WinPE package manager at the Recovery Media options. I think I'm going to have to build a custom PE environment and include my existent disc drivers in it. Worryingly though, the PE kernel that is built has a minimum spec of windows 7, so it may not work. As to that "not tonight Josephine", it's late.

 

Slightly more worryingly however on reboot into Windows the message "The Cfg Registry key denied access to SYSTEM account programs so the Service Control Manager took ownership of the Registry key." was found in the Event Viewer System Log. Intuitively I'm not too worried about that, that message can be the sign of a trojan, but as it found the Ethernet driver it's not unlikely that the PE environment made a change to the Registry that XP didn't like. We'll see. F-Secure, the virus prog I use, found no problems. It also has an additional Rootkit diviner and that also found no problems. Along side this I ran Hitman Pro, which is a belt and braces program that finds things unsigned or  "suspicious" and tries to upload them to the could for analysis. Apart from Google's Analytics Package which keeps getting installed by an antipodean product I'm not allowed to mention, and a few C++ executables I and others have written nothing it regarded suspect was found.

 

That aside being a blessed nuisance, the boot environment does allow for the creation of images. That may be a quicker and more secure way than from within windows.

 

[edit: I rebooted from the same boot disc DVD into the Macrium Recovery Environment. I tried to reproduce the "find unsupported devices" issue detailed above. This time the program responded that it did not discover any unsupported devices. As this is a DVD and very definitely read only, then the only place that the program would be likely to save that information is in the registry or some file on the hard disc. I did a search using 'Everything' looking for *.ini, *.cfg and so forth with the 'datecreated:today' modifier in the search and found that the MR program has created a %SYSTEMDRIVE%\boot\macrium folder and a number of sub-folders with BCD entries (vista and above) and a file "reflect.cfg" which has binary content. There was also Drivers folder which contained Disk(empty), Network ( which contained my ULi Lan driver and .inf file) and USB ( also empty ). As might be expected the event viewer message was not reproduced. It does sound to me like the Recovery Media program did do something a bit cheeky to the registry when windows was not active, this would probably not happen with Vista and above. Sadly this sort of thing seems to be more the case with recent programs that still support XP.

There is also a Drivers folder therein with sub-folders Disk, Network and USB folders on the DVD Rom each of which is unpopulated. In the scale of things this is a minor problem. If the truth be told, I now remember, I had to load a floppy disc on the version of XP I originally installed to get the SATA devices recognised]

[bartholomewking]

Hello all and @Mta,

I did an uncompressed backup of my Linux partition and in summary this was indeed quicker.

 

Macrium Reported:

I/O Performance: Read 545.6 Mb/s - Write 172.3 Mb/s

CPU usage was also notably lower, running around about 25% most of the time.

Sadly the backup did not complete as the backup device ran out of free space, and I haven’t got any more readily available.

 

The bad part about this was it didn’t warn before attempting to do the backup that this might happen, and the good part was that invited a secondary location when it encountered the problem so that it could continue.

 

For me very sadly I cannot realistically use the product on my machine. It’s a driver issue. I can’t find drivers for the SATA/RAID device on the motherboard I have that run under windows 7. I don’t think they exist. Therefore the Windows PE environment does not find the SATA discs on my system. As I’ve got something that is proven and works for me I will continue with that. However I will put this for myself on the back burner and certainly re-examine this product when I have a newer machine running. It seems an excellent product and many of the features look superb. For me it’s not an XP issue, it’s a driver issue but I would encourage XP users with older machines to ensure that a restore is possible from the PE environment booted either from a USB memory stick or a DVD just to ensure there are no unpleasant surprises .

 

I’ll probably leave it installed on the machine as it has some extremely useful features that could be otherwise employed. There is a possibility of digging the drivers from a Vista install CD, I know they work because I’ve had Vista running on this box, however I can’t easily find them, and to be honest at this point it probably isn’t worth the time and effort.

 

Hope this info has helped.

 

N.B. The above apply to version 6 of Macrium Reflect. Version 5 inlcuded a Linux Boot Disc. Macrium don't seem to allow a download of this version, and the Version 5 boot disc cannot understand version 6 archives I have read.

[mta]

I have to 'fess up and say I've never used MR on an XP PC, but why are you running MR from Windows PE?

With Windows 7,8 and 10, you just double click the MR icon and you can image or clone straight from there.

Or are you referring to the restore side of MR?

[bartholomewking]

Hi Mta, I was just testing out the product.

 

Everything I ran was from Windows. However I wanted to see what would happen in a simulated disaster recovery situation, as I suspected there might be some problems with my drivers from past experience. Also, it would more likely be quicker to run a backup from the Recovery Medium as the machine is doing nothing else, no multitasking as such, nothing else going on, and all files as they were at the time that the machine was shut down. The disc image produced would be as near perfect as it could be ... in theory (remember my little firefox problem earlier in the thread?)

 

For others reading this: if your system disc gets trashed for some reason, you need to restore a backup image to it. In that instance you would (probably) need to run the PE environment from a USB disc/stick or a DVD. You can install the Recovery Environment as an alternative boot option on the system disc, but in the event that a disc drive actually fails, it's a bit like nailing the life raft to the deck of the Titanic.

 

Windows PE is a subset of Windows. It is a Microsoft Product, and to a certain extent independent of the version of operating system you are using. It can be modified by the OEM (in this instance Macrium) it gives you a limited set of Windows functionality tailored by the software author. So, it is more or less limited to booting, allowing you to save data and create images or clones, and more importantly to restore them without booting into Windows proper. The Macrium variant of PE also allows access to the command line. The Paragon products I have mentioned earlier can do similar things. They have a package called Paragon Rescue Kit (based also on PE but also can be Linux based I seem to recall). It too creates their version of an environment specifically for backup and restore, and fixing boot problems, the proprietary bit of each manufacturer's version understands their particular backup format.

 

There is an article with more detail here: https://technet.microsoft.com/en-us/library/cc766093%28v=ws.10%29.aspx (I think it’s a little out of date, but gives the gist)

 

With any of these backup software suites it is imperative to create some sort of boot medium for the circumstance in which the boot drive is corrupted/damaged/written off, otherwise the machine could become unusable and all the data on the internal boot medium (i.e. system hard drive) lost. When I say unusable, I mean until access to a recovery disc and the ability to restore from a backup. In Macrium one of the things that is necessary is to create the boot medium, or Rescue Media as the MR software calls it.

 

Under the menu item "other tasks" at the top of the main menu in MR there is an option to create such an item (the program tells you do this when it is run, unless you uncheck the nag screen). It gives you the opportunity to write the PE environment (or mini-operating system) to a bootable USB memory stick, or a DVD. These should be retained and put to one side for security. If there is a failure of the boot drive for some reason on the machine e.g. corruption sufficient to make the machine impossible to start up, then these can be used to rescue the situation. Myself I would knock out a couple of DVDs and put them securely to one side. My prejudice would be to use a DVD+R. USB memory sticks are all very well and portable, but can easily be written over, swallowed or flushed down the toilet ... and that would be bad

 

As the recovery environment is built, drivers are loaded onto it from the host machine upon which it was created, so that it will (actually, should) work for that machine. It is possible that not all the drivers required would be on Recovery Media created on another machine. (that would certainly be the case on my system for example).

 

On my system, the snag appears to be that the generic Windows PE environment is built upon the Windows 7 kernel, so the drivers for my SATA devices which date back only to XP and not beyond, are not understood. There do not appear to be any later versions available, although rather oddly, I can install Vista and it does see the discs. Sadly I haven't found a way of extracting them from the Vista installation disc. One of the significant changes from XP to Vista and subsequent Microsoft operating systems was the structure of device drivers. The net upshot is that I can't access two of the discs on my box from the Recovery Environment in the situation that the boot drive is disfunctional.

 

It seemed prudent therefore to test for such and other unexpected circumstances, as all the backups in the world are useless if one doesn't have the means to restore them!

 

Version 5 of the product had the means to create Linux based, and for that matter BartPE recovery discs, that would probably solve my particular problem. I had a look at the Macrium user forum, and the answer to the question why was it no longer available was something along the lines of “it’s expensive to maintain, and not many people use it”. They have no current plans to re-introduce this feature, but will bear it in mind.

 

Unfortunately the download from the site, only downloads a downloader for the installer program, and that is now version 6. Sites claiming to host version 5 point at the downloader. This downloads version 6. I suspect though if there is a version 5 setup that somebody has extracted in the past, then you might install more than you bargained for!

[hazelnut]

Do none of these work?

 

http://filehippo.com/download_macrium_reflect/history/

[bartholomewking]

Do none of these work?

 

http://filehippo.com/download_macrium_reflect/history/

Thank you Hazelnut.

He he, ok, perhaps I was a little hasty! That little lot should keep me from hibernating this winter

[bartholomewking]

I found a download. You have to go back as far as version Macrium Reflect 5.2.6526 http://filehippo.com/download_macrium_reflect/57471/ to get the actual installer [edit link corrected] . Later versions download a stub that downloads an installer, and points at the latest version even though it says version 5 by the looks of it.

 

This version does enable the creation of Linux Based and the earlier Windows PE 3.1

 

quote from the Rescue Media Wizard:

 

"Linux provides a compact and efficient rescue environment to restore all Windows operating systems.

Note: This option enables you to restore your imaged partitions to exactly the same size and position as they were at the time the image was created.

If you require greater flexibility then choose the Window PE edition"

 

I'll see how it goes and let you know.

[login123]

Hi, bartholomewking. 

Thanks very much for the analysis, very helpful & useful for me.

 

However, I think the link in post #17 goes to version 5.2.6551. 

Unless I copied something wrong, the link to version 5.2.6526 is: 

http://filehippo.com/download_macrium_reflect/57471/

[bartholomewking]

 

Hi, bartholomewking. 

Thanks very much for the analysis, very helpful & useful for me.

 

However, I think the link in post #17 goes to version 5.2.6551. 

Unless I copied something wrong, the link to version 5.2.6526 is: 

http://filehippo.com/download_macrium_reflect/57471/

Sorry login123, you're absolutely right on the link!! (I've changed it in the original post to prevent further confusion).

 

I have confirmed that the version I have been testing is 5.2.6526 from help->about within the software just to make sure.

And thank you for the suggestion in the first place Hazelnut.

 

Sadly, however from my own point of view, I have to report failure.

 

Unfortunately I have installed the older version (Macrium Reflect 5.2.6526) and created the Linux based recovery environment.

This also does not recognise my SATA discs.

 

My own Linux installation does, and the Paragon Linux recovery system also does.

 

Why there is this discrepancy, I don't know other than guessing. There is a verbose logging switch when configuring the Linux based recovery option on creating the environment, which results in linux boot messages as the system is started in this environment. However as far as I can see this log is not accessible post boot, and no amount of keystrokes that I have tried at boot-time will pause the screen as the messages scroll so that I can examine them in more detail.

 

(I’ve now managed to photograph them on my telephone)

 

From the photos I can see boot messages that appear to be failure to configure the SATA controller. I'll try and decipher what they mean but that could take some time ....

I’ve also read something similar to this effect on some Linux forums. I suspect there’s something weird going on with the mix of software/drivers and my configuration as individual discs versus RAID. The fact that two different Linux derivatives get differing result imply differences in their kernel ... But that is a whole different subject.

 

I suspect it is exacerbated by with my particular SATA controller and the version of the BIOS on my motherboard. Sadly this is at the latest revision that is available. I have noticed the odd strange thing changing in the BIOS in the SATA setup, post configuration. Although it’s interesting as to why (as ever), it doesn’t give me a practical problem, but it does de-value the functionality of the Macrium Product on this particular machine.

 

I don't think this is any kind of deficiency of the product, which on the face of it seems excellent. As I say, I suspect it's a peculiarity of my particular setup.

 

Having said which it does emphasise the need to test critical functionality such as backup and recovery.

[bartholomewking]

Just out of interest:

There appears to be a known bug in earlier Linux Kernels that fail to pick up on some SATA interfaces

 

Transcribed from photo mentioned in post #19 above as my machine attempts to attach one of the SATA drives:

 

ata1.00:failed to set xfermode (err_mask=0x40)
ata1.00:disabled
ata1:exception Emask 0x10 SAct 0x0 SErr 0x0 action 0x6 frozen t4
ata1:irq_stat 0x48000000, interface fatal error
ata1:hard resetting link

 

i.e. the system cannot attach the device
This would explain some of the discrepencies described in the same post.

[mta]

you hit upon an excellent - and far too often ignored - issue @bartholomewking.

even though a lot of people backup (but mind-numbing amazingly not everyone), very few of those actually do a disaster recovery test to see if/how that data gets restored.

[bartholomewking]

Yes Mta, it's from many sad and bitter experiences over the years