Speed Freespace and Finished result

I am starting this as a new thread because although all the aspects are reported elsewhere my input does not appear to sit well in any other thread. Also it will be a long post. I hope that it proves of some use in solving the slow down mystery.

I have just upgraded to 1.18.185 from a 1.16 version downloaded just under 4 months ago. When I used this version it defragmented in a time that I considered quite fast for the size of disk, file structure and amount of defragmentation. I have been using various defrag routines for about 25 years and have some "feel" for what is working well. I am aware that defrag tests require the use of cloned drives so that identical fragmented file structures are required for tests and the amount of free space on a drive can significantly affect defrag speed. I do not have results from cloned drives, but over the 7 partitions that I have defragged with Defraggler this last ten days I have made several observations which I hope will help the programmers spot where the problem arises.

First I think that there are at least 2 new bugs since 1.16 and some problems may be the result of side effects of another bug.

1/ First the display bug, which I have contributed to elsewhere, where after a completed defrag the display does not match the state of the drive. Minor inconsistencies in display are not uncommon within programs and can be explained by the "averaging" that is used to display a square where the square is made up of several files. During defragging a program is trying to show change and may "bias" the display in order to do so and so by the time defragging is finished the display can misrepresent the true state, and an analyse step is required to stabilise it. However Defraggler is leaving the display in a state where this does not seem to be the case including Red that should be Blue and files where it should be White etc.

2/ Defrag of freespace. This no longer seems to work for me and the help page http://docs.piriform.com/defraggler/using-defraggler/defragmenting-free-space-on-a-drive is confusing where the first part of the explanation seems good and then "Defraggler will fill up freespace chunks with whole files only. If you select Defrag Freespace (Allow fragmentation), Defraggler will use file fragments to fill up freespace." just confuses the issue even though it is correct. I was reading this to check just what Defraggler was supposed to do, but I ended up confused as what was supposed to be the outcome rather than the process.

Until now "defrag freespace" meant to me "consolidate freespace" that is move files in such a way that where possible (barring unmovable files) freespace ended up in 1 single uninterrupted chunk. The two options being A/ move files such that the files end up defragmented or B/ allow the moved files to be fragmented. Indeed with many of the older defrag programs this was step 1 in defragmenting a drive.

I have noticed that on several of my partitions the freespace is dotted with small files and neither of the defrag freespace options move them. This problem may also be affecting the overall program in that A/ there is no freespace fragment large enough to relocate some of my larger files which try to be moved and fail and B/ with the move to end option selected (my default) that any small file in the way is not removed first and so the moved file becomes fragmented.

3/ Defrag algorithm. A couple of my partitions did defrag quite quickly, however I noticed a higher level of CPU activity that I remembered from before. I am guessing that the speed ups that have been applied do this by doing a little more thinking (computation) and then making a more optimised set of moves to achieve the result. However there may be certain situations in which the computation screws up and the time taken exponentially expands. One aspect of this is that originally and with all other defrag routines, the one given is that the disk drive never stops transferring data from one point to another. Now there are VERY LONG time periods with no disk activity just CPU. The S: drive of this machine is currently defragging (S: is used for long term storage of software images - installation disks), no other program should be accessing it as restore is Off for this drive and Indexing is inhibited. The drive is 35GB and 17% free space (5.87GB) the fragmentation report is 10 files 104 fragments 10% fragmentation. Task manager reports DF.EXE has been running for 20:50:05 and has had 6.2G "I/O other" and 6.3TB "I/O other bytes) though this huge I/O were not disk movements.

Using full screen and looking at the device map initially the main observation was that freespace is fragmented such that the larger files cannot be defragmented. I am guessing again that the algorithm is failing to find a strategy within a reasonable - keep the heads moving - time period. The present disk map is showing far more Red than it should in that most of the files moved are now fragmented and the summary is 9 files with 735 fragments 39% complete. Stopping and analysing now shows 44 files with 843 fragments. This disk I would have expected to take between 30mins and 1 hour which is consistent with the last time it was defragged - there has since been significant rearrangement. S: contains installation files and so they range from 0 bytes to several MB, plus 5 files 3.5Gb to 6.5GB

Probably a good time to stop and list my hardware.

Machine 1. XP on an Athlon 3000+ 2GHz, 2GB RAM. C:30GB 36% free, D:39.1GB 38% free M:39.1GB 38% free, S:35.3GB 17% free, X:42.9GB 37% free. D: is a backup of M: using SyncBack

Machine 2. Windows 7 Intel Core 2Duo 1.8GHz, 2GB RAM. C:200GB 50% free, W:1TB 11% free

I have been running Defraggler for a week so far due to the slow speed. Only the two C: drives are being indexed and have Restore switched on. Strangely both C: drives defragmented in the shortest times, both well up on expectation but under 2 hours each.

W: I hand defragmented in the end by using File list and choosing carefully which order to defragment individual files so that large white space was created for the next defrag. Finally I ran Defraggler normally and 20 minutes later I had a well laid out drive map. The last time I used Defraggler on this drive (1TB) it ran nearly 24H but almost continuous disk movement and perfect result without help. This time I let it run 3 days with little disk movement and then I gave up and hand held it. W: is my TV archive and backup drive for C: and so has a majority of files about 1GB in size the bulk being <4GB but the largest is 10GB

X: I also gave up and hand held it, but achieved a 0% fragmentation but a messy looking disk map. I cannot get consolidated freespace, visually there are about 20 fragments. Not good because this drive has backup files written to it which are then copied to another backup drive. Initially some of the files had over a thousand fragments, but there are only a few hundred files on the disk. X: contains about 10 >1GB files

M: and D: are the most interesting as one is a copy of the other. M: is my MAIN drive (data) and is backed up (synced) every day to D: M: also contains backups from C: which are more frequent. Before the use of Defraggler this week, the only difference between the drives was the layout of the files, otherwise they were identical size and content. However Defraggler ran on one and completed in 4 hours (very slow - expected < 1H) and the other didn't finish after 4 days!!! Again some help got the result I needed, but still fragmented freespace.

All the problem drives have files >1GB (but so do the non problem drives). 100% CPU and NO disk activity is a common factor as is fragmented freespace. I followed a link from one thread to an NTFS which suggested that consolidating freespace in NTFS was impossible using the API!! I hope not as moving small files is key to achieving a good result.

On a positive note. I have lost no data and despite the slow downs have finished my task. Defraggler is basically a sound product and I look forward to the updates. As an embedded programmer and electronic design engineer I appreciate the complexities that lead to unexpected faults and the difficulty in tracking them down. Congratulations to those of you who give your time to bringing us such a useful free product.