Lost free space after aborting defragment

Defraggler Defraggler Bug Reporting
1

I was defragmenting my external drive. The partition I was doing is 781gb and was not close to being full. After running overnight and not getting very far, I cancelled it this morning and figured I would move a couple of large files to another disk so it would have an easier job of it. When I tried to restart it as a quick defrag, it warned me that there was less than 10% free space on the disk, which seemed very strange since I did not get the warning the first time. I ran Treesize on the drive and it's only finding 358.5gb in files on the disk, but both it and Defraggler are claiming I only have 10.8gb in free space (and I should say that it was 13gb before I started and then cancelled a quick defrag.

CHKDSK and the internal disk check from Defraggler claim there are no errors on the disk. How do I get back my missing disk space???

2

Hey I had this same problem using Defraggler running Windows 7 professional. I lost about 40gb of space after running the Defrag. I was looking at the options and saw in the tab, Action> Advanced> it gives you the option to defrag your free space so I ran this hopping this would help but I lost even more disk space. I also did the check disk for errors in the advanced tab as well and it came back with no errors. I am still searching for a solution, If I find one I will post it ASAP.

3

Maybe it's time to post this in the normal discussion forum and see how prevalent this is. I am disappointed that there has not been any acknowledgment of the problem.

4

Hi guys, and welcome to the forum to Lindino (first post).

You don't say which Operating System you're using Bill, but I'm guessing it's either Win 7 or Vista.

For both Operating Systems your loss of space issue is probably down to Volume Shadow Copy Service.

When defragging, Volume Shadow Copy Service thinks that new files are being written to the volume and tries to create shadow copies of them as defragmentation occurs. The reason is found here in Microsoft Help And Support:

The System Shadow Copy provider uses a copy-on-write mechanism that operates at a 16-KB block level. This is independent of the file system's cluster allocation unit size. If the file system's cluster size is smaller than 16 KB, the System Shadow Copy provider cannot easily determine that disk defragmentation I/O is different from typical write I/O, and performs a copy-on-write operation. This might cause the Shadow Copy storage area to grow very quickly. If the storage area reaches its user-defined limit, the oldest shadow copies are deleted first.

http://support.microsoft.com/kb/312067 (Scroll down to "CAUSE")

It continues making copies until it reaches that user-defined limit mentioned above, and this will happen with any defragging program.

How to Delete Volume Shadow Copies: (Vista and Win 7)

Hope that helps.

5

For both Operating Systems your loss of space issue is probably down to Volume Shadow Copy Service.

Thanks. It was also Windows 7 Pro. I ran the cleanup, and it did not seem to solve the problem. Then I decided to go to the backup/restore program since I have backups stored there. It also claimed there was not a lot of free space. I did the cleanup a second time, and now my space is back (about 200gb worth). So I am not sure if the cleanup took longer than expected, or was not reflected immediately, but the good news is I have my free space back.

Thanks.

6

I'm pleased to hear that.

I've seen a few mentions around the web about suspending Shadow Volume Service while carrying out a defrag, but I don't know the practicalities of that as I'm on XP.

It might be worth looking into.

7

OK, so I decided to try the defrag again after I got the free space back up to 30%. It's been running for 12+ hours, and my free space dropped down to 10% fre (80gb). I paused the defrag, then tried the same cleanmgr (twice) and the space is not freed up. I am thinking it might be the CHKDSK /F that recovered the space, but I can't run that because I only paused the defrag and CHKDSK says that something else is using the drive and I do not want to let it dismount it.

I read the stuff about 4k cluster sizes (which is what the drive has) but is there a way to fix that without destroying the contents of the drive? I think the big problem I have is a 180gb Win7 data migration file on the disk, it was what I used for transferring data from my old machine to this one. I'd rather not delete it. I probably have space on another drive to move it off the external drive so the defrag can run, and then I could put it back, but that will probably take a while to copy back and forth.

They claim on the MS page that there is a user defined limit to the size of these shadow copy files. Any idea how to set the limit so it's low?

Oh, and I am only at 38% complete, and my number of fragments has gone from 1815 at analysis to 2931 now.

8

Some information here Bill, and we do have a few members running Win 7, so maybe they could offer some input from their own experience which I can't do.

http://www.davescomputertips.com/articles/windows7/how_to_change_windows_7_system_restore.php

http://www.mydigitallife.info/2007/06/26/how-to-change-and-limit-system-restore-storage-space-usage-size-in-vista-with-vssadmin/

There is a limit apparently to how low you can set it and still have it functioning as it should.

With regards to cluster size, you can set that when you format a drive, and there are recommendations for Hard Drives of different sizes which I noticed somewhere yesterday. I'll see if I can find that table again.

Beyond that, I can't really say.

9

I've come to the conclusion Bill, that your best bet is the Win 7 Forums, where you'll get the right advice from folk who are actually running that Operating System.

There's so much different info out there, that I don't want to give you bum advice or send you down the wrong road.

http://www.sevenforums.com/

If you get what you're looking for from there, you could always enlighten us. :)

10

With regards to cluster size, you can set that when you format a drive, and there are recommendations for Hard Drives of different sizes which I noticed somewhere yesterday. I'll see if I can find that table again.

Yes, the table recommended 4k for a 1TB drive, but obviously that is what causes the problems. Oh well.

11

Out of curiosity, was something done to address this issue in the recent v2.02 update? I was finally able to compress the disk without losing space.

BTW, I installed a new version of Treesize Free, and interestingly it offered to start itself in Administrator mode this time. When It did that, it saw a bunch of missing files and I was no longer missing allocated space on the drive. But there was still a problem with missing free space.

12

Hello, I have the same problem in Vista (of course not only for aborted defrags).

The normal file in "System Volume Information" is like 300 MB, but after defrag of my windows vista partition it is 1,5 GB.

You just hit the check button again after defrag and see the new big framented file.

And after another defrag it even grows to 3 GB!

And there is another problem: you can read of many people online, who's system restore points are missing every some time.

I bet it has something to do with this problems. My restore points are also often suddenly gone.

That vista partition has 10 to 13 GB of free space. So space is not the problem.

If this problem could not be solved, I have to deinstal defraggler and use the windows own defrag.

Cause why should we accept that massive production of gigabytes of trashy shadow files and even loss of restore points?

And I do not like to disable the restore points and activate them after every defrag to delete that massive shadow file.

For me, this is a major bug.

(latest Vista SP2, drivers, defraggler.)

I think I should open a thread for this big problem.

13

Hi softskiller.

I really don't think it's a Defraggler bug, or even a Defraggler problem as such.

As explained above, with Vista and Win7 having a default 4kb cluster size, Windows can't tell the difference between defragging and normal file writes, I/O (In/Out) procedures.

The System Shadow Copy provider uses a copy-on-write mechanism that operates at a 16-KB block level. This is independent of the file system's cluster allocation unit size. If the file system's cluster size is smaller than 16 KB, the System Shadow Copy provider cannot easily determine that disk defragmentation I/O is different from typical write I/O, and performs a copy-on-write operation. This might cause the Shadow Copy storage area to grow very quickly. If the storage area reaches its user-defined limit, the oldest shadow copies are deleted first.

Windows is making backups of every file defragged. The results of this will be more pronounced if you use the "Move Large Files To The End Of Drive During Whole Defrag" setting, or any similar setting in other third party defraggers.

Every large file moved will be backed up by "Volume Shadow Copy".

Thus "System Volume Information" will pretty quickly expand until it reaches it's set upper limit, and then "Restore Points" will be deleted to maintain that upper limit. The amount of free space you have on your Hard Drive is immaterial, as it's the designated upper size limit which decides when System Restore points will be deleted.

How to Change Vista/Windows7 System Restore (System Protection) Disk Space Usage Size Limit

If you don't get this problem using Windows own Defrag utility, then it seems that MS have coded it in such a way as to avoid the 4kb cluster size I/O confusion. I've no idea about that.

The only thing I would suggest is to try other third party defrag programs, and find out if the same problem arises.

If you find one which doesn't trigger the "Volume Shadow Copy" service, then hang on to it, and if you wish, by all means inform us about it.

I'm pretty sure the devs are well aware of this, but as it's more an Operating System issue than a Defraggler one, I have no idea as to whether anything has changed with the latest version of Defraggler, or will change with a future one.

I don't know what else we can tell you about this except repeat the above answer each time the issue is brought up, and suggest maybe a visit to the Vista or Win7 forums where I'm sure the subject will have been raised.

Hope that helps.

14

Hello @ll,

I really don't think it's a Defraggler bug

It's OK, but I think, that a follow problem which I have, is a bug: after disappearing of all restore points any new one cannot be created automatically: I had to create a new restore point manually, after that the system restore works as usual.

OS: Win 7 HP x64.

Defraggler: V 2.02.253 (64bit)

15

The problem with Windows 7 and Vista increasing the size of the System Volume Information file (containing the Volume Shadow Copy for System Restore) also occurs with Windows' own built-in Disk Defragmenter - it is not caused by Defraggler. The Microsoft link given above is very helpful in explaining this.

If it is an issue, the only real solution is to begin with the disk volume formated with a cluster size of 16KB or larger for volumes on which System Restore will be enabled, though pretty much all PCs will be set up with a smaller cluster size by the manufacturer!

It needs to be flagged as a known issue in the FAQ with an explanation.

Switching off System Restore should be a last resort because it is a valuable part of Windows' protection. However, defragmenting a large volume of files at once, especially if the disk is nearly full, will cause System Restore to fail anyway for lack of space. If a disk is filling up with files gradually, regularly defragmenting avoids the problem of a massive surge in System Volume Information.

16

The problem with Windows 7 and Vista increasing the size of the System Volume Information file (containing the Volume Shadow Copy for System Restore) also occurs with Windows' own built-in Disk Defragmenter - it is not caused by Defraggler.

It's not so, I'm sorry.

Just now tested:

SVI directory size before defrag: 9,8 Gb (10Gb is a max. size)

SVI directory size after defrag with Windows-Own-Defrag: 8,55 Gb.

All the last restore points stay available.

FYI: I had uninstalled Defraggler.

17

It's not so, I'm sorry.

Just now tested:

SVI directory size before defrag: 9,8 Gb (10Gb is a max. size)

SVI directory size after defrag with Windows-Own-Defrag: 8,55 Gb.

All the last restore points stay available.

FYI: I had uninstalled Defraggler.

Interesting. However, I tested with Windows 7 Disk Defragmenter, and my SVI rose from 26.8GB to 29.78GB as reported by System Restore. It's interesting that you had used 98% of your max size setting for SVI before the Defrag. Can you be certain you didn't lose the oldest System Restore point? The very fact the size of the SVI changed suggests this. If the SVI was unaffected, it would have been 9.8GB before and after. Nothing about a defrag should reduce the SVI, except by removing a System Restore Point, which happens automatically every time the limit is reached. The way to test it reliably would be to increase the limit setting so there is plenty of space for the SVI to grow. If the SVI had stayed the same size, I would have asked whether your disk has cluster sizes of 16KB or greater. Have you read the Microsoft Support page linked to above? (I know it's written before Windows 7, but I'm sure it still applies, as the Shadow Copy service is still the same).

18

Can you be certain you didn't lose the oldest System Restore point?

No, I can't, it's good possible, that the oldest restore point(s) was(were) removed. But in case with Defraggler I'd lost ALL restore points - it's sure.

19

No, I can't, it's good possible, that the oldest restore point(s) was(were) removed. But in case with Defraggler I'd lost ALL restore points - it's sure.

The thing is, if it was a major defrag you did that time with Defraggler, the System Volume Information will have tried to grow to bigger than the 10GB limit you have set. This can easily happen. Then you do lose all System Restore Points. It's a problem with Windows 7 and Vista, as explained in the Microsoft Support page, and it would have occurred with Windows built-in Disk Defragmenter too, or indeed any defragmenter. It all depends whether the size of the SVI for all that file moving of small fragments is greater than your limit setting. Either increase the System Restore limit when you do a major defrag, or you have to accept losing all System Restore Points. In this, it makes essentially no difference what defrag program you use - there could be marginal differences in the number of file moves to complete the defrag, but it is unlikely to make any difference. I suspect Defraggler is pretty efficient anyway.

The main strength of Defraggler is the ability to do a Quick Defrag on just the files with small fragments, or defrag just selected files. There is actually little point moving around big chunks of large files in a full defrag, so that is the advantage over Windows built-in Disk Defragmenter. It saves a lot of time and disk bashing!

Unless your disk is getting full, there is probably no particular reason to set a low limit for System Restore.

20

I've found the page buried in the Defraggler documentation that goes some way to explaining this, but it's not strictly correct:

Windows Vista and volume shadow copies

The Microsoft link (as given by an earlier poster above) is more helpful:

Microsoft Support: Shadow copies may be lost when you defragment a volume

The Defraggler documentation says the problem is with third-party defragmentation tools, but it occurs with Windows built-in one too, as explained in the Microsoft Support page.

In this page:

Problems running Defraggler

the Defraggler documentation says Windows Vista creates an automatic System Restore point when you run Defraggler, but it isn't true. The Microsoft Support page explains what really happens, which is that the System Shadow Copy provider keeps track of all file write events at a 16KB block level, so it grows the Shadow Copy information stored in the System Volume Information file, and it can grow so large that some or all System Restore points have to be removed, to respect the user's chosen limit on disk space for System Restore.