I had to abort at 99% to shutdown my pc. Now its taking hours to defrag again

[legendarysnake]

yesterday when i shut my pc it had 3515 total fragments left. Today when i started it said that it has now 4635 total fragments. 

 

From 5 hours ago it has also 535 fragmented files(whatever that mean) and now its only 532. 

 

Still 7% fragmentation. Why its taking so long? Its not possible to save my progress to a text file and restore the progress? Im on the same situation today. I need to shutdown my pc again with 50% of fragmentation happening. 

Its a 2tb external hdd. 

 

The point is, it makes no sense that the fragmentation didnt finished yet. 

 

I just restarted and continued to 50% but now it has 13% fragmentation! I really dont get how defraggler works...

[nukecad]

To start with how did you abort Defraggler? Is you drive now showing much less space available than before?
You can always 'Stop' a defrag if you think it's taking too long - but you should do it properly and after clicking 'Stop' let Defraggler finish what it is currently working on and tidy up before closing it.
(If you have aborted defraggler in the wrong way, that is if it was stoppped by force or by a crash, then it may have left a lot of temporary files behind).

A 2 TB HDD is going to take a long time to defragment using Defragglers default options.
That's just a consequence of todays much larger drives.
But keep reading because you can change how you use Defraggler to do things much more quickly.

If you have aborted and started again then the standard options are going to take (almost) as long as the first time.
That's because the standard options are not just 'defragmenting' files (which has already been done, although there will be some new bits to do), but also 'consolidating' the whole disc - and it's that consolidation that takes the time.

Quote

I really dont get how defraggler works.

I'll try to explain that - Because as said you can speed things up if you know just what defragmentng is, and what Defraggler is doing.

To start with it depends on just what you (or the defragmenter app) mean by defragmenting.

There are two things that are commonly called 'defragmenting', but they are very different and one takes a lot more resources/time than the other.

'Defragmentation' proper means getting you files into one piece each, so they can be read slightly faster.
'Consolidation' is also commonly called defragmenting and means getting your files into the smallest number of possible clusters on the disc, which may (will) actually fragment the files themselves to 'fit them in' to the smallest number of clusters. That's the one that takes more resources/time, and it's not realy needed on todays larger discs.

When most people talk about defragmenting they are actually thinking about consolidation, that's simply because defragmenting software has been (wrongly) calling it that for years, but user needs have changed over time as discs got bigger so the difference between the two is more important now.

By default Defraggler does a combination of both consolidation and defragmentation, but you can also specify one or the other.

With a large drive it's a good idea to do a full Defrag (defrag & consolidate) once or maybe twice to get the disck in order, but then after that only to do 'file only' defrags.
(You might want to do another consolidate when you have done some uninstalling of things).

To do a 'file only' defragment in Defraggler:
Open defraggler and analyse the drive.
Click on 'View Files' or click on the 'Files' tab.
Select the tickbox at the top of the list to select everything found.
Click on 'Defrag Checked'.

That will just defragment the files without trying to consolidate the whole drive.
So it will be quicker and use less resources.

Note that when doing a files only defrag it is not unusual for it to finish with a 'Defrag Aborted' message.
That's because some file(s) have been opened/changed between analyzing and defragmenting, so for safety it has been skipped, the rest of the files have been defragged.

Like any tool that can be used in multiple ways what you get out of Defraggler depends on knowing how to use it in the best way for you.

In your place I'd let the current run complete, if it hasn't already.
Then I'd do a files only defrag so you can see the difference in time taken.
Then, if you want to, do an new Ananlyze and post a screenshot of the drive map after doing that so that we can comment on how it looks.

[legendarysnake]

Hi, and wow i didnt expected a 50 lines of text haha! So if defraggler its doing consolidating on my drive right? 

 

"

By default Defraggler does a combination of both consolidation and defragmentation, but you can also specify one or the other." or that one. 

 

If defraggler does have both options, what exactly windows does? I tried in the past(on windows 7 ultimate) to defrag with windows and its even slower. 

 

I think that i moved so many files on this hdd on the last 2 years maybe that is taking over 30 hours(so far) to finish it. 

A new info that i have is that i just pressed the stop button and the defragmentation aborted. When i resume it, voilá! is back to 50% again. I didnt knew that defraggler actually saves the progress of it. But damn, its really taking long. My hdd on windows take much less time to do it... probably because its a 7200 rpm hdd instead of a... i dont know how much spin an external hdd from seagate usb 3.0 actually has. 

 

"To do a 'file only' defragment in Defraggler:
Open defraggler and analyse the drive.
Click on 'View Files' or click on the 'Files' tab.
Select the tickbox at the top of the list to select everything found.
Click on 'Defrag Checked'."

 

im gonna try this when this defrag finishes . Now its at 88%(38% progress today) after 13 hours with the pc on. And maybe i will finish tomorrow. 

 

Thanks for helping

[nukecad]

Sorry another essay,
the difference between defragmenting/consoldiation is fairly simple once you get your head around it;  but it can be complicated to explain because everybody has been using the wrong terms for years.
That didn't realy matter until we started getting terabyte drives (and SSDs) for home use, but those have made the difference more important.

(TBH spinning HDDs are on the way out for home use, they'll go the same way as 3.5" floppy discs, and then defragmenting/consolidation will become a niche thing for 'old tech' and maybe big servers. That will probably happen quicker than most imagine).

I'm not sure just how much of which Windows 'Defrag & Optimise' does for a HDD these days, but most defragmenters will do a consolidation because that's what most users think defragmenting is.
It is noticable that the Windows 'Defrag & Optimise' no longer shows a drive map, so no longer shows any consolidation happening on HDDs although I'm sure it still does it.
(If you have an SSD then a drive map is meaningless anyway, so that's one reason they may have stopped showing it).

Back to your drive:
If you have been doing a lot of writing/deleting/moving things about on the disc then a consolidation will take time, but once done shouldn't need doing again for ages.
It's mainly when you have deleted things that leaves 'white holes' (free space) scattered about the drive map where those files used to be.
Consilidation tries to fill those up again, so that all the white space (free space) is after the data.
Again it's not really needed with todays bigger drives, it is something that you wanted in the days of smaller 32GB/64GB drives, but most people prefer a tidy looking drive map so still do it.
Consolidating the drive also means there is less chance of new files getting fragmented when you save them.

Another thing is that you say you are doing an external drive connected by USB - that's going to take longer than an internal drive connected directly to the motherboard by SATA.
Just one of those things, SATA has a faster transfer rate than USB and defragging/consolidation is transfering stuff from the drive to RAM and back again.

As for the 'saving' of progress it's actually because the defrag and/or consolidate is done a bit at a time.
Enough is read from disc to fill your RAM, and then written back to disc 'tidied up', then more is read to fill the RAM and then written back, and so on.
So when you stop it part way through it finishes processing what it has in RAM and writes it back to disc, then clears up any temporary files it had put on the disc, so everything already done stays done.
However when you run consolidate again (full defrag) it still has to take another look at what it already consolidated last time, (it doesn't 'remember' what it did last time), but shouldn't have to process it much.

The other thing to note is that I believe the % fragmentation reported is just that - the actual fragmentation of the files - and not how much the disc has been consolidated.
It's that (confusing) terminology again; it's reporting the file fragmentation not the cluster consolidation status.

To put that another way - If you look at the drive map the % fragmented is the red blocks, (clusters with fragmented files in them), Consolidation isn't too bothered about those, it's just trying to fill in the 'white holes'.

So if Defraggler is working on consolidating the clusters on the drive it's just moving about the blue and red blocks, and that % fragmented won't change or it may even go higher as Defraggler fragments some files to fit them in to partially filled clusters.
(We do get people asking here why running a full defrag has made their % fragmented go up).
Don't worry about it, even if you see it increasing, a 'files only' defag will sort it out later.

If you do stop the full defrag again then you could do a 'files only' next time to get that % down.
Then do an analyze to see what the cluster distribution looks like.
If it's still full of 'white holes' then you can go back to full defrag/consolidate again if you still want to try and fill them in.

[Augeas]

Nerds' corner here, I'm afraid...

 

12 hours ago, nukecad said:

I'm not sure just how much of which Windows 'Defrag & Optimise' does for a HDD these days, but most defragmenters will do a consolidation because that's what most users think defragmenting is.

 

Do they? I've always thought of defragmentation being an attempt to make all the clusters of a file - the fragments - into one contiguous entity (I almost said consolidate them). I have no idea what everyone else thinks. Do most defraggers do a consolidation?

 

12 hours ago, nukecad said:

It is noticable that the Windows 'Defrag & Optimise' no longer shows a drive map, so no longer shows any consolidation happening on HDDs although I'm sure it still does it.

(If you have an SSD then a drive map is meaningless anyway, so that's one reason they may have stopped showing it).

 

It's not often appreciated that a defragger doesn't look at the drive to get its drive map. It just - in NTFS's case - looks at the Master File Table. That's the only place where file fragmentation is defined, as the entry for each file holds the cluster runs (one or many) that belong to the file. So the drive map is a mirage.

The drive map on an SSD would be the same, showing exactly the same fragmentation that would occur if the drive were an HDD. That's because it's NTFS that decides which cluster or group of clusters are to be used by a file, and NTFS doesn't care what the storage device is. Clusters are held as logical block addresses (LBA's), and correspond to physical addresses in an HHD. On an SSD they are mapped onto non-specific page addresses by the Flash Translation Layer in the SSD controller .

Personally I think that defragmentation is overblown. I have never defragged a disk in over 20 years (apart from a pre-sale cleanup). It might save a few milliseconds but I've just spent zillions of those typing all this.

 

 

[nukecad]

14 hours ago, Augeas said:

Do they? I've always thought of defragmentation being an attempt to make all the clusters of a file - the fragments - into one contiguous entity (I almost said consolidate them). I have no idea what everyone else thinks. Do most defraggers do a consolidation?

Well yes I'd say so.
Getting all the used clusters into one contiguous piece is consolidation.
A 'Cluster' is a specific defined area on a disc, not parts of a fragmented file. A file is fragmented because it's split over one or more disc clusters.

For years defragmenters have been doing consolidation, and so that's what people have come to think that defragmenting is.
That's not exactly wrong though - consolidation is 'defragmenting whole clusters on a disc' as opposed to 'defragmenting individual files'.
One is concerned with clusters, the other with files.
So both are technically 'defragmenting', but they are doing two very different things. Two things that can sometimes be at cross purposes*.

Most defragging apps have a drive map, people like to watch as it moves things about.
It wouldn't be as dynamic if it was just defragmenting the files.
You can see that if you do a files only defrag in Defraggler.
The blocks (clusters) will change colour but will only ocassionaly seem to 'move' as an emptied one turns white or a newly used one turns red/blue. (of course they also turn green/yellow as the data in them is being processed).

We occasionally see someone asking how to get "all the files/clusters at the start of the drive with no gaps", which is the holy grail of consolidation and can hardly ever be achieved. You realy only ever see it on a newly formatted drive.
But if you want to try and do that and fill up the clusters so that you are using the least number of clusters possible then you have to fragment some files to fill the clusters completely.
*(That's when the % fragmentation shown by defraggler can sometimes go up following the default defrag; because it's fragmented some files to fill up and consolidate some clusters).

Quote

Personally I think that defragmentation is overblown.

I totally agree with that too.
Unless there has been very heavy usage and thousands of writes/deletes on the drive over time then defragging is not realy needed.

It may have been useful back in the days of smaller discs and slower operating systems - particularly in the days of floppy discs when it could take 10 minutes or longer to load a programme from floppy (wasn't unknown to take half an hour to load a game from floppy) - but isn't realy needed these days.

In fact I'd say the only real reason most people defrag nowadays is to see a 'pretty' drive map - and if they didn't have a defragmenter in the first place they wouldn't even see a drive map to know if it looked pretty or not.
(Which could be another reason why Windows have dropped theirs).

As you say, unless your files are scattered all over the disc through heavy use, then in use the difference a consolidation/defrag makes is not going to be noticable at all.

[Augeas]

First of all I apologise to Legendarysnake for pinching his thread, and for not offering any solution. Perhaps Defraggler is still using the old transactional defrag API's and this causes the slow progress on large drives, but I don't know enough about this to go any further. My advice is that if you are having problems with Defraggler then revert to Windows defragger, or Storage Optimizer or whatever it's called now. It's free, you have it, and it works.

Right. I don't know if this is due to the vagaries of English language, or the difficulties of expressing oneself accurately and concisely (yep, that's me), or some misapprehension somewhere, but there's a lot here that's confusing.

 

3 hours ago, nukecad said:

Getting all the used clusters into one contiguous piece is consolidation.
A 'Cluster' is a specific defined area on a disc, not parts of a fragmented file. A file is fragmented because it's split over one or more disc clusters.

 

A disc from the factory is formatted ineradicably into sectors of a specific size. A cluster is specified when formatting with NTFS as a multiple of contiguous sectors. My 500 gb disc, for instance, holds around 975 million 512-byte sectors, and around 122 million 4096-byte clusters. Clusters are addressed by LBA's, as can be seen with Recuva. Like the drive map, clusters are a logical construct.

A file is one or more clusters. If a file has multiple clusters allocated, and they are contiguous, the file is not fragmented, or if you like it has one fragment. If the clusters aren't contiguous the file is fragmented. Defragmentation is reallocating the file's clusters so that they are contiguous.

 

2 hours ago, nukecad said:

For years defragmenters have been doing consolidation, and so that's what people have come to think that defragmenting is.
That's not exactly wrong though - consolidation is 'defragmenting whole clusters on a disc' as opposed to 'defragmenting individual files'.
One is concerned with clusters, the other with files.

 

This is unfortunately incorrect. You cannot 'defragment.. whole clusters on a disc', it doesn't make sense. You can only defrag files, and files are constructed of one or more clusters. A cluster only holds data from one file.

 

2 hours ago, nukecad said:

Most defragging apps have a drive map, people like to watch as it moves things about.

It wouldn't be as dynamic if it was just defragmenting the files.
You can see that if you do a files only defrag in Defraggler.
The blocks (clusters) will change colour but will only ocassionaly seem to 'move' as an emptied one turns white or a newly used one turns red/blue. (of course they also turn green/yellow as the data in them is being processed).

 

I think that this is where the confusion lies. To quote Piriform's Defraggler help pages, 'The blocks in a defrag drive map represent multiple clusters. Each square in the Drive Map represents a section of your hard drive. The size in KB (kilobytes) or MB (megabytes) of the section displayed depends on the size of the drive; the bigger your drive, the larger the storage area represented by each colored square. The order of the squares is the same as the order of sections of your hard drive - the square in the upper-left represents the first section of the drive.'

The blocks aren't clusters. If they were there would be 240+ million of them for Legendarysnake's 1tb drive. Yes, people like to see the map change colour, we're all children deep down.

 

3 hours ago, nukecad said:

But if you want to try and do that and fill up the clusters so that you are using the least number of clusters possible then you have to fragment some files to fill the clusters completely.

*(That's when the % fragmentation shown by defraggler can sometimes go up following the default defrag; because it's fragmented some files to fill up and consolidate some clusters).

 

I'm sorry but this is er, complete nonsense.

All this of course is in offered in the spirit of greater understanding.

 

[nukecad]

I agree that we seem to have digressed from Legandarysnake's issue, I also apologies for that.

@legendarysnake please ignore our debate (unless you want to read it) - how are you getting on now?

Yes I realise the the blocks in the drive map are not actual disc clusters, it was just a shorthand to refer to them like that.
(Slack language I agree).

I'll just quickly comment about fragmentation percentage possibly growing when you consolidate a drive.
It isn't usual to see it happen with the default 'Defrag', but it can happen.
I've seen it happen myself on a 'messy' drive, and we occasionally see it in questions asked on this forum.
In Defraggler itself a 'Defrag Freespace' is a consolidation - and it gives you an option to allow it to fragment files or not while it's doing it.
(TBH I'm not sure which one a default 'Defrag' does, but from previously asked questions assume that it will allow fragmentation - I guess we'd need one of the dev's to give a definitive answer to that).

From the Defraggler documentation:

Quote

If you select Defrag Freespace, Defraggler will fill up free space chunks with whole files only. If you select Defrag Freespace (Allow fragmentation), Defraggler will use file fragments to fill up free space,

[legendarysnake]

To be honest, im out of blue of this discussion about clusters haha. 

nukecad and augeas its interesting to read about your discussion im also happy that you guys are learning stuff with it.

 

So about the defrag. im really losting my patience. Yesterday i moved a single file to my disk before do the defrag and voila. now its 112gb fragmented files instead of just 50gb. So i had an idea. Since i have lots of files(specially image files) im compressing then to .rar to see if theres any diference. So instead of 200 files of images now i have a single .rar. 

I have used Treesize and used a option to see how many files does my hdd have. at the moment i have 32.526 files..

So like i said, im trying to reduce compressing it. When defraggler is doing his work, i noticed that a single .jpg file can take 5 seconds to be defragmented. 

 

Anyway, what you guys think about my idea to reduce the file using winrar to make the defraggler faster? will it work?

[nukecad]

Hi again @legendarysnake I'm not sure just what is going on for you.

But reading back through your posts I realised that we may be missing a piece of information that could be relevant.

You've said that it's a 2TB drive, but how much free space is there left on it?

The things you describe are the kind of things that can happen if there isn't enough free space on the drive.
It's usually recommended that you have 15% to 20% free space to sucessfully defragment a HDD.
https://www.howtogeek.com/324956/how-much-free-space-should-you-leave-on-your-windows-pc/
 

Quote

If you don’t have 15% free space, Windows won’t be able to properly defragment the drive. Windows will only partially defragment the drive, and it will grow increasingly fragmented over time. However, this just applies to mechanical hard drives that need defragmentation, and not the solid-state drives generally found in more modern PCs.

If this does apply to your drive then you could try moving some files off it onto another drive, that should then give the free space needed for a successful defrag.

PS. There is a way to do a 'manual' defrag of an external drive yourself, if you have a spare drive. (One that doesn't contain already fragmented files).
Simply moving/copying a file to a different drive, one that doesn't contain already fragmented files, will write that file to the new drive all in one unfragmented piece.
Moving/copying a bunch of files to another drive should do them one at once, thus defragmenting them in the process.

That's basically what a defragmenter does but all on one disc, it moves files and writes them in one piece to a free part of the disc, (then later moves them back when it's moved enough files to make sufficent continuous free space where they used to be), which is why it needs a percentage of free space to move the files about.

I'm not too sure about your compression method, is it actually moving the files?, but if you are compressing files onto another drive then it should do the same thing.

[nukecad]

@GuelphiteI have moved your question into it's own thread, click HERE to be taken to it.

 

[trium]

...

what size in the middle have your fragmended files?

190 - 248 mb?

 

how full is your 2 tb hdd? not only this 100 or 170 gb or?

with usb 3 connection perhaps it would be faster (defrag) if you copy your whole drive to another (and back again - with another external can you save this time of course :-) ). perhaps this can help faster as 13 hours or so

on the other side - with usb 2 it seems to be also fine, the external hdd cant read the files faster

[legendarysnake]

On 15/03/2022 at 12:33, nukecad said:

Hi again @legendarysnake I'm not sure just what is going on for you.

But reading back through your posts I realised that we may be missing a piece of information that could be relevant.

You've said that it's a 2TB drive, but how much free space is there left on it?

The things you describe are the kind of things that can happen if there isn't enough free space on the drive.
It's usually recommended that you have 15% to 20% free space to sucessfully defragment a HDD.
https://www.howtogeek.com/324956/how-much-free-space-should-you-leave-on-your-windows-pc/
 

If this does apply to your drive then you could try moving some files off it onto another drive, that should then give the free space needed for a successful defrag.

PS. There is a way to do a 'manual' defrag of an external drive yourself, if you have a spare drive. (One that doesn't contain already fragmented files).
Simply moving/copying a file to a different drive, one that doesn't contain already fragmented files, will write that file to the new drive all in one unfragmented piece.
Moving/copying a bunch of files to another drive should do them one at once, thus defragmenting them in the process.

That's basically what a defragmenter does but all on one disc, it moves files and writes them in one piece to a free part of the disc, (then later moves them back when it's moved enough files to make sufficent continuous free space where they used to be), which is why it needs a percentage of free space to move the files about.

I'm not too sure about your compression method, is it actually moving the files?, but if you are compressing files onto another drive then it should do the same thing.

Sorry for taking so long to reply. I have 561gbs free(30%). Well, after that i compressed to rar and got ride of a lot of files(since they are compressed now). The defragmentation went up to 20% of the disk. Heres my current stats of the drive. And how many files are on it(courtesy of tree size)

 

[nukecad]

Looks like it's getting there eventually, if slowly, you are now down to 27 fragmented files from the original hundreds, and a 'guesstimated' 7 hours.

Im not sure if that's because you have collected/clumped them files into one or more rar's though.
And TBH I'm not sure if that was a good move or not, Defraggler now probably sees one very large file that needs to be defragged rather that a lot of smaller ones.
And it won't care just what is inside that file, just that the whole rar itself is in fragments.
Plus a larger file will need more space to put it into in one piece, so defraggler may now have to move more about to try and create that larger space.

Personally I'd prefer to see quite a bit more free space for moving large files about, but unless you can temporarily move some files off the disc to elsewhere then you have what free space you have.

TBH I think that, again personally, as long as I had another 2TB of drive space available (say a couple of 1Tb discs, or 4x 500GB's), I'd move all the files off the drive then move them back again.
Depending on the disc transfer speeds then it may be quicker doing things that way, as long as you have the extra storage of course.

 

[trium]

Ps: in your case with an "not os drive" try the "file defrag" like another post above

I mean this would work faster too.

[legendarysnake]

anyway thank you for the help so far, i just gave up.. for now. When i have my notebook back i will defrag using it, since it use less energy, it will probably take several days unti the defrag is over.