high CPU, low IO, slow defrag

I have a large drive (1.5TB) with lots of small files on it. Defraggler is using a lot of CPU (effectively pegging one core), and it's taking over 1 second to defrag each file.

I tried running the profiler on it, but since you guys stripped the symbols (why?), I can't give you anything interesting to go on... It seems to be spending a lot of time in EnterCriticlSection...

Slow file transfer with high CPU usage / interrupts could indicate the IDE controller is operating in PIO mode (not DMA) or the hard drive is pending failure.

Check your drive's SMART status using Speccy (or HD Tune) and see if anything looks suspicious i.e. Reallocated Sector Count > 0.

Richard S.

Slow file transfer with high CPU usage / interrupts could indicate the IDE controller is operating in PIO mode (not DMA) or the hard drive is pending failure.

Check your drive's SMART status using Speccy (or HD Tune) and see if anything looks suspicious i.e. Reallocated Sector Count > 0.

Richard S.

nope, the drive is definitely operating correctly.

i ran procmon.exe and it seems that one of defraggler's threads that is pegging the CPU is doing a GET_VOLUME_BITMAP after every file. i think this is related...

i'm going to run it under xperf to see if i can get anything more useful.

ok, i can't get anything helpful out of xPerf due to lack of symbols, care to share the PDB?

but it seems there are two threads which basically take turns pegging the core. almost all of the CPU time is spent inside the defraggler64.exe code, ie. it's not in kernel or other DLLs.

but it seems there are two threads which basically take turns pegging the core. almost all of the CPU time is spent inside the defraggler64.exe code, ie. it's not in kernel or other DLLs.

ok, one thread seems to spend a considerable amount of time looping through the items in the 'file types' dialog. it looks like it's doing this in an extremely inefficient manner: the 'lock xadd' in the following loop is causing the CPU to purge cache & sync in a tight loop. It looks like this is getting called for every file on the disk, for every file type in the types dialog. is that right? it would seem that moving the lock to a higher level would be a good place to start.

(defraggler64.exe is loaded at 0x13f510000)

000000013F55C7D0  mov         rdx,qword ptr [rbx]  
000000013F55C7D3  sub         rdx,18h  
000000013F55C7D7  or          eax,0FFFFFFFFh  
000000013F55C7DA  lock xadd   dword ptr [rdx+10h],eax  
000000013F55C7DF  add         eax,0FFFFFFFFh  
000000013F55C7E2  test        eax,eax  
000000013F55C7E4  jg          000000013F55C7EF  
000000013F55C7E6  mov         rcx,qword ptr [rdx]  
000000013F55C7E9  mov         rax,qword ptr [rcx]  
000000013F55C7EC  call        qword ptr [rax+8]  
000000013F55C7EF  add         rbx,10h  
000000013F55C7F3  cmp         rbx,rsi  
000000013F55C7F6  jne         000000013F55C7D0  

the other thread that's pegging the CPU is also looping through this list of file types. although it's a different piece of code. but it's also got the dreaded lock in a tight loop:

000000013F63E819  jne         000000013F63E826  
000000013F63E81B  lea         r12,[rdi-18h]  
000000013F63E81F  lock add    dword ptr [rdi-8],1  
000000013F63E824  jmp         000000013F63E866  

ok, here's a repro.

on a large drive with a lot of small, fragmented files, the following uses approximately 0 (zero) CPU:

df64.exe e:

the following pegs 1 core:

df64.exe e: /Large .arj 0 /Large .tar 0 /Large .rar 0 /Large .zip 0 /Large .cab 0 /Large .tar.gz 0 /Large .tgz 0 /Large .arj 0 /Large .lzh 0 /Large .7z 0 /Large .gzip 0 /Large .gz 0 /Large .uue 0 /Large .bz2 0 /Large .img 0 /Large .mig 0 /Large .mp3 0 /Large .wav 0 /Large .aiff 0 /Large .rm 0 /Large .ra 0 /Large .flac 0 /Large .bin 0 /Large .iso 0 /Large .mdf 0 /Large .dmg 0 /Large .vxd 0 /Large .avi 0 /Large .mkv 0 /Large .mpg 0 /Large .mov 0 /Large .mp4 0 /Large .m4v 0 /Large .mpeg 0 /Large .wmv 0 /Large .bup 0 /Large .ifo 0 /Large .vob 0 /Large .msi 0 /Large .msp 0 /Large .msu 0 /Large .wim 0 

ok, here's a repro.

on a large drive with a lot of small, fragmented files, the following uses approximately 0 (zero) CPU:

df64.exe e:

the following pegs 1 core:

df64.exe e: /Large .arj 0 /Large .tar 0 /Large .rar 0 /Large .zip 0 /Large .cab 0 /Large .tar.gz 0 /Large .tgz 0 /Large .arj 0 /Large .lzh 0 /Large .7z 0 /Large .gzip 0 /Large .gz 0 /Large .uue 0 /Large .bz2 0 /Large .img 0 /Large .mig 0 /Large .mp3 0 /Large .wav 0 /Large .aiff 0 /Large .rm 0 /Large .ra 0 /Large .flac 0 /Large .bin 0 /Large .iso 0 /Large .mdf 0 /Large .dmg 0 /Large .vxd 0 /Large .avi 0 /Large .mkv 0 /Large .mpg 0 /Large .mov 0 /Large .mp4 0 /Large .m4v 0 /Large .mpeg 0 /Large .wmv 0 /Large .bup 0 /Large .ifo 0 /Large .vob 0 /Large .msi 0 /Large .msp 0 /Large .msu 0 /Large .wim 0 

This guy should get free Defrag support for life ;) Nailed it, I guess.

Hi,

thanks for all information. We will look into this.

Best regards

Romanoff

Having the "move large files to the end" also seems to make it more CPU intensive ("pegging" one core).

edit: or maybe not

I have the same problem, high cpu usage, slow defrag.

(it was fast for 11%, then it slowed down, and started using 100% cpu)

post-43661-0-17537500-1296171034_thumb.png

Hi,

thanks for all information. We will look into this.

Best regards

Romanoff

did this ever get fixed? i'm still seeing very high CPU consistently. I'm wondering if it's still due to all those InterlockedIncrements?

it looks like you're iterating over a vector<shared_ptr<>>, very bad... you need to optimize for the common case, not the rare one...

Spongman as this is an old thread I just want to make sure that you are seeing this in the most recent version of defraggler

Spongman as this is an old thread I just want to make sure that you are seeing this in the most recent version of defraggler

yes, still seeing it.

please try the two commands I gave in post #5 and compare CPU usage...

I'm also having the same problem with the most recent release of defraggler - are there any news on this? Defraggler estimates "> 1 day" for defragmentation (on a 500 GB drive)....

yep, this user is seeing unwarranted slowness since v 2.09.391 was released although the option to "quick defrag" seems to operate at more like the speed of the old release.

I am not sure if this happens since the 2.09 release only.

But this is a definitly a bug which should be fixed - i've never seen a defrag application which is cpu-bound instead of io-bound.

i'm STILL seeing this (see comment #5). i also see a bunch of high CPU comments on the forums, i'm pretty sure this remains the issue.

any word on a fix for this?

Sorry for wake up an old thread, but I have the same problem (high cpu usage, 1 core / really low defrag speed, few KB/sec or 1 file/1-3 sec). I'm using the latest version.

Any chance to fix it?

Sorry for wake up an old thread, but I have the same problem (high cpu usage, 1 core / really low defrag speed, few KB/sec or 1 file/1-3 sec). I'm using the latest version.

Any chance to fix it?

Many people do not have a problem.

If half a dozen people posted system specifications then common factors might become obvious for the developers to investigate, e.g. :-

Operating System;

x86 / x64;

System Drive / Secondary Drive / External drive;

Make of HDD.

Hint, Hint :)

Win 7 SP1 Ulti x64 (Full updated)

Intel Core 2 Quad @ 2.83GHz / 8GB DDR2 @ 1066MHz

Sys hdd : WD 1TB Caviar Blue / Sec hdd : 2x WD 640GB Caviar Blue @ Raid 0

The problem is system independent, at least hardware independent..

While defragging a partition I monitored it with HDTune and I saw something strange, after each file (from 10KB small to 1GB big) there is a 2-3 sec pause. While defragging small files ( <10MB) defraggler use 1 core of my cpu..

Why 3 sec pause after each file? Why use 1 cpu core while defragging small files ( <10MB) ?

Definitely there is a big problem in program's algorythm.. Please fix it, it is a good program.

Edit: I forgot the images

post-65849-0-78433000-1363913470_thumb.png

post-65849-0-77575900-1363913477_thumb.png

are you running the same commands as in post #5 ?

does the fact you are running HDTune while defragging skew the result ?

what else is running ?

could your RAID 0 be adding an overhead as the data blocks span 2 drives ?

does the same issue happen in Safe Mode ?

for testing purposes, have you created another partition, copied some files across and ran DF on that, or on another PC for that matter ?

if it was still a big problem, I suspect there would be many more posts, so in the absence of those, you have to treat it as something particular to your setup.

so begins the process of elimination....