Suggestion for the new version of RECUVA

[Willy2]

- Add an option/ options to hide all files that are "Unrecoverable" / "Poorly recoverable" / etc. Present these options to the user before RECUVA analyzes a drive.

- Add one or more options to overwrite entries in a directory file. Or include "overwrite directory file entries" program code in the "overwrite selected file(s)" subroutines that are already available/used in the current version. As far as I know it's impossible to directly write to/ erase information from the Master File Table (MFT). That's why the program must overwrite entries in the directory files  itself. Perhaps it's possible to overwrite a directory entry with e.g. zeros (or another character) that would signal to the NTFS that such a directory entry is "vacant / empty" ?

- The developers of Recuva could take this one step further. Perhaps it's possible to even reduce the size of a directory file itself. So, if a directory file contains say 100 entries of which say 30 refer to files that don't exist anymore or are "securely overwritten" then perhaps it's possible to reduce the size of that directory file to the remaining 70 entries.

The reduction of the size of the directory file could happen under a separate option or included in the "Securely overwrite files" subroutines.

- Then the "Securely overwrite files" program code could contain the following things/subroutines :

1)) Securely overwrite files

2)) Overwrite "empty" directory file entries.

3)) Reduce the size of directory files

- In the past I had a program called "Clean Disk Security" ( http://www.theabsolute.net/sware/clndisk.html CDS) that was able to reduce the amount of directory entries in a directory file. Several times in the past I ran RECUVA before and after running CDS. As a result of running CDS I noticed that the amount of e.g. "unrecoverable files" (as reported by Recuva) shrank (very) dramatically. Unfortunately I don't have that version (v7.xx) of that program (CDS) anymore.

[Willy2]

- And here is a bug that need to be fixed in the next version of RECUVA:

- When RECUVA is busy analyzing drives then something very odd shows up. The program shows a ridiculous high %  and a very high negative %. Very, very odd. See the 2 pictures attached.

- Question: Do compressed files make these ridiculous high % show up in the program ? Like they do in CCleaner ?

[Augeas]

3 hours ago, Willy2 said:

- Add an option/ options to hide all files that are "Unrecoverable" / "Poorly recoverable" / etc. Present these options to the user before RECUVA analyzes a drive.

Yes, could be an Option in Options/Actions. The scan would still be as long as before, as all records in te MFT have to be scanned to see if they are recoverable or not.

Quote

- Add one or more options to overwrite entries in a directory file. Or include "overwrite directory file entries" program code in the "overwrite selected file(s)" subroutines that are already available/used in the current version. As far as I know it's impossible to directly write to/ erase information from the Master File Table (MFT). That's why the program must overwrite entries in the directory files  itself. Perhaps it's possible to overwrite a directory entry with e.g. zeros (or another character) that would signal to the NTFS that such a directory entry is "vacant / empty" ?

A directory is a record, or number of records, in the MFT. NTFS will reduce the size of the directory on file deletion, shuffling the live entries up to overwrite the deleted entry. There are no references to deleted files in an NTFS directory. The MFT is heavily protected and any writes to it with, for instance, a hex editor are backed out in seconds. There is no concept of a 'vacant' directory entry.

Quote

- The developers of Recuva could take this one step further. Perhaps it's possible to even reduce the size of a directory file itself. So, if a directory file contains say 100 entries of which say 30 refer to files that don't exist anymore or are "securely overwritten" then perhaps it's possible to reduce the size of that directory file to the remaining 70 entries.

See above. Directory size is maintained  by NTFS.

Quote

The reduction of the size of the directory file could happen under a separate option or included in the "Securely overwrite files" subroutines.

See above.

Quote

- Then the "Securely overwrite files" program code could contain the following things/subroutines :

1)) Securely overwrite files

2)) Overwrite "empty" directory file entries.

3)) Reduce the size of directory files

See lots of above.

Quote

- In the past I had a program called "Clean Disk Security" ( http://www.theabsolute.net/sware/clndisk.html CDS) that was able to reduce the amount of directory entries in a directory file. Several times in the past I ran RECUVA before and after running CDS. As a result of running CDS I noticed that the amount of e.g. "unrecoverable files" (as reported by Recuva) shrank (very) dramatically. Unfortunately I don't have that version (v7.xx) of that program (CDS) anymore.

I couldn't say what that was doing. Some older software did things that are not legit or possible today.

(All the above is 'Generally speaking'. There can be exceptions.

[Augeas]

3 hours ago, Willy2 said:

- And here is a bug that need to be fixed in the next version of RECUVA:

- When RECUVA is busy analyzing drives then something very odd shows up. The program shows a ridiculous high %  and a very high negative %. Very, very odd. See the 2 pictures attached.

- Question: Do compressed files make these ridiculous high % show up in the program ? Like they do in CCleaner ?

 

I don't know, I've never seen this. All I get is an increasing percentage to 100% when the scan is finished.

[Willy2]

- Disagree. The directory file(s) and the MFT are 2 different things. The directory file(s) can be defragmented whereas the MFT can't with a program called Defraggler. Didn't you notice ? Never used Defraggler ?

[Augeas]

You're quite welcome to disagree. An NTFS directory is a record in the MFT. A large directory may have several MFT records. An exceptionally large directory may have a separate index cluster allocated. I don't know whether we are using the same definition for directory.

By the way I have just edited my big response, please use the later version.

More by the way, no, I have never used Defraggler.

[Willy2]

- Fragmented directory files shown in Defraggler: See the picture inside the red rectangle.

- Options for "unrecoverable files": Agree. The program indeed still needs to analyze the entire drive. But including the "unrecoverable" and other "less recoverable" files in the file list overwhelms & confuses the user (too much).

[Augeas]

These Folder entries are the index clusters for large folders as described in my previous post, and indeed as described in

https://www.ccleaner.com/docs/defraggler/defraggler-settings/defraggler-options-advanced-tab

However the topic is Recuva suggestions, not Defraggler, and Recuva cannot amend the size of a directory, which includes these index clusters.

[Andavari]

5 hours ago, Willy2 said:

- Add an option/ options to hide all files that are "Unrecoverable" / "Poorly recoverable" / etc. Present these options to the user before RECUVA analyzes a drive.

+1

It would be so much better not having to even look at those. Although we can sort, but still.

[Willy2]

Quote:

"A directory is a record, or number of records, in the MFT. NTFS will reduce the size of the directory on file deletion, shuffling the live entries up to overwrite the deleted entry. There are no references to deleted files in an NTFS directory. The MFT is heavily protected and any writes to it with, for instance, a hex editor are backed out in seconds. There is no concept of a 'vacant' directory entry."

No, when a file is deleted then NTFS doesn't reduce the size of the directory & doesn't overwrite the deleted entry & doesn't shuffle all live entries up. If that was the case then RECUVA wouldn't be able to find & recover those "deleted files", would it ?

It wouldn't make sense either in another way. Imagine a (VERY) large MFT with A LOT OF entries. When I delete one file that has an entry at the very beginning of the MFT, then it would take NTFS A LOT OF / way too much time to "shuffle all live entries" to overwrite only one deleted entry. And that undermines all your 3 other replies as well.

In other words: There are entries that are "vacant" (or as Microsoft puts it: "unallocated").

[Augeas]

3 hours ago, Willy2 said:

Quote:

"A directory is a record, or number of records, in the MFT. NTFS will reduce the size of the directory on file deletion, shuffling the live entries up to overwrite the deleted entry. There are no references to deleted files in an NTFS directory. The MFT is heavily protected and any writes to it with, for instance, a hex editor are backed out in seconds. There is no concept of a 'vacant' directory entry."

No, when a file is deleted then NTFS doesn't reduce the size of the directory & doesn't overwrite the deleted entry & doesn't shuffle all live entries up.

Yes it does. Use a hex editor to find a directory record in the MFT. You will see that file names are held in ascending name order in the $Index_Root attribute. Delete one of those files and the remaining file names are moved up to fill the gap and an EOF marker overrides what was the last file name. Larger directories will have MFT extension records, and one or more Index clusters. The principle is the same. This is easy to observe with a small directory, and I have. There is no process I know that can flag a file within a directory as deleted. Show me an MFT directory record containing a deleted file.

Quote

 If that was the case then RECUVA wouldn't be able to find & recover those "deleted files", would it ?

Yes it would. Recuva doesn't look at the directory records, so it doesn't matter what's in them. Recuva looks for deleted FILE records in the MFT, and lists those. Deleted files are not listed in directory records in the MFT. Directory information for a deleted file is found by following the directory chain-back address in the deleted file's record. (As far as I understand from my experience using Recuva.) If you run Recuva you will see many deleted files listed that have no directory information - the directory records no longer exist, but Recuva finds the files. 

Quote

It wouldn't make sense either in another way. Imagine a (VERY) large MFT with A LOT OF entries. When I delete one file that has an entry at the very beginning of the MFT, then it would take NTFS A LOT OF / way too much time to "shuffle all live entries" to overwrite only one deleted entry. And that undermines all your 3 other replies as well.

This is a misunderstanding of the structure of the MFT and what has been said here. No MFT records are shuffled anywhere, nor has that claim been made. The shuffling is of file names and associated info within the MFT records for the directory, not MFT records themselves. My three other replies stand.

Quote

In other words: There are entries that are "vacant" (or as Microsoft puts it: "unallocated").

These are unallocated MFT records. They are not the same as file names held within a MFT directory record.

I'm beginning to wonder whether the terms 'MFT' and 'Directory' are being confused?

[Willy2]

Here the confusion has only grown. I understand some things in your explanation and some other things are still shrouded in confusion.

Do you have some weblinks that provide good information about what the structure is of the MFT and how the "Index ifles" (see a previous reply of mine in this thread) fit in the overall "MFT picture" ? You have layed out a number of dots. But I fail to see how all those dots (e.g. index files) connect with each other.

"Fiddling around" with those index records also has produced some interesting insights.

(Remember what my signature is for each post here on this forum. (Something with "A discussion ....... "))

[Augeas]

You could read through http://kcall.co.uk/ntfs/index.html  although it is heavy going. The part headed MFT Records, or MFT Extension Records, describes the index clusters (called Folder Entry in Defraggler) for a file, the principle is the same for a folder.

Microsoft sometimes calls directories indexes, and we call them folders. It is confusing.

The MFT is a file, which holds one or more 1k records for every file on the drive, including itself. A folder, or directory, consists of one or more records in the MFT. Large files, or large folders, may have separate index clusters allocated which hold the addresses of the many MFT records used by the file or folder.

Yes, I have noted your signature, and agree entirely. We're a long way from Recuva Suggestions, but it's fun, sort of.

[Willy2]

- I noticed something else: when I add files to a folder/directory then the index file grows. But when I remove one or more files from a folder then it seems the index file doesn't shrink. And from that outdated info is where RECUVA seems to pull up infomation, in an attempt to find/recover deleted files. So, when one deletes the folder itself the change of recovering anything is "very limited".

- I used to know how the filing system worked but that was in the "good old" FAT days. But it's clear NTFS works in a different fashion.

- I also had a MS-DOS program that was able to make the MSDOS directory files smaller. But alas, that only worked on a FAT filing system.

[Willy2]

- In a previous post in this thread I posted 2 pictures in which the RECUVA GUI showed an absurd %. Perhaps the program "miscalculated" that % as a result of scanning drives that don't use the NTFS filing system ? Because I have 3 drives that can be connected to an USB port on my laptop. Each of these 3 USB drive uses a different filing system (NTFS , FAT and ExFAT).

[Andavari]

On 29/04/2021 at 06:28, Willy2 said:

- I used to know how the filing system worked but that was in the "good old" FAT days. But it's clear NTFS works in a different fashion.

The way NTFS works (in Windows 10 at least) shocked me the other day, the MFT size is over 511MB on my C:\ drive SSD.

[Willy2]

1 hour ago, Andavari said:

The way NTFS works (in Windows 10 at least) shocked me the other day, the MFT size is over 511MB on my C:\ drive SSD.

- You can reduce the size of the MFT by using "Wipe Free Space" in CCleaner. When the disk is nearly full then one of the things Windows will do is reduce the size of the MFT as much as possible.

- In spite of being advised to NOT Defragment I still would advise to defragment an SSD drive. But limit it to say every 3 or 6 months. The reason is the way data is written to a SSD.

https://pureinfotech.com/why-solid-state-drive-ssd-performance-slows-down/

[Augeas]

4 hours ago, Andavari said:

The way NTFS works (in Windows 10 at least) shocked me the other day, the MFT size is over 511MB on my C:\ drive SSD.

That's nothing to worry about, it's only 500,000 live and deleted files. On my 120 gb C drive SSD my MFT is 472 mb, and I am only using 36 gb. Win 10 install allocates and deletes a lot, a very large lot, of files. Remember that large files, and large directories, will use multiple MFT records so the total file count is probably under 500k.

WFS will not touch the MFT, unless it's an entire disk erase. Windows does not reduce the size of the MFT, nothing does, apart from a reformat. When a disk is nearly full then NTFS will allocate files within the MFT Zone, which is not the same as reducing the size of the MFT.

[Willy2]

- Well, Defraggler tells a different story. Based on what I see there is that as the amount of files grows the MFT grows as well. But when the disk becomes fuller & fuller - at some point I see the MFT actually becomes smaller.

According to my info the MFT is always much larger than is required. So, if the amount of entries would occupy say 20 MB in the MFT then the MFT is say 60 MB. And that makes perfect sense. It means that Windows doesn't need to waste precious time on enlarging the MFT.

[Andavari]

I'll never use WFS on the Micron 1100 M.2 256GB OEM SSD, especially since works perfectly fine.

Now a Windows Full Format is a different story, because two weeks ago my Sk Hynix S31 Gold 2.5 500GB SSD supposedly had a bad sector according to ChkDsk but none of the other disk utilities found anything, and since there's hardly anything on it I cleaned it with DiskPart and then did a Full Format, now ChkDsk doesn't find any bad sectors.

Edit:
Observation: ChkDsk in Windows 10 v20H2 seems somewhat buggy, and it's the version that found "bad sector" on my Sk Hynix SSD that didn't really exist. With that version of ChkDsk I also have to boot with Installation Media (USB Flash Drive) to properly run ChkDsk on my C:\ drive, otherwise if I don't it will just get stuck at 100% completion.

Edited May 2, 2021 by Andavari

[nukecad]

I've had similar 'bad sector' problems with 20H2.

Recently I got a new SSD and 20H2 would not format it because of 'bad sectors' - another machine with 1909 formated it with no problem.