If what you say is correct, then there'd be no point in having a recycle bin at all.
Since most programs put piles of garbage on your machine, I often try deleting files that seem to be using up disk space for no useful purpose. If the program that put them up then refuses to work, I restore them. That is quite separate from backing up, which is obviously important.
Incidentally Norton's NProtect, for all its other faults, does provide timed removal of deleted files.