Complete destruction of file without destroying hard drive?

[lugoteehalt]

Shred is not guaranteed to completely destroy a file, it may be possible, at considerable expense, to rescue the file. Bad news for privacy.

But if the file was on a flash drive, memory stick, it would be destroyed completely, irreversibly wouldn't it? Flash memory is something to do with electrons in 'traps', it is difficult to believe that any trace of the previous memory is left when a new one is written in.

So for perfect privacy put sensitive stuff on flash memory and the state can screw itself.

Sorry if I'm going over old ground - not my part of the forest.

[retsaw]

But if the file was on a flash drive, memory stick, it would be destroyed completely, irreversibly wouldn't it?

No, it wouldn't. When you delete a file most filesystems just remove the pointer to the file and mark the space available to be reused, but the actual file will remain until the part of the disk it was on is reused, this is the case regardless of the underlying media, be it a hard drive, flash, or something else. What shred does is overwrite a file before it is deleted ensuring the data is no longer stored, but it doesn't work with journaled filesystems because they don't let you overwrite the file directly.

If you want to ensure you can't recover a file, then you want to overwrite the whole partition with random data, though it is probably better to ensure you only keep sensitive files on an encrypted filesystem and also use encrypted swap, though even this could potentially be circumvented by rebooting your computer and recovering the encryption key from RAM if you keep the filesystem mounted and the attacker can get your computer while it is on/suspended or shortly after you turning it off, but once you delete the key and leave the computer off for for say 20 minutes* then the file should not be able to be recovered.

*I think the limit for recovering data from RAM is 15 minutes after the computer has been turned off, that is unless the RAM chips are cooled.

[Calipso]

I think thats why he mentions 'shred'. I think it does more than just run rm on teh file.

just a guess.

[retsaw]

Yeah, I just re-read what he said. In any case, it hasn't been proven that is is possible to recover data that actually has been over-written on a modern hard drive, that doesn't mean it is impossible, just that if it is possible the people who can do it have kept it a secret. And I wouldn't really be sure that overwriting a file on flash means it can't be recovered, especially if the flash drive has a controller that does some form of wear-levelling. So if I wanted to be sure of my privacy I would use encryption. Though a drawback with using encryption to hide data in the UK, is that you have to give up your encryption keys to the police when asked or face a jail sentence.

[folkert]

So if you want to keep some files, en completely remove other files on the same partition, you must move the files you want to keep to another device of partition, and then shred the complete partition instead of just the files.

[irlandes]

Some months ago, I asked if there were a Linux utility which functioned like Norton's Utility used to on Windows. Someone pointed me at hexedit, because Khexedit would not work.

In fact hexedit let me read and edit down to the byte in the HD.

It might be a bit of work, but perhaps there is some built in way to enter the file and change the bytes to some other value? I don't have time to check it now. But, it does indeed let you see and change the bytes in a HD file as opposed to shredding an entire partition.

[JayMan8081]

You could also look into srm which is part of the secure_deletion toolkit. From the man page:

srm is designed to delete data on mediums in a secure manner which can not be recovered by
thiefs, law enforcement or other threats. The wipe algorythm is based on the paper "Secure
Deletion of Data from Magnetic and Solid-State Memory" presented at the 6th Usenix Security
Symposium by Peter Gutmann, one of the leading civilian cryptographers.

[ArgPirate]

When you get right down to it, some one with enough determination (read money) has a chance of recovering files blown of the disk, no matter what the method. Once you shred or secure rm a file there is still a chance of being able to recover what the bits "used" to be set at, regardless of what the secure delete algorithm decides to write over it with.

Of course this type of discovery is not cheap and its not something that would not likely be used for anything short of "fate of the country" type situations. If you're really worried about it the best approach would be to get a new disk, and install on an encrypted partition (and swap) as well as doing the "secure deletes".

In any case, I don't see how a usb stick would be considered perfect privacy.

[lugoteehalt]

Point I was trying to make when starting this thread was that the shred (which repeatedly overwrites the file in all sorts of ways trying to remove all traces from the magnetic surface) man page states that it cannot be relied on completely, the only way to be absolutely sure is to smash the hd and set fire to it.

(Also you should shred the entire device, partition, on a journalled file system. So you have a special partition for sensitive stuff.)

Was assuming this still true.

(Didn't know about RAM keeping stuff for 20mins, perhaps a closedown program could run overwriting the stuff in ram?)

The kernel is that no trace is left of a file, when it has been overwritten, with flash memory. So the problem with hd's, that traces of previous files remain in the magnetic coating, is obviated. After all they cheap laptops only have flash and no hd.

http://electronics.howstuffworks.com/flash-memory.htm

[bwkaz]

The kernel is that no trace is left of a file, when it has been overwritten, with flash memory.

Except that's wrong in at least one case.

Flash drives require wear levelling, because each bit in a flash setup can only be overwritten some number of times. Eventually the writes stop working. I'm unsure what the number of cycles is, but I believe it's somewhere in the range of a couple hundred thousand -- once you write to a given bit that many times, it becomes useless. And if your flash drive is constantly being written to in the same place (say, the FAT file index, or the ext3 superblock), those bits will die first.

There are two ways to do wear levelling (that I know of) -- do it in the filesystem, where the FS code moves data around on the drive to try to ensure that most of the bits get written to at about the same rate, and do it on the device, where the FS just writes out to certain block numbers, and the device moves those (logical) block numbers around in its storage to keep the bit write patterns semi-constant.

Now, if the device is doing this wear levelling, then that means that two consecutive writes to the same (logical) block are going to actually put their data in two different (physical) blocks. So you can't overwrite data, because the device won't let you. Someone who was able to yank the flash chips out of the device and read them without the device's wear levelling controller in the line will be able to recover that data (unless the device has randomly chosen to overwrite those physical blocks, but you have no control over this).

(hexedit-type programs don't help here either: they only give you the logical block view of the device.)

If the device does not do wear levelling on its own, then it might be safe, I'm not sure. But if it does do wear levelling, then it definitely is not safe.

[lugoteehalt]

Except that's wrong in at least one case.

Flash drives require wear levelling,

Have no idea what I'm talking about but this occurs:

In order to speed reading or writing or both, don't know, a flash drive blanks relatively large chunks of memory and then re-writes them; when it is written to http://electronics.howstuffworks.com/flash-memory.htm. It therefore seems, prima facie, likely that when something is erased it actually is erased, and not merely marked for subsequent overwriting should the need arrise.

Purest speculation I grant.

[Icarus]

i don't know about that...when I run a deep scan with Recuva on my flash drive I'm able to restore things that would typically have been overwritten by now. Granted it is FAT which is the easiest of easy to restore from...

One report I read said that with a 4GB flash drive it would take about 10 years of consecutive read/writes to burn out every single block.

"dd if=/dev/zero of=/dev/sda" is the only 100% real way to wipe data and make it unrecoverable, but that gets the entire drive and can't be directed at a single file. I don't trust and 'secure' delete 100%, maybe 40% as there are 100's off tools that can be used to recover data. I know I've used at least a dozen in windows and linux with success

[bwkaz]

In order to speed reading or writing or both, don't know, a flash drive blanks relatively large chunks of memory and then re-writes them;

Not all drives do that though.

http://electronics.howstuffworks.com/flash-memory.htm.

Um, yeah. howstuffworks is generally OK, but almost never gives you the whole story.

It therefore seems, prima facie, likely that when something is erased it actually is erased, and not merely marked for subsequent overwriting should the need arrise.

But the flash drive's electronics are what determine what happens when that block is overwritten in the first place.

If the drive does wear levelling on its own, then it does logical -> physical block translation on every read and write request. Writes change the logical -> physical pointer for the target logical block #, while reads do not. To choose an example at random, the first write to logical block number 5 (with any data) would actually go to physical block 2346523; the second write to logical block number 5 would go to physical block 3656744512. (Or whatever.)

After the second write, the first write's data -- whatever it was -- is still available at physical block 2346523. The drive's firmware probably won't let you access this physical block directly (until it maps another logical block to it during a write, at which point it overwrites the old data) -- but you can remove the flash chip(s) and hook it/them up to a reader yourself; at that point the reader would use the physical addresses.

This changing of logical -> physical pointers (and actually, the presence of logical -> physical pointers in the first place) is the whole point of wear levelling, and the only reason to do it is to keep the rate of erasure (close to) constant across the entire device. From what I can tell, it's also fairly uncommon in flash drives (especially in cheap ones). But there are some that do it.

(And actually, it occurs to me that it may be possible to have more physical flash on the device than you report to the OS, as well. One reason to do this would be to use the extra as spare write targets for when a given block gets too many write/erase cycles, but no other block is available to swap it to. It's a waste of some flash storage, but you can increase the lifetime of the entire device by doing it, so there is a tradeoff. If the device does that, then even overwriting the entire device won't necessarily erase every physical block -- but overwriting the entire device several times should do it, eventually. No way to verify though, short of taking the flash chips out and skipping the device's firmware...)

[lugoteehalt]

I submit.