-
i845, You could compress the partition image with a command similar to this:
cat /dev/hda | bzip2 -c > image.bz2
-
Originally Posted by Sepero
i845, You could compress the partition image with a command similar to this:
cat /dev/hda | bzip2 -c > image.bz2
Thank you. That's exactly what I wanted!
Just to clear things up a bit, will the above command give me a bit-by-bit replica of my partition? I mean, I've used cat to insert/retreive text from files, but have never tried anything like this (theoritically, though, cat should do the job, as it will output the contents of /dev/hda1, like it would for any other file)....
Just wanted to be sure.
Last edited by i845_; 02-05-2007 at 12:12 PM.
-
Extract from AwesomeMachine's thread
Code:
This will make a file that is an exact duplicate of the sdb2 partition.
You can substitue hdb, sda, hda, or whatever the disk is called. or
Code:
dd if=/dev/sdb2 ibs=4096 | gzip > partition.image.gz conv=noerror
Makes a gzipped archive of the entire partition. To restore use:
Code:
| gunzip >
for bzip2(slower,smaller), substitute bzip2 and bunzip2, and name the file
Code:
.bz2
I have not use cat on a device like Sepero suggested but cat has joinned up several CD iso images into a DVD iso and so should be able to hanle binary patterns without problem.
The product of dd is always a replica of the original. The compression and decompression with the same tool should return the same replica image.
I prefer to save the image exactly as the partition in uncompressed form as that allows me to cherry-pick individual files for restoration.
-
I would use gzip rather than bzip2, bzip2 will produce smaller files but is a lot more processor intensive so will take longer unless your machine is powerful enough to compress the data at the speed it can be read from your hard drive.
-
- bzip2 gives a LOT better compression if there are huge amounts of zeros and/or ones in a row.
- gzip sometimes (very rarely) gives better compression.
- In my typical experience, bzip2 takes about 3 times longer than gzip.
You can make a single Linux partition HIGHLY compressable by writing zeros to it's free space. (How do you do that?) I'm glad you asked. Run a command like this:
cat /dev/zero > file; rm file
You should make sure that 'file' is created on the partition you want to compress. For those that don't understand what this does, it creates a file filled with nothing but zeros. It fills it until your partition is 100% full. That way it consumes all of your free space.
After your partition is completely filled, it immediately deletes the file, giving your free space back. The zero's remain in your free space.
(It's also a really fast, generic way to prevent someone from recovering files you've deleted. Not as good as shredding though.)
Originally Posted by i845_
Just to clear things up a bit, will the above command give me a bit-by-bit replica of my partition?
It's kind of funny that you should mention that because when I wrote it, I was considering skipping 'cat' completely. Just using bash redirection can be kind of cryptic though:
bzip2 -c < /dev/hda > image.bz2
Yes, the 'cat' command will do bit-by-bit replica of anything, including block devices. Saikee's suggestion with 'dd' is better though (my suggestion with 'cat' was just because I'm lazy ). Using 'dd' or 'dd_rescue' is better because if setup correctly, you can run it on devices that give errors. When using 'cat' it will immediatly fail on any device error.
Originally Posted by saikee
I prefer to save the image exactly as the partition in uncompressed form as that allows me to cherry-pick individual files for restoration.
If you're wanting to make a compressed image (of a Linux filesystem) that you can also extract individual files from, you might want to look into squashfs. You may already know that it is used to created most LiveCD's today.
I hope you all are able to understand my post here. I put in more info than I thought I would. Please don't hesitate to post further questions.
-
@Sepero
Yup, I have seen that command (cat /dev/zero > file; rm file) before... on a previous post of yours. I didn't understand its function then. Many thanks.
This topic just keeps getting more and more interesting. It's really amazing how combining small utilities under Linux gives you so much power...
@saikee
This might be off-topic, but cat joining ISO CD images into DVD images sounds interesting...
Last edited by i845_; 02-06-2007 at 05:07 AM.
-
First off I'd just like to say what a great little walk through you have written here saikee. It is very interesting and educational. My computer org teacher actaully sent me an email regarding this very thing and at the time, having no use for it, I just blew it off. I am now currently in a situation at work where this would be very useful and was wondering if someone could possibly help trouble shoot or give me some pointers on what to do for I am relatively new to dd and imaging/cloning in general.
I am in the process of swapping out two 80 gig SATA drives from two Dell Optiplex 745's here at work, and replacing them with smaller 40 gig SATA drives, since they are student machines and will not need all 80 gigs. So I started to RIS one of them when I thought to myself, "I really don't want reinstall all the little tedious things when I remembered the write up my teacher did about something very similar to this. Since I have since deleted his email, that was no longer an option and I decided to google it. I found this thread and was very impressed with the write up.
It wasn't until I was just about to do the DD command that I noticed you said the destination drive needs to be either the same size or larger than the source drive. I noticed that later on in the thread you referenced such tools as Parted Magic (which by the way is great, and I thank you so much for recommending it and testing it out). So using that tool I decided to shrink the 80 gig partition (on the source drive) down to about 38099MiB (39950MB) so I could just DD the partition rather than the entire drive (which obviously wouldn't work). So I decreased the size of the partition using Parted Magic, and than booted up in Knoppix 5.1 and proceeding onto doing the DD command.
Once it completed I booted down the machine. Took out the 80 gig (source) drive and left only the destination drive in there. Attempting to boot into windows, nothing happened. It just sat there. So now I have booted into Knoppix, the hard drive appears but upon clicking on it, it says the following;
"Could not mount device.
This reported error was:
Failed to read last sectory (78027696): Invalid argument
Perhaps the volume is a RAID/LDM but it wasn't setup yet, or the
wrong device was used, or the partition table is incorrect.
Failed to startup volume: Invalid argument
Failed to mount "/dev/sda1": Invalid argument
The device "/dev/sda1" doesn't have a valid NTFS.
Maybe you selected the wrong device? Or the whole disk instead of a partition? Or the other way around?"
Also wanted to make note that I followed all saikee's steps, created the new partition, rebooted the computer, etc. Also I did get a sucessful DD message at the end saying everything had copied over.
Is there something that the DD is missing since I am only copying over the partition and not the entire drive? Is it even possible to do this method going from a larger driver to a smaller? Any ideas or suggestions would be greatly appreciated!
Last edited by jonathansmith68; 06-26-2007 at 12:23 PM.
-
Welcome to JustLinux!!!!!
In your case I would run the XP in its reduced size format to satisfy that it likes the new home first before cloning it.
In cloning of a partition you need to make sure the number of sectors, not the Mb, matches exactly between the source and the target because the transfer is done in sectors.
It looks like you have not managed to have the sectors exactly matched because there is a report of error in the last one. Thus the partition table failed to satisfy XP's integrity check and the system isn't boot because XP would not know how to fix it.
My suggestion is to resize the partition well below 40Gb, run it first and make sure the number sectors matches exactly in the cloning process. Once the clone disk operates you can use Gparted to resize the XP to take up the full disk.
-
Thank you for the such quick reply! I am taken your suggestions into consideration and I am currently re-shreding and re attempting to DD the now-smaller partition onto the destination drive. Also in response to one of your tips, I did in fact boot up windows last time with the 80 gig partition shrunken down to 39 gigs. Surprisingly after a quick disk check it booted up just fine! No need to do a fix mbr.
I do have one question though. I noticed on your first post, in the code section it says to enter the command "dd if=/dev/hda1 of=/dev/sda1 bs=32256" and then in the explanation it says that you used the number 32768. I scrolled down to check the number you used in section B and this time you had used 32768 in the code section. I assumed the 32256 was just a typo and was just curious on which the first code section was supposed have.
Again, thanks for all the tips, hints, I'll have to try my second attempt and report back how it works!
-
To be honest I have not bottomed out the bs parameter myself but both numbers are about the optimum in my experience.
The 32768 is from multiplying 512 bytes per sector with 64 sectors.
The 32256 is from 512 bytes times 63 sectors as a hard disk is being reported 63 sectors per track. I am not sure if the 63 sector has the first sector, normally reserved as the boot sector inluded or not or even the counting starts from 0 instead of 1. dd differs from other copying operation in that it is the only one that clones the boot sector. Therefore it is impossible not to have the cloned disk bootable if the boot loader has been duplicated in exactly the same position as the original.
In any case the number is intended to represent the full track in a hard disk.
Declaring the bs as large as possible gains nothing as the operation still has to be chopped up bit by bit in the transfer. If bs size isn't declared then dd defaults to only 512 bytes per transfer and that will slow down the cloning process considerably as the CPU spends more time on housing keeping activities and data verification processes.
Both XP and the Vista conduct a check on the partition table and so it pays to have the XP at the first partition position with nothing else on the disk, as the two partition tables, of the source and the target disks, would be identical except for the free space..
Last edited by saikee; 06-26-2007 at 04:26 PM.
-
Arg! Still no luck!
I attempted it again, this time with a different 80 gig source hard drive and a freshly shreded destination hard drive of 40 gigs (attempting the same experiment, just with a different source drive).
I started off by shrinking the size of the 80 gig partition. After it finished shrinking with parted magic i restarted and booted into Linux. I used the cfdisk command and changed the units so that sectors would be the type of unit showing. The now recently shrunken partition read 31453.48MB (or 61432560 sectors according to cfdisk). After that I did a shutdown and removed the 80 gig drive, followed by inserting the 40 gig in order to make an identical (size, boot flag, primary) partition. When i did the cfdisk command to make the partition i noticed that a partition the size of 31453.48MB also yeild the resulting 61432560 sectors. I don't know if this is just by default, the conversions that the unix terminal does or what, but i took it as, for both hard drives, there is the same MB/sector ratio. After that I wrote to the partition table, rebooted as recommended, put back in the source hard drive and continued on with the DD command.
This time I decided to choose the bs value of 32256, just because of the fact that i read in the fdisk command that both of these drives had 63 sectors per track. Even though you said both numbers work. Just thought I'd try the other one just for the hell of it.
I entered the dd command and at the very end it spits back this error message:
"dd if=/dev/sda2 of=/dev/sdb1 bs=32256
dd: writing `/dev/sdb1': No space left on device
975120+0 records in
975119+0 records out
31453438464 bytes (31GB) copied, 706.556 seconds, 44.5MB/s"
I understand what it means, but I am curious as to why this happened? I had, according to cfdisk the exact same number of sectors (and MB) for both partitions. ALSO I figure, there was only 6 gigs of the 30+ gig partition actually taken up by data, and I assumed that since the DD is an exact bit for bit copy, that it probably copies it over in a linear-type way rather than random. So i figured whatever portion of a sector of information or whatever I'm missing it was probably nothing and should be fine. Right?
Well I shut down the computer, take out the source drive leaving only the destination drive in and put in a windows xp pro sp2 disk in. I boot up the computer from the CD so I can run the fixmbr command. It asks me if I really want to and i say yes. It reboots and attempts to boot with the new hard drive. Same exact problem as before. Windows has no idea what to do. Blank black screen with a flashing cursor (the kind of cursor for typing). I tried rebooting with no luck. Booted into Linux and got the same error message when trying to mount the drive:
"Could not mount device.
This reported error was:
Failed to read last sectory (78027696): Invalid argument
Perhaps the volume is a RAID/LDM but it wasn't setup yet, or the
wrong device was used, or the partition table is incorrect.
Failed to startup volume: Invalid argument
Failed to mount "/dev/sda1": Invalid argument
The device "/dev/sda1" doesn't have a valid NTFS.
Maybe you selected the wrong device? Or the whole disk instead of a partition? Or the other way around?"
Any ideas? I've about had it with this! ahah, but I really appreciate all your previous help, and if you do have any suggestions, I'd love to finally get this thing working!
Last edited by jonathansmith68; 06-28-2007 at 04:55 PM.
-
You need to have the same partition number for the source and target.
When a XP is installed it has a record of which controller, which partition and which disk. If you look at its boot.ini (hidden) typically it will have information like
Code:
timeout=30
default=multi(0)disk(0)rdisk(0)partition(1)\WINDOWS
[operating systems]
multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Microsoft Windows XP Professional" /fastdetect /NoExecute=OptIn
If you have the two partitions matched exactly in size then the number of records must correspond to each other.
The No. of records times the bs size should exactly equal to your partition size. If it doesn't it could only be the last record is not a complete unit of bs.
If you are cloning sda2 I recommend you clone sda1 to sdb1 first using identical size. The idea is to make sure the partition table matched up as well.
A sure fire method is to clone the whole disk but you need to have a bigger target.
Post the output of "fdisk -l" if you still have a problem. Using smaller target disk may give trouble if its geometry is different to the bigger source disk (different number of heads and sectors). 80Gb disks are likely to run at ATA133 with standardised 255 heads and 63 sectors whereas 40Gb disks may hang on to ATA66 or ATA100 which can have different geometries.
Last edited by saikee; 06-28-2007 at 08:13 PM.
-
You are right, I feel very stupid all of the sudden. It would make sense that both partition tables would have to be exactly the same in order for windows to not be confused. It totally makes sense to me now.
Now I have two problems before I try this again. The first partition is one of those stupid Dell Recovery Partitions, not by choice, but the image that we RIS here around the office had a dell recovery partition on it. So in order for me to keep the partition table the same I'd have to create this Dell Utility partition as well. When doing a fdisk -l command, it spits back saying that the System Partition type: "Dell Utility." I haven't checked to see if that type is in the cfdisk command but, I'm assuming it's not... Any idea's on what I could do with that? I'd assume it's just a FAT16 or FAT32 partition but, like I said fdisk says it's "Dell Utility."
Also here is what I get when I enter fdisk -l:
"Disk /dev/sda: 80.0 GB, 80000000000 bytes
255 heads, 63 sectors/track, 9726 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sda1 1 6 48163+ de Dell Utility
/dev/sda2 * 7 3830 30716280 7 HPFS/NTFS
Disk /dev/sdb: 40.0 GB, 40000000000 bytes
255 heads, 63 sectors/track, 4863 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/sdb doesn't contain a valid partition table"
So that makes me think they have the same geometry, since both number of heads and sectors are the same. than again i don't know really anything about the geometry of hard drives and maybe the different number of cylinders makes a difference too. but considering that's the only number i see different and there is clearly a difference in overall size, i'm under the impression that the geometry's of both hard drives are in fact the same. but if that's the case why did i get the "No Space Left on Device" error?! Please Also note that the reason it says there's no valid partition table as of now is because I just shreded the drive yesterday in order to attempt this again here in a little bit.
If you have any last suggestions, that'd be great. I think I'm going to try it again here in a few minutes by DD'ing both partitions, one at a time. Again my only problems I think I'll run into are the file system for the recovery partition and possibly getting the "No Space Left on Device" error again.
All of your help has been great, keep it coming if you have any more ideas. Thanks!
EDIT: Just noticed there is in fact a "Dell Utility" type partition you can choose from in the cfdisk command! Now all I need to know is the question about the possibly differences in geometries.
Last edited by jonathansmith68; 06-29-2007 at 11:49 AM.
-
I think your disks look OK to me and the running out of space is more to do with the way you cloned it.
dd is good in that it has no problem of whatever the content in the hard disk disk. It only copies the binary bits of "1" and "0". The important points are
(a) The partition type number should be the same. You can rectify this after the disk has been cloned but the best way is to have it written in the partition first as it might affect the way the partition being mounted for information un-related to dd. This is why it is a good practice to reboot every time the partition table has been modified, especially you are dealing with non-Linux partitions inside Linux.
(b) DD only cares about where it should start and where it must stop. Take your sda2 for example
Code:
/dev/sda2 * 7 3830 30716280 7 HPFS/NTFS
it will be cloned successfully if dd starts at the 7th cylinder and finishes at the 3830th cylinder. According to your information in Post #43 the sdb disk should have no problem of taking sda1 and sda2 because it has 4864 cylinders. Partitions are always in units of a whole cylinder. By telling dd the input device is sda2 dd will be forced to start from the 7th and finish at the 3830th cylinder. Bearing this in mind when you specify the output device.
(c) The partition type number, which is 7 for sda2 for ntfs and de for sda1 for dell utility, and the booting flag "*" are parts of the partition table which is not copied, unless the entire disk is cloned. Therefore you have to make sure they are written by the partitioning tool, like cfdisk or fdisk (Linux version).
-
I am almost completely stumped. I decided to try a different method. I followed the link that you had on the top of your first post ( this thread ). In post 14 of page 1 it reads:
"...If the source drive contains a partition which will make the smaller target drive think it has more cylinders that it really does, this will interfere with making a proper target image. This why, unless the target drive is bigger than the source, I do not copy the entire drive. But, if you wish to partition the target first, and skip the MBR on both drives, you can still have a smaller target than source drive. Here is the command to skip the MBR, thereby keeping intact different disk geometries.
dd if=/dev/sda of=/dev/sdb seek=1 skip=1 conv=notrunc,noerror.
When dd gets the MAS, or maximum addressable sector on the target, it will stop. Whatever there was room for will be there.
It should further be noticed that dd is a bitstream duplicator. It copies bit for bit, regardless of file system, or anything else."
After reading this I had thought that I had just found the holy grail of for what I was looking for! I tried this command out, and forgot to leave the bs out, so it took a little bit longer, but I suffered. Anyways after it was all done with the DD i found it strange that the same "No Space Left on the Device" error came up at the end even though according to the post, no errors are supposed to pop up. I did notice the command on the post has a period at the end of "noerror." I just assumed it was the end of the line so he put a period there, but was I supposed to add this into the command?
I am interested if anyone has used a command like this and can confirm or deny if it's correct because after the DD finished and I got the error, I turned off the system, took out the source drive, and tried booting up into windows. Same problem, except this time it knew instantly that there was nothing on the drive and attempted to pXe boot. I found this rather strange and booted into knoppix. When trying to mount it, it gave me an error saying something along the lines of: "cannot mount hard drive, there is no filesystem or the filesystem is not recognized." Going back into cfdisk showed that the drive was 39 gigs of unpartitioned space!?
After failing yet again, I re-read his post. It says "...But, if you wish to partition the target first..." I'm confused as to what this means. I thought if you were DDing the whole hard drive, making an exact copy that all the partitions and everything transfered over...? Should I maybe try it again and create both the Dell Utility and NTFS partition and then try this method again?
Posting Permissions
- You may not post new threads
- You may not post replies
- You may not post attachments
- You may not edit your posts
-
Forum Rules
|
|