Hot File Based Cloning of Linux Servers Over the Network

I'd like to expand on "rjcarlet"'s blog "Cloning/Copying Systems to Reduce Build Times for a *NIX System.  and touch on a method I found to work well in Intel IT.

Network based cloning is not new, but I could not find any free solutions out there that allowed you to clone a server while its still up and running.  I wanted to share a proven method I've been using for years.  Once you have a server configured and built the way you like it and need it replicated you can perform the following replication all while the source server is still running!  No more worrying about cloning from an outdated system.  Your system can have all the latest security patches, configurations and applications.

I found that cloning for the purpose of replication has its benefits and drawbacks.  For one, if you're cloning, you can be sure that all the servers will be 100% identical and will behave the same.  However the flip side is that if you have a defect, you're carrying it over.  I have found that in this case the benefits outweigh the drawbacks.

Things you'll need:

1. A OpenSuse boot cd/USB disk for your destination server.
2. An NFS or CIFS share to store build scripts on.
3. A source and destination server.

Considerations:

• If you're going to clone in a production environment, you may want to build over a dedicated network drop to avoid impacting your primary production LAN.
• This method works best for replicating to identical hardware but does work for migrating to different hardware.  You will need to download the new drivers from the manufacturer and then use the new system as your gold build.

Caveats:

In the interest of time, I'll only detail the high level steps.  If I get enough reader interest I'll go into more detail in another blog.

Steps:

1. If your going to be using this solution often, I found that injecting a base startup script (see example startup script attached) in the OpenSuse iso to configure networking (IP, Gateway..) and to mount a NFS share works the best.  This way you can run your core scripts from an NFS/CIFS share rather than have to burn a new boot cd/USB disk every time you have a script revision.  Simply update the scripts on the share and your done.
2. Configure the destinations server's logical drivers in its raid configuration and boot it off the OpenSuse iso.
3. Set the IP address for the server.
4. Dump the partition table using fdisk from the source server and import using fdisk on your destination server.  If you are migrating to different hardware you'll need to manually redo the partitioning using fdisk.
5. Format the new partitions and mount under /mnt, /mnt/usr, /mnt/var/ and so on.
6. Use rsync to copy the source server files over to the new /mnt partitions.  I found "rsync -Hiuvl " options to work the best.  Don't forget to exclude /proc, /dev, /var/run and any nfs directories you may have mounted on the source.
7. Once the rsync is done you'll need to setup the master boot record using grub.
8. Change the working directory  to /mnt/etc and modify the following: (Note: I've automated all of this so that I don't have to do any of this manually)
   1. /etc/HOSTNAME
   2. Modify the IP addresses /etc/sysconfig/network/ifcfg*
   3. Delete any udev records for the network cards under /etc/udev/rules.d
   4. Modify the etc/fstab if you are moving to different hardware to reflect the new partition device files .

That's basically it.

Additional components can be added to make the build process more streamline, but I found it actually makes it more complex when time comes to troubleshoot failures.  Example,  adding a TFTP boot server will allow you to circumvent the need for a boot iso.

Let me know what you think.