Experience While Copying a 42GB file into a Seagate USB Hardrive in Ubuntu Linux…

Posted on February 26, 2008 by fawadnazir

First of all i will split the file into 10GB pieces. The file name is: enwiki-20070908-stub-meta-history.xml

fawad@crete:~/wiki$ split -b 4000m enwiki-20070908-stub-meta-history.xml

Now i will copy a piece by piece into my seagate external hardrive. I am copying this file from a remote server to my machine

root@fawad-laptop:/home/fawad# scp fawad@crete.ex.nii.ac.jp:/home/fawad/wiki/enwiki-20070908-stub-meta-history.xml /media/usb0/niidata/split/
fawad@crete.ex.nii.ac.jp’s password:
enwiki-20070908-stub-meta-history.xml 10% 4091MB 10.5MB/s 57:03 ETAFile size limit exceeded (core dumped)
This means only files less than 4GB are supported. So i again spilled the files to 4GB file each.

Another important thing to note is that i had to use root login to copy file to my external seagate USB hardrive.

Now once i have splited the files into 4GB chunks. Now use the following command to copy all the files from this folder to one of the folder in the seagate harddrive.

root@fawad-laptop:/home/fawad# scp -r fawad@crete.ex.nii.ac.jp:/home/fawad/wiki/wikidata/ /media/usb0/niidata/wikifawad@crete.ex.nii.ac.jp’s password:
.nfs0000000000d5005100000001                                                               100%  435MB  10.6MB/s   00:41
xad                                                                                         27% 1101MB   9.5MB/s   05:05 ETARead from remote host crete.ex.nii.ac.jp: Connection reset by peer
xad                                                                                        100% 4000MB   9.8MB/s   06:49
xag                                                                                        100% 1233MB   9.8MB/s   02:06
xac                                                                                         38% 1522MB  10.6MB/s   03:53 ETAh
xac                                                                                         43% 1734MB  11.1MB/s   03:25 ETA
xac                                                                                        100% 4000MB  10.1MB/s   06:38
xaf                                                                                        100% 4000MB  10.2MB/s   06:32
xae                                                                                        100% 4000MB  10.5MB/s   06:22
xaa                                                                                        100% 4000MB   9.9MB/s   06:46
xab                                                                                         39% 1573MB   9.9MB/s   04:05 ETARead from remote host crete.ex.nii.ac.jp: Connection reset by peer
xab                                                                                        100% 4000MB   9.6MB/s   06:57

Filed under: Linux | Tagged: , , ,

« »

2 Responses

  1. Babil, on October 30th, 2008 at 2:45 am Said:

    Hi Fawad, that 4GB limitation is because – you most likely had a FAT32 file-system on your usb hard drive. If you would format the file-system to NTFS or linux EXT2/EXT3, you wouldn’t have this problem. They both support file sizes in tera-bytes. EXT4 would support as much as 100 tera-bytes in newer linux kernels.

  2. fawadnazir, on October 30th, 2008 at 10:07 am Said:

    Yes you are right babil. I had to access to that machine so i had to work around the problem. Once i did work around, i though it good to share :) .

Leave a Reply