![]() Without diagnostic information like CPU % utilization or load average, memory/swap usage, packets per second, etc., I can only shoot from the hip. Hard for me to believe that it's the first time in ten years I've had to do something like this I guess being fully in AWS for all that time has me spoiled. Anyone with more experience at this task than I would be most welcome. These systems were not created with LVMs, or any other filesystem level snapshotting, and it doesn't seem time effective to make chunked tarballs of the directories, since it would just be copying/compressing data into multiple files before shipping them across the wire. Is there a tool out there that transparently chunks these large files and ships them in pieces, in parallel? I can see when I start hitting ~100MBytes/sec when the network starts being the bottle neck, but when it's just the large files left, the transfer rates slow to a measly 25MBytes/sec. rsync takes about 12 hours, rclone about 6.5 hours for 1 TB. I have dozens of customers, with 500GB files each (the largest files 500GB, total of 1.3 TB) in one cloud provider that I need to shuttle to another. Windows and Mac users can install the graphical desktop client from DataExpedition as well if they wish so:ĭownload Expedat from here, installation instruction from here, and product documentation from there.After two days of research with nothing knock-out promising, I'm finally asking for help. For the documentation see the man page: man rsyncĪs above, just swapping the source and destination: cp /nas/FAC/////file /path/to/dest cp -r /nas/FAC/////folder /path/to/dest Accessing DCSR NAS from CHUV network To copy a folder to the new NAS: cp -r /path/to/folder /nas/FAC////įor more complex operations, consider using rsync. To copy a file to the new NAS: cp /path/to/file /nas/FAC//// The folder hierarchy is: /nas/FAC//// Cluster -> NAS Step 4: navigate through the directories to find your working directory and push/fetch files as you see fitįrom the login nodes the NAS is available under /nas. Step 2: enter the NAS url ( smb:///RECHERCHE), you also may add this server as a favourite by clicking on symbol Step 1: open the Finder, in Go menu, select "Connect to Server" # When finished, don't forget to umount it $ sudo mount -t cifs -o username=unil_username,domain=ad,rw,iocharset=utf8,dir_mode=0700,file_mode=0700,uid=$(id -u) ///RECHERCHE ~/nasdcsr # In the following, ~/nas is chosen as a mountpoint, but it could be elsewhere Don't forget to adapt the username= field in the sudo mount command! # First install the packages So here is a way to mount the NAS using command line interface. It is possible to use a GUI but there are as much solutions as desktop managers. Step 4: click on "This PC" and then on the new network drive username: ad\unil_username (for example ad\ulambda).Step 3: enter your Unil credentials and clikc "OK" Step 2: enter the address of the Samba server and click to finish Step 1: right click on Network in File explorer and choose "Map network drive…" Note: If you work from home, you first need to connect to Unil's VPN using Pulse Secure (see VPN instructions). Step 6: navigate through folders (1 is for local files, 2 is for cluster files) and copy the files in one way or the other Step 2: enter the configuration elements (SCP file protocol, hostname, and credentials). Scp file.txt copy the entire directory "dirA" to your home directory: To copy file "file.txt" to your home directory: Linux and Mac users simply have to open a terminal.įrom laptop to cluster: $ scp /path/to/fileToCopy cluster to laptop: $ scp /path/to/dest To copy data between your laptop and a cluster you will have to use scp (secure copy). NAS: the Network Attached Storage from UNIL.Cluster: one of the clusters managed by the DCSR (wally or axiom).Laptop: the personnal computer you use for your daily work (usually a laptop).The three main locations are hereafter referred to as: laptop, cluster, and NAS. ![]() This page presents the way to move data between the different locations used for either processing or storing data.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |