Dealing with Large Data Transfers¶
If you have to transfer a large amount of data (in the order of several tens of GBs), or a large amount of files (several thousands), to or from CERNBox, some precautions are in order, as interactive tools such as OS native file browsers or the CERNBox Web UI may not be the most suitable options.
Web Transfers¶
If you need to serve large files via web, whether anonymously or not, you should use EOS Web Sites. Yet, we recommend chunking very large files to help users download them in the absence of transfer recovery mechanisms. Please note that in all cases the limit of 50 GB per file applies.
If you need to upload large files to EOS and are member of an Experiment, you are recommended to use HTTP Third-Party Copy (TPC) via the File Transfer Service.
Otherwise, if you need to use the CERNBox Web UI, data served through it is limited to 20 minutes of transfer time for each single transfer. Additionally, uploads are limited to 8 GB. If you and/or your recipient(s) are hit by this limit, we recommend to switch to other protocols as suggested below.
Note that if your recipient does not hold a CERN account, and you can only use CERNBox Public links, you have to chunk large files such that you stay below the limit, which also helps in case a transfer needs to be redone.
Other Data Transfers¶
The most efficient way to transfer data to/from CERNBox is to directly access the underlying storage. In case of EOS, you are recommended to share the relevant folder(s) with the target account(s), as opposed to "Share publicly", such that it can be accessed without passing through the CERNBox Web UI.
Depending on your OS, your options include:
- If you know how to use XRootD on a Linux-based systems, this is the best option as it offers checksum verification as well.
- If you have FUSE access (on Linux or MacOS), then a recommended tool is
rsync, in particular with the following options:
- If you use Windows Network sharing, known as CIFS or SMB on a Windows-based system, then a recommended tool is
robocopy, or Total Commander for a GUI-based file manager. We recommend to invoke therobocopycommand on a PowerShell with the following options: