Introduction

The SHARK SLURM cluster is maintained and hosted by the LUMC. It is currently the cluster that most FSW users are working on.

Resources

Data Management

• All FSW users share a total of 40TB of spaces of the cluster, 80 compute nodes and have 10GB of personal space on their home directories.
• Once the shared space is full,  the output of any jobs will not be saved (and will be discarded instead). So always keep an eye on the current status of the shared folder /exports/fsw/  and the expected output size of your jobs.
• Never have your important data (e.g. raw) or scripts only on the cluster! Remember that none of the data on the exports folder or personal folder is back-upped. That means, if the data gets corrupted, is overwritten, or there is a failure in the drives, the data may be lost permanently.
• Files can be transferred for long-term & secure storage to FSW using sftp (see here).

F.A.Q.

Data Transfer FSW

Choose a Method

There are various ways to transfer data between SHARK and the FSW network. Here we focus on two main options: syncing files with rsync (using MobaxTerm) and copying files with GUI (using WinSCP or FileZilla).

• Syncing files using MobaxTerm (Recommended)
  • This is the best method to use for syncing large datasets with many files from Shark to FSW (or the other way around). It works from your windows workstation on the FSW or by using the Citrix Remote Desktop from any other machine/location.
• Copying files using WinSCP.
  • This method is best for quickly copying smaller datasets from Shark to FSW (or the other way around), and is a bit more accessible if you are not comfortable with using a terminal. It works from your windows workstation on the FSW or by using the Citrix Remote Desktop from any other machine/location.

Find a suitable destination

• Your personal P-drive has limited storage capacity .
• The J-drive on the other hand, offers more storage space, usually in /Public/ResearchData/  There you will find folder for the various institutions and sections. If you do not have access (including write permissions) for a folder there you can request one via the ISSC (Workgroup storage, 'Research data' bulk storage). To get access to a folder owned by a colleague he/she needs to submit the request. Also keep in mind, J-drive storage is relatively expensive at ~450 euro's per year per TB. These costs are billed towards your section. For very large data-sets and long-term archiving, perhaps consider external and/or Open Science oriented options (e.g. DANS) .  
• OneDrive / Google Drive are not suitable options, as these are not secure.

Syncing with MobaxTerm

[Optional: Start Citrix]

If you are not working on an FSW windows machine (i.e. at home, on your laptop or Mac), first start-up the Citrix Desktop:

• Log in to the Citrix Remote: https://remote.campus.leidenuniv.nl/ You can use the light (browser) or full version, it does not really matter.
• Go to the "Desktops" tab
• Open a new Basic Desktop
  

[Optional: Start SURKO]

SURKO's are virtual Windows machines that can be reserved through this ISSC for 4-8 weeks, and remain always on for that period. Normally they are used to run intensive computations on Windows specific software (e.g. Brain Vision Analyzer) but for our purposes they can be used for data-transfers that take a long time and should not be interrupted. If you wish to use a SURKO for data-transfer, please contact labsupport@fsw.leidenuniv.nl

• The full ISSC manual for connecting tot the SURKO can be found here.
• Open an RDP application. On windows this can be done by pressing the Windows key, and searching from "remote desktop connection".
• Click on "Show Options" to expand the menu.
  
• In computer fill in the computer nameyou received from the ISSC, e.g.: WV0065241.vuw.leidenuniv.nl
• For user name, enter your ULCN name preceded by 'vuw\', e.g.: vuw\olferskjf
• Go to the Advanced Tab and click on settings
  
• Select "Use these RD Gateway server settings" and fill in "rds.campus.leidenuniv.nl" and press OK
  
• Go back to to the General Tab, select "Save As" and give is a name you will remember later. This will create a shortcut that you can use later on to quickly reconnect to the SURKO.
• Finally, click on "Connect" to start the SURKO.

Start MobaxTerm

From here the steps are the same for FSW machines and the Citrix Remote:

• If you don't have MobaxTerm yet, download the portable edition version of MobaxTerm (https://mobaxterm.mobatek.net/download-home-edition.html) and unzip it, e.g. to a folder on the desktop
• Start the MobaxTerm client from the folder by double-clicking the .exe file in the extracted folder.
• If it is the first time you started MobaxTerm, go to Settings > Configuration >General.
  • For Persistant home directory, enter:  _MobaFolder_
  • For Persistant root, enter: _MobaFolder_\slash
  • Press OK at the bottom and restart MobaxTerm if asked. You can ignore any warnings about network drive speed
    .
• Start a new local session in MobaxTerm, using the "Start local terminal" button on the home screen, or go the "Session" in the top left then select "Shell" and press OK.

Preparing Rsync

You have now opened a local shell terminal. Most Linux terminal commands you know will work here in the same way. Conveniently, MobaxTerm also has a built in Linux file syncing tool called Rsync. To make this work with Shark we need to provide an Rsync command

• Paste the following command into notepad or Word (or any text editor of your choice). And fill in the yellow highlighted parts. Note that this is all one line!

• This command takes all the files and subfolders from the source directory on shark and syncs them to target directory on the J drive. Files that are already present on the J drive and have the same filesize and modification time will not be overwritten. This means that if your transfer gets interrupted (for instance because Citrix is closed, you lose network connection etc), re-running this command should pick up the transfer right where you left off (without having to start from the beginning).
• Replace all the yellow highlighted [..] parts (also remove the [ ] symbols!)
  • [username]: should be your username on the cluster (e.g. for me it is kjfolfers)
  • [source folder to be synced from]: the path to the main folder you want to sync the files from. Rsync with the -av parameter works recursively, so subfolder will also be synced. (e.g for me that could be kjfolfers/transfertest). In Rsync there is a difference between ending the source directory with or without a slash. If you omit the slash, the source directory folder itself (in this case transfertest) will also be created in the destination folder. If you include the slash, the contents of the folder will be put directly into the destination folder.
  • [folder to be synced to]: the path to the folder on the J drive where you want to sync to. You need to have write access to this folder. You could check that by opening file explorer navigating to the folder and making a new text file there. (e.g. for me such a folder could be Psychologie Cognitieve/olferskjf/sharktest/). Important: the folder paths used in the command has forward slashes "/" for folder separation, while windows natively (e.g. in file explorer) used back-slashes "\", this means you can't just copy the file path from explorer. Additionally, for older versions of rsync/MobaXterm if you have spaces in your folder names, you need to prepend them with a backslash. For example if your file-path in explorer looks like :
    "drives\ResearchData\FSW\Psychologie\My test folder" 
    the corrected version would be:
    "drives/ResearchData/FSW/Psychologie/My\ test\ folder/"
    If you receive errors regarding file paths, try including or excluding the backslash before spaces.
• As an example, the full command could look like this (note the command should be one line, without enters):

• Rsync can be run with many configurations. The one used above is pretty much the default setting for archiving (that is copying all folders and files). However, in some cases you may want to use different parameters to better suit your needs. For instance symbolic and hard-linked files and folders may need special treatment that the regular -av parameter does not give. Please consult the help pages for rsync and/or contact SOLO or cluster admins for help if needed.

Running Rsync

The first time you use rsync, I highly recommend starting with a small test source folder with some files to see if everything works as expected!

• [OPTIONAL] By including the -n parameter (e.g. rsync -av -n -e ...) to the command, you can do a "dry run" this will show you a list of which files would be synced, without actually doing anything yet.
  
• [OPTIONAL] By including the -P parameter, you will get information about the transfer speeds and the file progress (this will make the output in the terminal a bit more complex though).
• Copy and paste the command into the MobaxTerm console (use right mouseclick or the middleclick) and run it.
• The first time you run this command, you will be asked to fill in your password for the Shark cluster twice.
• Mobaxterm will ask you if you wish to save your password for later, if you do so, you will not have to fill it in the next time
• If you use the rsync command as specified in the instruction, a log file (rsync_log.txt) will also be written in the specified target folder. This is especially helpful in case of very long transfers, when you may not notice the sync has ended (or the terminal / SURKO has already been closed). You can open the log file in any text editor to inspect the progress.

Optional: Keep Alive

If you are experiencing trouble with the computer or Citrix Desktop closing due to a timeout. You could try the following:

• After starting the rsync command, open an additional local session in mobaxterm
• Run the following command in the terminal:

• The above command runs every minute to store a timestamp in a text file. It can help to keep the Citrix desktop alive. The transfer should even continue if the Citrix desktop is closed (if prompted confirm that you want the session to remove active). Virtual desktops may be still be automatically shut-down after some time, but running the rsync command again should pick up the transfer where it was right before shutdown.

Copying with WinScp

While this method may be a bit more user-friendly, given the graphical user interface, keep in mind that transfers with WinSCP will stop once you shut down your local FSW computer (or when using Citrix, after a certain period of time). After which, you will manually need to restart the transfer again, and it might be hard to keep track of which folders and files have already been copied succesfully.

• Log in on your FSW computer (physically or via Remote Desktop Protocol or via the Citrix Remote environment).
• Make sure WinSCP is installed, if not, install it from the Software Center or download and unzip the portable (non-installer) version here . If you are using the Citrix environment, you can start the WinSCP application directly (you do not need the full desktop view).
• Start WinSCP
• Select "New Site"
• Fill in the session information:
  • File Protocol: SFTP
  • Host Name: res-hpc-lo01.researchlumc.nl  (or res-hpc-lo02.researchlumc.nl) Note: if neither work, try the ip addres: 145.88.76.243 or 145.88.76.217
  • Port number: 22
  • User name / Password: you SHARK account details.

• If you want you can click "save" the save these settings for future sessions.
• Then click "Login"

• When asked to approve a key or fingerprint, press OK.
• You will now see your FSW folders and files in the left window and your SHARK folders and files in the right window.
• The reach the various FSW folder (e.g. the J or P Drive), use the dropdown menu (or for SHARK, the menu on the right side):

• To transfer files you can now: drag & drop them from the one of the windows to the other, or, select them, right-mouse click and press download, they will then be copied to the currently open folder in the opposite window.

Data Migration ALICE

For migrating your project from SHARK to ALICE, please see the manual here.