Skip to main content

Request access to the Social Science Gateway

To request an account on the Social Science Gateway (SSG), please click here.

Request access to the Synthetic Data Server

Account request procedures for the Synthetic Data Server (SDS) are described here.

We support:

The International Year of Statistics (Statistics2013) APDU Logo foas-logo-small (2)

Step 6: Getting your data onto the XSEDE compute resource

Once your data prep stage is done, you may want to transfer your data to your preferred XSEDE resource. Compute node 2 (compute-0-2) is directly connected to the XSEDE backbone. You can transfer data out using  gsiscp,  scp from OpenSSH-HPN (installed under /opt/openssh-hpn/bin), or use GridFTP  to initiate a transfer from anywhere. A final alternative would be to use Globus Online (separate but free registration required). The examples below show transfers to blacklight.psc.teragrid.org, but should apply in similar fashion to any other XSEDE resource.

Using OpenSSH-HPN

The SCP provided by the OpenSSH-HPN project is performance optimized. You can find it on the head-node and compute-0-2 under /opt/openssh-hpn/bin. It works otherwise exactly like regular OpenSSH/SCP.
LOC=/opt/openssh-hpn/bin
$ $LOC/scp  largefile $USER@blacklight.psc.teragrid.org:
largefile                          100%  765MB   2.2MB/s   1.6MB/s   05:42
Your times will vary, depending on network traffic: we’ve observed speeds between 1.6 and 40MB/s.

Globus Online

Globus Online is a (primarily) web-based interface to GridFTP that has been designed to be much more user-friendly. The basic principle is the same as for GridFTP: you initiate transfers between points X and S (XSEDE and SSG) from a third location (for instance, your browser), at high speeds. In addition, Globus Online can also be used to transfer data to and from your workstation or laptop. The quickest way is to follow the Globus Online User Guide for XSEDE users at https://www.globusonline.org/xsede/userguide/. The SSG endpoints start with ‘ssg#’, and there are two:
  • ssg#scratch-0-2 – corresponds to compute-0-2:/scratch (i.e., the scratch space on compute node 2)
  • ssg#home – your home directory, as well as the project space (/home/ssgprojects), allowing for most transfers at the highest speed.
When prompted for SSG credentials, simply enter your SSG login and password. For XSEDE credentials, use the credentials for the resources that you are transferring data to and from (you can look them up on your XSEDE User Portal).

Using GridFTP

GridFTP is part of the Globus Toolkit. Setup is a bit more involved. You use  ’globus-url-copy’ if you are using automated scripts (see https://www.teragrid.org/web/user-support/transfer_examples for examples), but for one-off transfers, the use of Globus Online is easier.

CAC GridFTP server url: gsiftp://gridftp1.cac.cornell.edu/$PATH where possible paths are

  • PATH=/mnt/ssg/scratch-0-2
  • PATH=/mnt/ssg/home/$USER

Preparing the transfer

  • [OPTIONAL] You can use Globus tools from your workstation as well as the headnode. If using your workstation, you will need to install  Globus Toolkit on your workstation, and available in your session (we tested with Globus TK 5): [http://grid.ncsa.illinois.edu/myproxy/tgsso.html#install] 
    GLOBUS_LOCATION=$HOME/globus
export GLOBUS_LOCATION
. $GLOBUS_LOCATION/etc/globus-user-env.sh
  • obtain your X509 certificate from the TG resources [http://grid.ncsa.illinois.edu/myproxy/tgsso.html#logon]. Note that (username) is your XSEDE/TG Portal username (not necessarily identical to the one you use on XSEDE resources), and the pass phrase is the password you use on the Portal (see above linkfor exceptions) 
    myproxy-logon -T -l (username) -s myproxy.teragrid.org

    yielding

    Enter MyProxy pass phrase:
 A credential has been received for user username in /tmp/x509up_u25555.
 Trust roots have been installed in /home/username/.globus/certificates/.

    Make a note of where the file is stored (here, in /tmp/x509up_u2555), you will need it in the next step.

  • (if running this remotely) transfer your X509 certificate to your local PC (or use the browser on the SSG headnode)
  • register your certificate to the CAC GridFTP server resources at https://www.cac.cornell.edu/services/myacct.aspx. You will need to enter (i) your SSG login (ii) your SSG password (iii) the location of the certificate received above.
  • manually download a custom signatory file from the CAC proxy server at https://myproxy.cac.cornell.edu/CA/ to $HOME/.globus/certificates. Note that strictly speaking, this should be done automatically by ‘myproxy-logon’, but whether it is an incompatibility between GlobusTK 5 and the version of Globus running on CAC’s servers is unknown.
  • wait 2 minutes
  • then finally you can use the globus-url-copy command.

Example transfer

globus-url-copy -stripe -tcp-bs 11M -vb
gsiftp://gridftp1.cac.cornell.edu///mnt/ssg/home/$USER/out.txt
gsiftp://gridftp.blacklight.psc.teragrid.org/~/out.txt