Skip to main content

Access to ECCO

The Economics Compute Cluster (ECCO) has migrated to the BioHPC environment, and accounts are not handled on this website anymore. See the BioHPC User Guide.

Latest Tweets

Upgrading Synthetic Data Server [DOWNTIME June 27-June 30, 2014]

Print Friendly, PDF & Email

All SDS users:

We are ready to migrate users to the new SDSx cluster, which greatly improves the computational ability offered to your analyses. There will be a downtime from Friday, June 27 to Monday, June 30, 2014.

(If you are a SDS user currently participating in the Advanced Workshop on SIPP Synthetic Beta please see special note at end).

The migration will offer access to significantly upgraded computing resources:

  • 3x 256GB of RAM instead of 1x 128GB
  • 96 cores available for compute jobs (see cluster description)
  • better job control

Migration timeline

  • new accounts are currently not being created on SDS anymore. All new accounts will be created on SDSx.
  • on Friday, June 27, 5PM, no further logins to SDS.
  • Friday, June 27, 5PM through Monday, June 30, 5PM, limited  downtime (primarily to sync remaining programs and data from SDS to SDSx).
  • on Tuesday, July 1, general availability of SDSx

Please let us know immediately if this is a problem for your research, and we will try to accomodate you.

Participants in the Advanced Workshop

you will already be using the SDSx during the workshop. Your programs and data, if any, were synchronized on Friday night (June 20, 2014), and should be all there this morning. If you notice anything missing, please let us know.

Changes

Few changes will be needed for your programs. If you have been using the paths listed on the website  for your programs, everything will run as usual.

However, launching software will change. The new cluster uses Torque/Maui job scheduler (on ROCKS6, for the cogniscenti), very similar to our other cluster ECCO and to the research clusters at Census (and many other universities). We have some custom tweaks that make life for many users easier.

Key changes:

  • Logon is now to sdsx.vrdc.cornell.edu (not sds...). Users can either change this directly in their existing NX configuration file (leaving everything else unchanged), or download a new file from the SDS setup page.
  • SAS, Stata​, R, RStudio launched from the GUI menu are limited to 2 hour runtimes. Only Stata/IC is available from the menu
  • I​nteractive terminal sessions allow you to launch any program from the command line, you can start an interactive session from the GUI menu or from the command line ('iqsub')
  • Interactive sessions are limited to 2 hours, but other versions of Stata and R can be launched. You will also need to take into account the 'module' command.
  • Job submission from the command line is done using 'q-commands': if previously you started SAS jobs as "stata do myprog.do" you should now use "qstata myprog.do", and equivalently for "stata-se" and "stata-mp". Note that launching programs using Xstata does not work, unless you are in an interactive session.
  • For longer running jobs, or resource requests that don't fit the standard 'q-commands', you will write your own qsub command. Examples are provided on the website. You will also need to take into account the 'module' command.
  • On SDSx, currently, there are few resource limits on non-interactive jobs: each job can request up to 256GB of RAM and 8CPUs. You can have up to two simultaneous jobs running, but can submit further jobs to the queue. We will be monitoring this for congestion, and may modify those limits appropriately. The most up-to-date queue limits can be queried using the 'qstat -q' command, and we regularly update the SDSx queue list.
Over the course of the next week, many of the SDS-related documentation on this site will be updated to reflect the new cluster.
We hope that this will improve the usability of the Synthetic Data Server. Please let us know if you have any concerns.

 

3 comments to Upgrading Synthetic Data Server [DOWNTIME June 27-June 30, 2014]

  • Lars Vilhuber

    Known issues

    • OpenOffice not yet available: it is coming
    • KDE not available: please open a ticket at sds-help@cac.cornell.edu if you are interested in having KDE available. GNOME has equivalent functionality.

    Important information

    • Home directories from the old SDS are available under $HOME/sds-old
    • All Stata and SAS datasets have been compressed using BZIP2. In order to be able to use them, you will need to decompress them. From the command line, use "bunzip2 (name of file)"; from the file manager, right-click and choose "Uncompress here". PLEASE CONSIDER DELETING UNNECESSARY FILES. The new system has quotas on home directories, and you may not be able to do any additional work if you surpass your quota.
  • Lars Vilhuber

    Current status Sunday June 29 2014

    • All project directories under /rdcprojects have been synchronized
    • Home directories are still compressing/synchronizing
  • Lars Vilhuber

    Added a note that logon is now to sdsx.vrdc.cornell.edu, not to sds.vrdc.cornell.edu.