Skip to main content
Stanford
Research Computing Nero GCP

Data Storage and Transfer Options

Two Primary Data Storage Options

1. BigQuery a Serverless SQL Data Warehouse

The primary path for receiving STARR data is via BigQuery.
Once you have a Nero project, you can then request for the STARR data extract to be delivered to your Nero project.

Cost: ~$20/month/TB

2. Cloud Storage via Buckets

Cloud storage is for object storage for files of any size. Data stored in buckets are like the folders on your laptop, you can have as many buckets inside of your project as you need, and fine-tune each bucket’s access settings to keep your data secure.

The cost of storing your buckets of data is related to the size of the data and to how often you access the data.

Cost: ~$25/month/TB for storage and $5/TB traversed for query

By default, buckets are set to go into deeper storage the longer they are untouched. This is ideal for most use cases, but needs to be considered as part of a larger data strategy.

Accessing your Stored Data

Most people will interact with their stored data via the GCP Console or by using the Google Cloud CLI tools, such as gsutil or gcloud commands.

List data into your buckets

gsutil ls -l gs://<bucket>

List BigQuery datasets in a project

bq ls <project-name>:

Show a table’s schema and metadata

bq ls <project-name>:<table>

Transferring Data into Nero

1. Using Google Cloud CLI

Moving data into your buckets

gsutil -m cp -r <local-path> gs://<your-bucket>/
# or the newer equivalent:
gcloud storage cp -r <local-path> gs://<your-bucket>/

Creating a dataset in BigQuery

# From a local file, auto-detecting the schema
bq load --autodetect --source_format=CSV \
  my_dataset.new_table \
  ./data.csv

2. Using sftp from your Nero Instance

If the data you are receiving is less than 70GB and you have room on your /home/jupyter partition, the following method can work, for larger data transfers, please reach out to us. sftp the data from the external party to your Nero instance

  1. Connect to your Nero GCP VM instance
  2. Try to sftp from there to the remote server 
    sftp <username>@<sftp_remote_hostname>
  3. cd to the path on the remote server where the data is; ls to list content to confirm
  4. Download the file with get for a single file; or get -r to copy the directory and everything within

From your Nero instance, upload that data to a Nero Cloud bucket

  1. gcloud auth login - login and authorize using your Stanford credentials (using local web-browser and copy the verification code back) can check with gcloud auth list to make sure your Stanford account is active
  2. gsutil ls gs://<bucketName> to confirm you can access the bucket
  3. gsutil -m cp -r /path/to/downloaded/files gs://<bucketName>
  4. gsutil ls -r gs://<bucketName> to confirm or check via the web

Deleting Data

By default, manually deleting data performs a soft delete; the data no longer appears in your bucket but it can be retrieved or restored within a pre-set period.

The default setting is 7 days, meaning your data is truly deleted after a week. Users with Storage Admin privileges can set a soft delete policy each bucket using the GCP console. The maximum soft delete delay is 90 days.

On This Page

Support