Cloud File Storage¶

Prerequisites
File names
Python
Command line

This tutorial covers downloading and working with files from our Cloud Storage buckets via two methods: the pgb-utils Python package, and the gsutil CLI.

For more information, see:

Prerequisites¶

Complete the Initial Setup. Be sure to:
- set your environment variables
- enable the Cloud Storage API
- install the google-cloud-bigquery package if you want to use Python
- install the pgb-utils package if you want to plot the data using Python
- install the CLI if you want to use the command line

File names¶

We store the alert packets as Avro files named as “{objectId}.{candid}.{ztf_topic}.avro”

Python¶

Setup¶

Imports

import fastavro
from google.cloud import storage
from pathlib import Path
import pgb_utils as pgb

Name some things

# fill in the path to the local directory to which you want to download files
local_dir = ''

my_project_id = os.getenv('GOOGLE_CLOUD_PROJECT')
pgb_project_id = 'ardent-cycling-243415'

Download files¶

Download alerts for a given objectId

objectId = 'ZTF17aaackje'
bucket_name = f'{pgb_project_id}-ztf-alert_avros'

# Create a client and request a list of files
storage_client = storage.Client(my_project_id)
bucket = storage_client.get_bucket(bucket_name)
blobs = bucket.list_blobs(prefix=objectId)

# download the files
for blob in blobs:
    local_path = f'{local_dir}/{blob.name}'
    blob.download_to_filename(local_path)
    print(f'Downloaded {local_path}')

Plot cutouts and lightcurves¶

The functions in this section were adapted from https://github.com/ZwickyTransientFacility/ztf-avro-alert/blob/master/notebooks/Filtering_alerts.ipynb.

Open a file (see the previous section to download files)

paths = Path(local_dir).glob('*.avro')
for path in paths:
    with open(path, 'rb') as fin:
        alert_list = [r for r in fastavro.reader(fin)]
    break
alert_dict = alert_list[0]  # extract the single alert packet

print(alert_dict.keys())

Plot cutouts

pgb.figures.plot_cutouts(alert_dict)

Cast to a dataframe and plot lightcurves

lc_df = pgb.utils.alert_dict_to_dataframe(alert_dict)
pgb.figures.plot_lightcurve(lc_df)

Plot everything together

pgb.figures.plot_lightcurve_cutouts(alert_dict)