View and Access Resources

A broker instance has many resources associated with it. See also:

The following sections contain links to the GCP Console where you can view resources, plus descriptions of what you can view/do there, and some useful shell and Python commands. (Dashboards first, then alphabetical.)

Dashboards

GCP Console

Go directly to the dashboard for a specific broker instance by replacing “<survey>” and “<testid>” with appropriate values in the following url:

https://console.cloud.google.com/monitoring/dashboards/builder/broker-instance-<survey>-<testid>

This has charts related to:

  • VM network and CPU usage

  • Pub/Sub streams: rates and nightly cumulative totals

  • Dataflow jobs: lag metrics and number of vCPUs in use

BigQuery

GCP Console

Expand the list under your project name, then expand a dataset, then click on a table name to view details and options.

See also:

Cloud Functions

GCP Console

Click on a function name. From here you can:

  • view live* monitoring charts

  • view and edit job details

  • view the source code

  • view the logs

  • test the function

See also:

Cloud Scheduler

GCP Console

From here you can - view job details and logs - create, edit, pause/resume, and delete jobs

See also:

The auto-schedulers of all broker instances share the following logs:

Cloud Storage buckets

GCP Console

Click on the name of a bucket to view files and options.

See also:

Compute Engine VMs

GCP Console

Click on the name of one of your VMs ({survey}-night-conductor-{testid} or {survey}-consumer-{testid}). From here you can:

  • start/stop the instance

  • access the logs

  • view and edit the metadata attributes

  • view and edit other configs

  • click a button to ssh into the instance

  • view performance stats and live* monitoring charts

Here are some useful shell commands:

General access:

vm_name=  # fill this in
zone=us-central1-a

# start it
gcloud compute instances start --zone="$zone" "$vm_name"
# stop it
gcloud compute instances stop --zone="$zone" "$vm_name"
# ssh in
gcloud compute ssh --zone="$zone" "$vm_name"

# set metadata attributes
ATTRIBUTE1=value1
ATTRIBUTE2=value2
gcloud compute instances add-metadata --zone="$zone" "$vm_name" \
      --metadata "ATTRIBUTE1=${ATTRIBUTE1},ATTRIBUTE2=${ATTRIBUTE2}"
# unset attributes
gcloud compute instances add-metadata --zone="$zone" "$vm_name" \
      --metadata "ATTRIBUTE1=,ATTRIBUTE2="

Example: Use night-conductor to start/end the night (see also Auto-scheduler)

survey=ztf
testid=mytest

#--- Start the broker
NIGHT=START
KAFKA_TOPIC=NONE  # leave consumer VM off; e.g., when using consumer simulator
# KAFKA_TOPIC=ztf_yyyymmdd_programid1  # replace with a current topic to ingest
# set metadata attributes and start night-conductor
instancename="${survey}-night-conductor-${testid}"
zone=us-central1-a
gcloud compute instances add-metadata "$instancename" --zone="$zone" \
        --metadata NIGHT="$NIGHT",KAFKA_TOPIC="$KAFKA_TOPIC"
gcloud compute instances start "$instancename" --zone "$zone"
# this triggers night conductor's startup script

#--- Stop the broker
NIGHT=END
# set metadata attributes and start night-conductor
instancename="${survey}-night-conductor-${testid}"
zone=us-central1-a
gcloud compute instances add-metadata "$instancename" --zone="$zone" \
      --metadata NIGHT="$NIGHT"
gcloud compute instances start "$instancename" --zone "$zone"
# this triggers night conductor's startup script

Example: Set night-conductor’s startup script

survey=ztf
testid=mytestid
nconductVM="${survey}-night-conductor-${testid}"
broker_bucket="${GOOGLE_CLOUD_PROJECT}-${survey}-broker_files-${testid}"
startupscript="gs://${broker_bucket}/night_conductor/vm_startup.sh"
# set the startup script
gcloud compute instances add-metadata "$nconductVM" --zone "$zone" \
        --metadata startup-script-url="$startupscript"
# unset the startup script
gcloud compute instances add-metadata "$nconductVM" --zone "$zone" \
        --metadata startup-script-url=""

Dataflow jobs

GCP Console

Click on a job name. From here you can:

  • view details about the job

  • stop/cancel/drain the job

  • view and interact with the graph that represents the pipeline PCollections and Transforms. Click on a node to view details about that step, including live throughput charts.

  • view a page of live* monitoring charts (click “JOB METRICS” tab at the top)

  • access the logs. Click “LOGS” at the top, you will see tabs for “JOB LOGS”, “WORKER LOGS”, and “DIAGNOSTICS”. Note that if you select a step in the graph you will only see logs related to that step (unselect the step to view logs for the full job). It’s easiest to view the logs if you open them in the Logs Viewer by clicking the icon.

Command-line access:

  • To start or update a job from the command line, see the README at broker/beam/README.md

  • Job IDs: To update or stop a Dataflow job from the command line, you would need to look up the job ID assigned by Dataflow at runtime. If the night conductor VM started the job, the job ID has been set as a metadata attribute (see Compute Engine VMs).

Logs Explorer

GCP Console

View/query all (most?) logs for the project.

Pub/Sub topics and subscriptions

GCP Console (topics) | GCP Console (subscriptions)

Click on a topic/subscription. From here you can:

  • view and edit topic/subscription details

  • view live* monitoring charts

* Live monitoring charts have some lag time.