Run the Broker

Use this document to run a broker instance manually (typically a Testing instance).

See also:

  • Workflow

  • ../components/auto-scheduler to schedule a (typically Production) instance to run automatically each night.

  • Night Conductor

  • View and Access Resources

Start the Broker

Note

This section contains instructions to manually start the consumer VM. Cloud Functions and Cloud Run are always on; they do not need to be started or stopped. If you are running a test and do not need the consumer VM, you can skip this section.

Fill in the broker instance keywords below with the same ones used to deploy the broker instance.

survey=
testid=

If you want the consumer to use non-default Kafka and/or Pub/Sub topics, set the appropriate TOPIC_FORCE variable in the VM’s metadata. The code below gives examples. These will be automatically unset when the VM shuts down.

KAFKA_TOPIC_FORCE=""                                # default, today's topic (UTC)
# KAFKA_TOPIC_FORCE="ztf_20220302_programid1"       # Kafka topic from Mar 2, 2022

PS_TOPIC_FORCE=""                                   # default alerts Pub/Sub topic
# PS_TOPIC_FORCE="my-alerts"                        # Pub/Sub topic named my-alerts

# Set the topics as metadata:
metadata="KAFKA_TOPIC_FORCE=${KAFKA_TOPIC_FORCE},PS_TOPIC_FORCE=${PS_TOPIC_FORCE}"
gcloud compute instances add-metadata "$vm_name" --metadata="${metadata}"

Start the consumer VM.

Note

Upon startup, the consumer will automatically try to connect to the survey and begin ingesting alerts. If you want to avoid this, set the startup script url to an empty string in the VM’s metadata. See View and Access Resources: Compute Engine VMs.

vm_name="${survey}-consumer-${testid}"
gcloud compute instances start "${vm_name}"

Stop the Broker

Note

This section contains instructions to manually stop the consumer VM. Cloud Functions and Cloud Run are always on; they do not need to be started or stopped.

gcloud compute instances stop "${vm_name}"

Options for Ingesting Alerts

You have three options to get alerts into the broker. Production instances typically use #1; testing instances typically use #3.

  1. Connect to a live stream. Obviously, this can only be done at night when there is a live stream to connect to. If there are no alerts in the topic, the consumer will poll repeatedly for available topics and begin ingesting when its assigned topic becomes active. Use the Kafka Topic Syntax with today’s date (UTC timezone).

  2. Connect to a stream from a previous night This is not recommended since alerts will flood into the broker as the consumer ingests as fast as it can. For ZTF, you can check ztf.uw.edu/alerts/public/; tar files larger than 74 (presumably in bytes) indicate dates with >0 alerts. Use the Kafka Topic Syntax with a date within the last 7 days.

  3. Use the consumer simulator to control the flow of alerts into the broker. Leave the consumer VM off. See Consumer Simulator for details.

Kafka Topic Syntax

Topic name syntax:

  • ZTF: ztf_yyyymmdd_programid1 where yyyymmdd is replaced with the date.

  • DECAT: decat_yyyymmdd_2021A-0113 where yyyymmdd is replaced with the date.