stream-looper.md

Setup stream-looper VM and topic

Create the Pub/Sub topic and allow public subscriptions

PROJECT=$GOOGLE_CLOUD_PROJECT
TOPIC="ztf-loop"
gcloud pubsub topics create $TOPIC
topic_path="projects/${PROJECT}/topics/${TOPIC}"
fname="/Users/troyraen/Downloads/topic_policy.yaml"  # created in external-connection.md
fnametmp="/Users/troyraen/Downloads/topic_policy_tmp.yaml"
# download the topic to get the etag
gcloud pubsub topics get-iam-policy $topic_path --format yaml > $fnametmp
# update the fname file with the current etag from the tmp file
# set the new policy
gcloud pubsub topics set-iam-policy $topic_path $fname

Create the VM and setup the consumer simulator

vmname="stream-looper"
zone="us-central1-a"
machinetype="e2-standard-2"
startupscript="#! /bin/bash
apt-get update
apt-get install -y python3-pip screen ipython3
pip3 install pgb_broker_utils
"
gcloud compute instances create "$vmname" \
    --zone="$zone" \
    --machine-type="$machinetype" \
    --scopes=cloud-platform \
    --metadata=google-logging-enabled=true,startup-script="$startupscript"
# unset the startup script
gcloud compute instances add-metadata "$vmname" --zone="$zone" --metadata=startup-script=""

# ssh in
# gcloud compute ssh $vmname

Set a startup script to run consumer simulator indefinitely

Python script to trigger at startup. Save the following code to a root-executable file at /home/consumer_sim/run-looper-indefinitely.py:

#!/usr/bin/env python3
# -*- coding: UTF-8 -*-
from broker_utils import consumer_sim as bcs

# set args to publish 1 alert every second
alert_rate = (60, 'perMin')
kwargs = {
    'instance': None,
    'runtime': (1, 'night'),
    'publish_batch_every': (1, 'sec'),
    'sub_id': 'ztf-alerts-reservoir',
    'topic_id': 'ztf-loop',
    'auto_confirm': True,
}

# run indefinitely
while True:
    bcs.publish_stream(alert_rate, **kwargs)

Set a startup script to execute the python file in a background thread:

startupscript="#! /bin/bash
dir=/home/consumer_sim/
fname=run-looper-indefinitely
nohup python3 \${dir}\${fname}.py >> \${dir}\${fname}.out 2>&1 &
"

gcloud compute instances add-metadata "$vmname" --metadata=startup-script="$startupscript"

Stop and restart the vm:

gcloud compute instances stop "$vmname"
gcloud compute instances start "$vmname"