v0.7.0

  • Implements SuperNNova as a Cloud Function.

  • Adds the extragalactic transients filter as a Cloud Function, emitting the Pub/Sub stream that SuperNNova listens to.

See PR #71

Working notes:

Test the changes

Note: The best place to see that/how well this is working is the ZTF production instance, not the testing instance created below. The Cloud Function, etc. is isolated from the rest of the broker pipeline and can’t break anything else, so I threw it into production early on. See:

Code used to create and run the broker testing instance

Create/delete a broker testing instance

# get the code
git clone https://github.com/mwvgroup/Pitt-Google-Broker
cd Pitt-Google-Broker
git checkout v/0.7.0/tjr
cd broker/setup_broker

# create/delete the instance
# survey="decat"
survey="ztf"
testid="v070"
teardown="False"
# teardown="True"
./setup_broker.sh "$testid" "$teardown" "$survey"

# name some things
consumerVM="${survey}-consumer-${testid}"
nconductVM="${survey}-night-conductor-${testid}"
zone="us-central1-a"

# upload credentials
localDir="/Users/troyraen/Documents/broker/repo"
# start the VM if necessary
# gcloud compute instances start "$consumerVM" --zone "$zone"
gcloud compute scp "${localDir}/krb5.conf" "${consumerVM}:~/krb5.conf" --zone="$zone"
gcloud compute scp "${localDir}/pitt-reader.user.keytab" "${consumerVM}:~/pitt-reader.user.keytab" --zone="$zone"
# log in and move the files to the right places
gcloud compute ssh $consumerVM
sudo mv ~/krb5.conf /etc/.
consumerDir="/home/broker/consumer"
sudo mkdir -p $consumerDir
sudo mv ~/pitt-reader.user.keytab ${consumerDir}/.

Start the broker

topic="${survey}-cue_night_conductor-${testid}"
cue=START
attr=KAFKA_TOPIC=NONE
# attr=topic_date=20210820
gcloud pubsub topics publish "$topic" --message="$cue" --attribute="$attr"

Run the consumer simulator

from broker_utils import consumer_sim as bcs

testid = 'v070'
survey = 'ztf'
instance = (survey, testid)
# alert_rate = (5, 'once')
alert_rate = 'ztf-active-avg'
runtime = (10, 'min')  # options: 'sec', 'min', 'hr', 'night'(=10 hrs)

bcs.publish_stream(alert_rate, instance, runtime=runtime)