docs/source/working-notes/troyraen/v0.5.0/README.md

v0.5.0


Results Summary

Auto-scheduler and cue-response checker appear to be working smoothly.

Documentation is back up on ReadTheDocs.


Documentation of Changes

Started but didn’t finish; TODO in a future version (this is v0.7.0):


Test the Changes

Create/delete a broker instance

Create instance

# get the code
git clone https://github.com/mwvgroup/Pitt-Google-Broker
cd Pitt-Google-Broker
git checkout v/0.5.0/tjr
cd broker/setup_broker

# create/delete the instance
# survey="decat"
survey="ztf"
testid="v050"
teardown="False"
# teardown="True"
./setup_broker.sh "$testid" "$teardown" "$survey"


# name some things
consumerVM="${survey}-consumer-${testid}"
nconductVM="${survey}-night-conductor-${testid}"
zone="us-central1-a"

# upload credentials
localDir="/Users/troyraen/Documents/broker/repo"
# start the VM if necessary
# gcloud compute instances start "$consumerVM" --zone "$zone"
gcloud compute scp "${localDir}/krb5.conf" "${consumerVM}:~/krb5.conf" --zone="$zone"
gcloud compute scp "${localDir}/pitt-reader.user.keytab" "${consumerVM}:~/pitt-reader.user.keytab" --zone="$zone"
# log in and move the files to the right places
gcloud compute ssh $consumerVM
sudo mv ~/krb5.conf /etc/.
consumerDir="/home/broker/consumer"
sudo mkdir -p $consumerDir
sudo mv ~/pitt-reader.user.keytab ${consumerDir}/.

Test the broker

Test the auto-scheduler

Schedule the broker to ingest a live topic for a few minutes and then shutdown.

Dashboard during test. Expect to see normal broker operation from startup to shutdown, except that there are no alerts in this topic.

gcloud scheduler jobs resume ztf-cue_night_conductor_START-v050
gcloud scheduler jobs resume ztf-cue_night_conductor_END-v050
gcloud scheduler jobs update pubsub ztf-cue_night_conductor_START-v050 --schedule '01 00 * * *'
gcloud scheduler jobs update pubsub ztf-cue_night_conductor_END-v050 --schedule '11 00 * * *'

Trigger from a Pub/Sub message (bypass the cron job)

survey=ztf
testid=v050
topic="${survey}-cue_night_conductor-${testid}"
cue=START
# attr=KAFKA_TOPIC=NONE  # leave consumer VM off
attr=topic_date=20210727
gcloud pubsub topics publish "$topic" --message="$cue" --attribute="$attr"