v0.4.0

  • Documentation of Changes

  • Test the Changes

  • Results Summary


Documentation of Changes

  • PR #60

  • rename-resources.md: accommodate generic survey

  • schemas.md: create schema files for broker utils and BigQuery

  • pypi.md: publish pgb-broker-utils

  • See also: new docs at ../../docs/source/broker


Test the Changes

Dashboards (linked to specific times of tests below):

See also View and Access Resources.

Create a broker instance

# get the code
git clone https://github.com/mwvgroup/Pitt-Google-Broker
cd Pitt-Google-Broker
git checkout v/0.4.0/tjr
cd broker/setup_broker

# create the instance
survey="decat"
# survey="ztf"
testid="v040"
teardown="False"
./setup_broker.sh "$testid" "$teardown" "$survey"

# name some things
consumerVM="${survey}-consumer-${testid}"
nconductVM="${survey}-night-conductor-${testid}"
zone="us-central1-a"

# upload credentials
consumerDir="/home/broker/consumer"
localDir="/Users/troyraen/Documents/PGB/repo"
sudo gcloud compute scp "${localDir}/krb5.conf" "${consumerVM}:/etc/krb5.conf" --zone="$zone"
sudo gcloud compute ssh "$consumerVM" --zone="$zone"  --command="mkdir -p ${consumerDir}"
sudo gcloud compute scp "${localDir}/pitt-reader.user.keytab" "${consumerVM}:${consumerDir}/pitt-reader.user.keytab" --zone="$zone"

# ~stop the VMs after installs are done (this takes ~20 min.~ Now they auto-shutdown
# check the CPU usage on the Dashboard, it should fall below 1%)
# gcloud compute instances stop "$consumerVM" "$nconductVM" --zone="$zone"

Run the broker

# start the night
NIGHT="START"
KAFKA_TOPIC="NONE"
# KAFKA_TOPIC="ztf_yyyymmdd_programid1"
gcloud compute instances add-metadata "$nconductVM" --zone="$zone" \
        --metadata NIGHT="$NIGHT",KAFKA_TOPIC="$KAFKA_TOPIC"
gcloud compute instances start "$nconductVM" --zone "$zone"
from broker_utils import consumer_sim as bcs

testid = 'v040'
survey = 'decat'
sub_id = 'decat-alerts-reservoir-testschema'  # production instance doesn't exist yet
# survey = 'ztf'
# sub_id = 'ztf_alerts-reservoir'  # production instance names are not yet updated
instance = (survey, testid)
# alert_rate = (100, 'once')
alert_rate = 'ztf-active-avg'
runtime = (30, 'min')  # options: 'sec', 'min', 'hr', 'night'(=10 hrs)

bcs.publish_stream(alert_rate, instance, sub_id=sub_id, runtime=runtime)
# end the night
NIGHT="END"
gcloud compute instances add-metadata "$nconductVM" --zone="$zone" \
        --metadata NIGHT="$NIGHT"
gcloud compute instances start "$nconductVM" --zone "$zone"

Delete the broker instance

# survey="decat"
survey="ztf"
testid="v040"
teardown="True"
./setup_broker.sh "$testid" "$teardown" "$survey"

Results Summary

ZTF broker seems to run like normal, no more or different bugs/errors than normal.

DECAT broker’s Dataflow jobs both work (many alerts make it all the way through), but are both very buggy. May not be stable for production use. bq_sink has large number of errors, apparently due to mag=NaN (see Issue #61). value_added producing large number of known issues (for example, see here).