djp-pep8: Summary

The value_added Dataflow job fails because I have made a backwards incompatible change to broker_utils. Everything else in the pipeline looks like it runs fine.

I did not test the changes to broker_utils… the components get it from PyPI, not from this repo.

Code used to create and run the broker testing instance

Create/delete a broker testing instance

# get the code
git clone https://github.com/mwvgroup/Pitt-Google-Broker
cd Pitt-Google-Broker
git checkout dfperrefort/pep8
cd broker/setup_broker

# create/delete the instance
# survey="decat"
survey="ztf"
testid="pep8"
teardown="False"
# teardown="True"
./setup_broker.sh "$testid" "$teardown" "$survey"


# name some things
nconductVM="${survey}-night-conductor-${testid}"

Start the broker

topic="${survey}-cue_night_conductor-${testid}"
cue=START
attr=KAFKA_TOPIC=NONE
# attr=topic_date=20210820
gcloud pubsub topics publish "$topic" --message="$cue" --attribute="$attr"

Run the consumer simulator

from broker_utils import consumer_sim as bcs

testid = 'pep8'
survey = 'ztf'
instance = (survey, testid)
# alert_rate = (5, 'once')
alert_rate = 'ztf-active-avg'
runtime = (10, 'min')  # options: 'sec', 'min', 'hr', 'night'(=10 hrs)

bcs.publish_stream(alert_rate, instance, runtime=runtime)