Auto-scheduler¶
Cue-response checker (checks that the broker responded appropriately to the auto-scheduler’s cue.)
See also:
Overview¶
Broker ingestion of live topics is auto-scheduled with the following process:
Cloud Scheduler cron job -> Pub/Sub message -> Cloud Function -> night-conductor VM startup
The cron job sends a Pub/Sub message that simply contains the cue:
START
or END
. The Cloud Function receives the message, sets
appropriate metadata attributes on the night conductor VM and starts it.
(Note that you can start/stop the broker manually at any time by sending a message to the auto-scheduler’s Pub/Sub topic. See Run the Broker.)
Two cron jobs are scheduled, one each to start and end the night. Both processes use the same Pub/Sub topic and Cloud Function.
Example: Change the time the broker auto-starts each night.
survey=
testid=
jobname="${survey}-cue_night_conductor_START-${testid}"
schedule='0 2 * * *' # UTC. unix-cron format
# https://cloud.google.com/scheduler/docs/configuring/cron-job-schedules
gcloud scheduler jobs update pubsub $jobname --schedule "${schedule}"
Example: Pause/resume the broker’s auto-schedule.
survey=
testid=
startjob="${survey}-cue_night_conductor_START-${testid}"
endjob="${survey}-cue_night_conductor_END-${testid}"
# pause the jobs
gcloud scheduler jobs pause $startjob
gcloud scheduler jobs pause $endjob
# resume the jobs
gcloud scheduler jobs resume $startjob
gcloud scheduler jobs resume $endjob
By default, the cron jobs of a Testing instance are paused immediately after creation so that the instance does not run automatically.
Cue-response checker¶
The response checker Cloud Function is triggered by the auto-scheduler’s Pub/Sub topic (see above). It receives the auto-scheduler’s cue and then checks whether the broker responds appropriately. If the response is not as expected, “Critical” errors are raised which trigger a GCP alerting policy.
Alerting policy¶
An alerting policy was created manually to notify Troy Raen of anything
written to the log named check-cue-response-cloudfnc
that has
severity 'CRITICAL'
. Every broker instance has a unique
check_cue_response
Cloud Function, but they all write to the same
log. Therefore, a new policy does not need to be created with each new
broker instance. (Also, recall that the auto-scheduler is typically only
active in Production instances.)
To update the existing policy, or create a new one, see: