docs/source/working-notes/troyraen/gcloud-examples.md
gcloud Examples
External reference:
Setup
The following variables are used in many of the examples on this page. Set them to your own preferences.
SURVEY="ztf"
TESTID="mytest"
export GOOGLE_CLOUD_PROJECT="avid-heading-329016" # this is our development project
export GOOGLE_APPLICATION_CREDENTIALS="/path/to/GCP_auth_key.json"
Default zone/region when creating VMs and other compute-related resources:
export CLOUDSDK_COMPUTE_REGION='us-central1'
export CLOUDSDK_COMPUTE_ZONE='us-central1-a'
BigQuery
Create a dataset
dataset_name="my-dataset" # choose a name for your dataset
# Create the dataset
# bq library comes with gcloud
bq mk --dataset "${GOOGLE_CLOUD_PROJECT}:dataset_name" \
--location="${REGION}"
Cloud Functions
CF_name="my-cloud-function" # choose a name for your Cloud Function
entry_point="run" # fill in name of module's entry-point function
trigger_topic="my-topic" # fill in name of Pub/Sub topic that will trigger your function
# deploy. run this command from within the directory containing your Cloud Function
gcloud functions deploy "${CF_name}" \
--entry-point "${entry_point}" \
--runtime python37 \
--trigger-topic "$trigger_topic" \
--set-env-vars TESTID="${TESTID}",SURVEY="${SURVEY}"
# delete
gcloud functions delete "${CF_name}"
Cloud Run
Initialize variables
PROJECT_ID=$GOOGLE_CLOUD_PROJECT
PROJECT_NUMBER=$(gcloud projects list \
--filter="$(gcloud config get-value project)" \
--format="value(PROJECT_NUMBER)" \
)
# name for the Run service you're deploying, and related resources
NAME_STUB="xmatch_AbrilCVs"
NAME_STUB_LOWER_DASH="xmatch-abrilcvs"
NAME="${SURVEY}-${NAME_STUB}"
NAME_LOWER_DASH="${SURVEY}-${NAME_STUB_LOWER_DASH}"
if [ "$TESTID" != "False" ]; then
NAME="${NAME}-${TESTID}"
NAME_LOWER_DASH="${NAME_LOWER_DASH}-${TESTID}"
fi
# service account with permissions to invoke Cloud Run
SERVICE_ACCOUNT_NAME="cloud-run-invoker"
DISPLAYED_SERVICE_ACCOUNT_NAME="Cloud Run Invoker Service Account"
SERVICE_ACCOUNT_ADDRESS="${SERVICE_ACCOUNT_NAME}@${PROJECT_ID}.iam.gserviceaccount.com"
# container image
IMAGE_URL="gcr.io/${PROJECT_ID}/${NAME_LOWER_DASH}:latest"
# Get the endpoint from Run deployment output (below),
# and be sure to add the `route` to the end of the url
# (assigned in your Run code via the tag `@app.route()`)
ENDPOINT=""
# Pub/Sub
TOPIC = "${SURVEY}-alerts" # trigger topic
TOPIC_PROJECT="ardent-cycling-243415" # project that owns the topic
SUBSCRIPTION="${TOPIC}-${NAME_STUB}" # push subscription, to be attached to trigger topic
if [ "$TESTID" != "False" ]; then
TOPIC="${TOPIC}-${TESTID}"
SUBSCRIPTION="${SUBSCRIPTION}-${TESTID}"
fi
ACK_DEADLINE=300
Deploy cloud run
# cd into the directory with the Run code
# cd /Users/troyraen/Documents/broker/abril/broker/cloud_run/abril_cv
# create and upload container
gcloud builds submit --tag "$IMAGE_URL"
# deploy to cloud run
gcloud run deploy "$NAME_LOWER_DASH" --image "$IMAGE_URL" \
--no-allow-unauthenticated \
--set-env-vars GCP_PROJECT="$PROJECT_ID",TESTID="$TESTID",SURVEY="$SURVEY"
# This returns a Service URL that you will need in the following steps.
# Set the variable:
# ENDPOINT="${service_url}${route}"
# where `route` is assigned in your Run code via the tag `@app.route()`
# Example service URL: https://xmatch-allwise-3tp3qztwza-uc.a.run.app
Allow Pub/Sub to create authentication tokens in the project:
gcloud projects add-iam-policy-binding "${PROJECT_ID}" \
--member="serviceAccount:service-${PROJECT_NUMBER}@gcp-sa-pubsub.iam.gserviceaccount.com" \
--role="roles/iam.serviceAccountTokenCreator"
Create a service account and give it permission to invoke cloud run. Alternately, you can re-use an existing service account, just add the policy binding.
gcloud iam service-accounts create "$SERVICE_ACCOUNT_NAME" \
--display-name "$DISPLAYED_SERVICE_ACCOUNT_NAME"
gcloud run services add-iam-policy-binding "$NAME_LOWER_DASH" \
--member=serviceAccount:"$SERVICE_ACCOUNT_ADDRESS" \
--role=roles/run.invoker
Create the subscription with the service account attached
gcloud pubsub subscriptions create "$SUBSCRIPTION" \
--topic "$TOPIC" \
--topic-project="$TOPIC_PROJECT" \
--push-endpoint="$ENDPOINT" \
--push-auth-service-account="$SERVICE_ACCOUNT_ADDRESS" \
--ack-deadline="$ACK_DEADLINE"
Cloud Storage
Download the avro file for a single alert:
bucket="${GOOGLE_CLOUD_PROJECT}-${survey}-alert_avros"
filename="<objectId>.<sourceId>.<kafkaTopic>.avro"
# filename="100000238.200000476050.elasticc-2022fall.avro"
local_dir="/<path>/<to>/<local>/<dir>"
gsutil cp "gs://${bucket}/${filename}" ${local_dir}/.
Compute Engine
Compute Engine Schedule
Create a schedule (only needs to be done once) unix-cron format
schedule_name="my-schedule" # choose schedule name
start_schedule="* * * * *" # choose start time. (UTC, unix-cron format, link above)
stop_schedule="* * * * *" # choose stop time. (UTC, unix-cron format, link above)
description="schedule" # fill in a description
gcloud compute resource-policies create instance-schedule "${schedule_name}" \
--description="${description}" \
--vm-start-schedule="${start_schedule}" \
--vm-stop-schedule="${stop_schedule}" \
--timezone="UTC"
Attach/unattach a schedule to a VM
vm_name="my-vm" # set this to the VM name
schedule_name="my-schedule" # set this to the schedule name
# Attach schedule
gcloud compute instances add-resource-policies "${nconductVM}" \
--resource-policies="${nconductVMsched}"
# Unattach schedule
gcloud compute instances remove-resource-policies "${nconductVM}" \
--resource-policies="${nconductVMsched}"
Pub/Sub
# create topic
TOPIC="mytopic"
gcloud pubsub topics create "$TOPIC"
# create subscription
SUBSCRIPTION="mysubscription"
TOPIC="mytopic"
TOPIC_PROJECT=$GOOGLE_CLOUD_PROJECT
gcloud pubsub subscriptions create "$SUBSCRIPTION" \
--topic="$TOPIC" \
--topic-project="$TOPIC_PROJECT"
Service account
See also service-account.md.
NAME="tjraen-owner"
PROJECT_ID=$GOOGLE_CLOUD_PROJECT
FILE_NAME_STUB="GCP_auth_key-user_test.json"
FILE_NAME="/Users/troyraen/Documents/broker/repo/${FILE_NAME_STUB}"
gcloud config set project $PROJECT_ID
gcloud iam service-accounts create "$NAME"
gcloud projects add-iam-policy-binding "$PROJECT_ID" \
--member="serviceAccount:${NAME}@${PROJECT_ID}.iam.gserviceaccount.com" \
--role="roles/owner"
gcloud iam service-accounts keys create "$FILE_NAME" \
--iam-account="${NAME}@${PROJECT_ID}.iam.gserviceaccount.com"