docs/source/working-notes/troyraen/v0.6.0/test-pubsub-pricing.md

Testing Pub/Sub tutorial from external account.

Summary:

Date of tests: 8/6/2021 (note there were no live ZTF streams Aug 5-7)

ardent-cycling-243415 account:

  • Pub/Sub billing charges look as expected.

  • Charges for VM in Germany look as expected.

  • All ingress/egress fees are $0.00

my-pgb-project-3 account:

I pulled and processed ~375000 messages with an average of 70607 bytes =~ 26.5 GB of data. This should have been >2x the data allowed in the Free Tier. I don’t know why it didn’t cut me off. I successfully pulled all this stuff from a VM in Germany with no egress restrictions/fees.

There is no way for this account to be charged. Billing is disabled and I have never created a billing account. I can’t even see a billing report because I don’t have a billing account.

Initial thoughts:

  • Testing pricing, free tier, egress, msg delivery

  • What happens if you don’t enable billing and reach the Free Tier limit?

  • Does “message delivery” mean delivered to the subscription or delivered to the user (on pull request)?

    • Note on Pub/Sub pricing page it says: “Storage of unacknowledged messages does not result in fees.”

Outline:

  • streamed ~300,000 messages to the subscription projects/my-pgb-project-3/subscriptions/test

  • setup a compute engine in Germany to pull the messages

    • compute engine is in pgb project since can’t do VMs with free accounts. setup credentials to project my-pgb-project-3 following v0.5.0/external-connection.md

  • pulled the messages from the test subscription and wrote a sampling of message sizes to file

Code to do it:

Use the stream-looper VM to publish the messages.

Setup VM to pull from my-pgb-project-3:

vmname=pubsub-test
zone=europe-west3-a  # Frankfurt, Germany
machinetype="e2-standard-2"
gcloud compute instances create "$vmname" \
    --zone="$zone" \
    --machine-type="$machinetype" \
    --scopes=cloud-platform

gcloud compute ssh $vmname --zone $zone

sudo apt-get update
sudo apt-get install -y python3-pip screen ipython3
sudo pip3 install pgb-utils

Pull and process messages:

import pgb_utils as pgb
import random

def callback(message):
    # save some stuff from a sampling of messages
    n = random.uniform(0,1)
    if n > 0.99:
        try:
            fout = 'track-msgs.txt'
            with open(fout, 'a') as f:
                f.write(f'{message.size}\n')
        except:
            pass
    # acknowledge
    message.ack()

sub_name = 'test'
pgb.pubsub.streamingPull(sub_name, callback, timeout=None)