Pitt-Google Broker
latest

Broker Software

  • Software Overview
  • Architecture Diagrams
  • Broker Instance Keywords
  • Components
  • A Reference for Common Setup Requirements
  • Run a Broker Instance
  • Alerts for Testing
  • Primers for Developers

API Reference

  • broker_utils.avro_schemas
  • broker_utils.consumer_sim
  • broker_utils.data_utils
  • broker_utils.gcp_utils
  • broker_utils.math
  • broker_utils.schema_maps
  • broker_utils.testing
  • broker_utils.types

Working Notes

  • Project Notes
  • Troy Raen’s Working Notes
    • billing: pitt-google-broker-billing
    • gcloud Examples
    • Performance Figures
    • Service Accounts
    • Streaming Stream Looper
    • HEALPix
    • Read The Docs
    • Rubin DPO
    • BigQuery Links
    • Cloud Run info
    • Cloud Build
    • gcloud Examples
    • Metadata Tracking System
    • PyPI setup for pgb-utils
    • Service Accounts
    • Cloud Function to load alerts to BigQuery tables
    • AbrilCVs: CV Catalog from Abril 2020
    • AllWISE: Cross match with the AllWISE catalog in BigQuery public datasets
    • Connect Pitt-Google to the Rubin alert stream testing deployment
    • Setup
    • Ingest the Rubin test stream
    • Pull a Pub/Sub message and open it
    • Alternative methods for handling the schema
    • SuperNNova: Implementing SuperNNova Classifier
    • ZTF Cadence
    • alerts-for-testing: ZTF Alerts for Testing Purposes
    • billing: pitt-google-broker-billing
    • ELAsTiCC Challenge
    • Ingest ZTF Archive
    • Performance Figures
    • python_fncs: Useful Python functions
    • Streaming Stream Looper
    • v0.3.0
    • v0.3.1:
    • v0.4.0
    • v0.5.0
    • v0.6.0
    • v0.6.2: Issue #68
    • v0.7.0
    • v0.7.1
    • djp-pep8: Summary
Pitt-Google Broker
  • Docs »
  • Troy Raen’s Working Notes »
  • BigQuery Links
  • Edit on GitHub

BigQuery Links¶

query = """
CREATE VIEW
  `avid-heading-329016.ztf_alerts.DIASource` (objectId,
    candid,
    jd,
    fid)
PARTITION BY CAST(jd, INT64)
SET OPTIONS (enable_refresh=false)
CLUSTER BY objectId
    AS
SELECT
  objectId,
  candidate.candid AS candid,
  candidate.jd AS jd,
  candidate.fid AS fid
FROM
  `ardent-cycling-243415.ztf_alerts.alerts`
UNION DISTINCT
SELECT
  objectId,
  prvcand.candid AS candid,
  prvcand.jd AS jd,
  prvcand.fid AS fid
FROM
  `ardent-cycling-243415.ztf_alerts.alerts`,
  UNNEST(prv_candidates) prvcand
"""
  • Python Client for Google BigQuery

  • https://stackoverflow.com/questions/50885946/python-bigquery-api-get-table-schema/50908479

  • https://cloud.google.com/bigquery/docs/information-schema-tables#columns_view

  • https://googleapis.dev/python/bigquery/latest/usage/queries.html

  • https://cloud.google.com/bigquery/docs/bigquery-storage-python-pandas#download_query_results_using_the_client_library

  • https://beam.apache.org/releases/pydoc/2.28.0/apache_beam.io.gcp.bigquery.html

  • https://beam.apache.org/documentation/io/built-in/google-bigquery/#reading-with-a-query-string

  • https://www.tutorialspoint.com/sql/sql-distinct-keyword.htm

  • https://cloud.google.com/bigquery/docs/reference/standard-sql/aggregate_functions#avg

  • https://cloud.google.com/bigquery/docs/best-practices-performance-input

  • https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types?hl=pl

  • .

  • https://alerce-new-python-client.readthedocs.io/en/main/tutorials/ztf_api.html#query-lightcurve

  • https://noao.gitlab.io/antares/client/tutorial/searching.html

  • .

  • https://learn.astropy.org/rst-tutorials/conesearch.html?highlight=filtertutorials

  • https://docs.astropy.org/en/stable/coordinates/

  • Pipelining Generators.-,Create%20Generators%20in%20Python,it%20becomes%20a%20generator%20function.)

Next Previous

© Copyright 2021, The Pitt-Google Broker Team Revision 88f3bc22.

Built with Sphinx using a theme provided by Read the Docs.
Read the Docs v: latest
Versions
latest
stable
v-0.6.1-tjr
v-0.6.0-tjr
v-0.5.0-tjr
u-tjr-workingnotes
u-tjr-vm_schedule
u-tjr-update_broker_utils
u-tjr-iam_roles_and_policies
u-tjr-component_integration_tests
u-tjr-architecture_diagram
troy
develop
Downloads
On Read the Docs
Project Home
Builds

Free document hosting provided by Read the Docs.