broker_utils.data_utils

The data_utils module contains common functions used to manipulate survey and broker data.

exception broker.broker_utils.broker_utils.data_utils.OpenAlertError[source]

Raised after methods for all known alert formats have tried and failed to open the alert.

broker.broker_utils.broker_utils.data_utils.alert_dict_to_dataframe(alert_dict, schema_map)[source]

Packages an alert into a dataframe. Adapted from: https://github.com/ZwickyTransientFacility/ztf-avro-alert/blob/master/notebooks/Filtering_alerts.ipynb

Return type

DataFrame

broker.broker_utils.broker_utils.data_utils.alert_lite_to_dataframe(alert_dict)[source]

Package an alert into a dataframe.

Adapted from: https://github.com/ZwickyTransientFacility/ztf-avro-alert/blob/master/notebooks/Filtering_alerts.ipynb

Return type

DataFrame

broker.broker_utils.broker_utils.data_utils.decode_alert(alert_avro, return_as='dict', drop_cutouts=False, **kwargs)[source]

*Deprecated. Use open_alert() instead.*

Load an alert Avro and return in requested format.

Return type

Union[dict, DataFrame]

broker.broker_utils.broker_utils.data_utils.load_alert(fin, return_as='dict', **kwargs)[source]

*Deprecated. Use open_alert() instead.*

Load alert from file at fin and return in format return_as.

Parameters

  • fin (Union[str, Path]) – Path to an alert avro file.

  • return_as (str) – Format the alert will be returned in. One of ‘bytes’ or argument accepted by decode_alert.

  • kwargs – Keyword arguments for decode_alert.

Return type

Union[bytes, dict, DataFrame]

broker.broker_utils.broker_utils.data_utils.open_alert(alert, return_as='dict', drop_cutouts=False, **kwargs)[source]

Load alert, decode it, and return it in the requested format.

Background: The broker deals with alert data that can be packaged in many different ways. For example:

  • Files in Avro format

  • Pub/Sub messages – the message payload is a bytes object with either Avro or json serialization

  • Cloud Functions further encodes incoming Pub/Sub messages as base64 strings

  • Any Avro-serialized object may or may not have its schema attached as a header (but we must have the schema in order to deserialize it)

This function adopts a brute-force strategy. It does not try to inspect alert and determine its format. Instead, it tries repeatedly to load/decode alert and return it as requested, trying at least one method for each input format listed above. It catches nearlly all Exception``s along the way. Set the logger level to DEBUG for a record of the try/excepts. If it runs out of methods to try, it raises an ``OpenAlertError.

Parameters

  • alert (Union[str, Path, bytes]) – Either the path of Avro file to load, or the bytes encoding the alert.

  • return_as (str) – Format the alert will be returned in. One of ‘bytes’, ‘dict’ or ‘df’.

  • drop_cutouts (bool) – Whether to drop or return the image cutouts.

  • kwargs – Keyword arguments passed to _avro_to_dicts(), _drop_cutouts() and alert_dict_to_dataframe(). Note that if alert is Avro and schemaless you must pass the keyword argument load_schema (see _avro_to_dicts()) with an appropriate value.

Return type

Union[bytes, dict, DataFrame]

Returns

alert data in the requested format

Raises

OpenAlertError` if none of the methods successfully open the alert –

broker.broker_utils.broker_utils.data_utils.ztf_fid_names()[source]

Return a dictionary mapping the ZTF fid (filter ID) to the common name.

Return type

dict