broker_utils.data_utils
The data_utils
module contains common functions used to manipulate
survey and broker data.
- exception broker.broker_utils.broker_utils.data_utils.OpenAlertError[source]
Raised after methods for all known alert formats have tried and failed to open the alert.
- broker.broker_utils.broker_utils.data_utils.alert_dict_to_dataframe(alert_dict, schema_map)[source]
Packages an alert into a dataframe. Adapted from: https://github.com/ZwickyTransientFacility/ztf-avro-alert/blob/master/notebooks/Filtering_alerts.ipynb
- Return type
DataFrame
- broker.broker_utils.broker_utils.data_utils.alert_lite_to_dataframe(alert_dict)[source]
Package an alert into a dataframe.
Adapted from: https://github.com/ZwickyTransientFacility/ztf-avro-alert/blob/master/notebooks/Filtering_alerts.ipynb
- Return type
DataFrame
- broker.broker_utils.broker_utils.data_utils.decode_alert(alert_avro, return_as='dict', drop_cutouts=False, **kwargs)[source]
*Deprecated. Use open_alert() instead.*
Load an alert Avro and return in requested format.
- Return type
Union
[dict
,DataFrame
]
- broker.broker_utils.broker_utils.data_utils.load_alert(fin, return_as='dict', **kwargs)[source]
*Deprecated. Use open_alert() instead.*
Load alert from file at
fin
and return in formatreturn_as
.- Parameters
fin (
Union
[str
,Path
]) – Path to an alert avro file.return_as (
str
) – Format the alert will be returned in. One of ‘bytes’ or argument accepted bydecode_alert
.kwargs – Keyword arguments for
decode_alert
.
- Return type
Union
[bytes
,dict
,DataFrame
]
- broker.broker_utils.broker_utils.data_utils.open_alert(alert, return_as='dict', drop_cutouts=False, **kwargs)[source]
Load
alert
, decode it, and return it in the requested format.Background: The broker deals with alert data that can be packaged in many different ways. For example:
Files in Avro format
Pub/Sub messages – the message payload is a bytes object with either Avro or json serialization
Cloud Functions further encodes incoming Pub/Sub messages as base64 strings
Any Avro-serialized object may or may not have its schema attached as a header (but we must have the schema in order to deserialize it)
This function adopts a brute-force strategy. It does not try to inspect
alert
and determine its format. Instead, it tries repeatedly to load/decodealert
and return it as requested, trying at least one method for each input format listed above. It catches nearlly allException``s along the way. Set the logger level to DEBUG for a record of the try/excepts. If it runs out of methods to try, it raises an ``OpenAlertError
.- Parameters
alert (
Union
[str
,Path
,bytes
]) – Either the path of Avro file to load, or the bytes encoding the alert.return_as (
str
) – Format the alert will be returned in. One of ‘bytes’, ‘dict’ or ‘df’.drop_cutouts (
bool
) – Whether to drop or return the image cutouts.kwargs – Keyword arguments passed to
_avro_to_dicts()
,_drop_cutouts()
andalert_dict_to_dataframe()
. Note that ifalert
is Avro and schemaless you must pass the keyword argumentload_schema
(see_avro_to_dicts()
) with an appropriate value.
- Return type
Union
[bytes
,dict
,DataFrame
]- Returns
alert data in the requested format
- Raises
OpenAlertError` if none of the methods successfully open the alert –