pgb_utils.beam¶

The beam module facilitates connecting Apache Beam pipelines to Pitt-Google Broker’s BigQuery databases and Pub/Sub streams.

class pgb_utils.pgb_utils.beam.ExtractAlertDict(*unused_args, **unused_kwargs)[source]¶

process(msg)[source]¶

Method to use for processing elements.

This is invoked by DoFnRunner for each element of a input PCollection.

The following parameters can be used as default values on process arguments to indicate that a DoFn accepts the corresponding parameters. For example, a DoFn might accept the element and its timestamp with the following signature:

def process(element=DoFn.ElementParam, timestamp=DoFn.TimestampParam):
  ...

The full set of parameters is:

DoFn.ElementParam: element to be processed, should not be mutated.
DoFn.SideInputParam: a side input that may be used when processing.
DoFn.TimestampParam: timestamp of the input element.
DoFn.WindowParam: Window the input element belongs to.
DoFn.TimerParam: a userstate.RuntimeTimer object defined by the spec of the parameter.
DoFn.StateParam: a userstate.RuntimeState object defined by the spec of the parameter.
DoFn.KeyParam: key associated with the element.
DoFn.RestrictionParam: an iobase.RestrictionTracker will be provided here to allow treatment as a Splittable DoFn. The restriction tracker will be derived from the restriction provider in the parameter.
DoFn.WatermarkEstimatorParam: a function that can be used to track output watermark of Splittable DoFn implementations.

Parameters

element – The element to be processed
*args – side inputs
**kwargs – other keyword arguments.

Returns

An Iterable of output elements or None.

class pgb_utils.pgb_utils.beam.ExtractHistoryDf(source='bigquery')[source]¶

process(row)[source]¶