Package org.apache.nemo.compiler.frontend.beam.transform

Class GBKTransform<K,InputT,OutputT>

java.lang.Object

org.apache.nemo.compiler.frontend.beam.transform.AbstractDoFnTransform<org.apache.beam.sdk.values.KV<K,InputT>,org.apache.beam.runners.core.KeyedWorkItem<K,InputT>,org.apache.beam.sdk.values.KV<K,OutputT>>

org.apache.nemo.compiler.frontend.beam.transform.GBKTransform<K,InputT,OutputT>

Type Parameters:

K - key type

InputT - input type

OutputT - output type

All Implemented Interfaces:

java.io.Serializable, Transform<org.apache.beam.sdk.util.WindowedValue<org.apache.beam.sdk.values.KV<K,InputT>>,org.apache.beam.sdk.util.WindowedValue<org.apache.beam.sdk.values.KV<K,OutputT>>>

public final class GBKTransform<K,InputT,OutputT>
extends AbstractDoFnTransform<org.apache.beam.sdk.values.KV<K,InputT>,org.apache.beam.runners.core.KeyedWorkItem<K,InputT>,org.apache.beam.sdk.values.KV<K,OutputT>>

This transform executes GroupByKey transformation and CombinePerKey transformation when input data is unbounded or is not in a global window.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
class	GBKTransform.GBKOutputCollector	Wrapper class for OutputCollector.

Nested classes/interfaces inherited from interface org.apache.nemo.common.ir.vertex.transform.Transform

Transform.Context

Constructor Summary

Constructors

Constructor

Description

GBKTransform(org.apache.beam.sdk.coders.Coder<org.apache.beam.sdk.values.KV<K,InputT>> inputCoder, java.util.Map<org.apache.beam.sdk.values.TupleTag<?>,org.apache.beam.sdk.coders.Coder<?>> outputCoders, org.apache.beam.sdk.values.TupleTag<org.apache.beam.sdk.values.KV<K,OutputT>> mainOutputTag, org.apache.beam.sdk.values.WindowingStrategy<?,?> windowingStrategy, org.apache.beam.sdk.options.PipelineOptions options, org.apache.beam.runners.core.SystemReduceFn reduceFn, org.apache.beam.sdk.transforms.DoFnSchemaInformation doFnSchemaInformation, org.apache.beam.sdk.transforms.display.DisplayData displayData, boolean isPartialCombining)

Method Summary

Modifier and Type	Method	Description
protected void	beforeClose()	This advances the input watermark and processing time to the timestamp max value in order to emit all data.
boolean	getIsPartialCombining()	Accessor for isPartialCombining.
void	onData(org.apache.beam.sdk.util.WindowedValue<org.apache.beam.sdk.values.KV<K,InputT>> element)	Every time a single element arrives, this method invokes runner to process a single element.
void	onWatermark(Watermark watermark)	Trigger timers that need to be fired at watermark and emit output watermark.
protected org.apache.beam.sdk.transforms.DoFn	wrapDoFn(org.apache.beam.sdk.transforms.DoFn doFn)	This creates a new DoFn that groups elements by key and window.

Methods inherited from class org.apache.nemo.compiler.frontend.beam.transform.AbstractDoFnTransform

close, getDoFn, getOutputCollector, onLatencymark, prepare, toString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

GBKTransform

public GBKTransform(org.apache.beam.sdk.coders.Coder<org.apache.beam.sdk.values.KV<K,InputT>> inputCoder,
                    java.util.Map<org.apache.beam.sdk.values.TupleTag<?>,org.apache.beam.sdk.coders.Coder<?>> outputCoders,
                    org.apache.beam.sdk.values.TupleTag<org.apache.beam.sdk.values.KV<K,OutputT>> mainOutputTag,
                    org.apache.beam.sdk.values.WindowingStrategy<?,?> windowingStrategy,
                    org.apache.beam.sdk.options.PipelineOptions options,
                    org.apache.beam.runners.core.SystemReduceFn reduceFn,
                    org.apache.beam.sdk.transforms.DoFnSchemaInformation doFnSchemaInformation,
                    org.apache.beam.sdk.transforms.display.DisplayData displayData,
                    boolean isPartialCombining)

Method Detail

wrapDoFn

protected org.apache.beam.sdk.transforms.DoFn wrapDoFn(org.apache.beam.sdk.transforms.DoFn doFn)

This creates a new DoFn that groups elements by key and window.

Parameters:

doFn - original doFn.

Returns:

GroupAlsoByWindowViaWindowSetNewDoFn

onData

public void onData(org.apache.beam.sdk.util.WindowedValue<org.apache.beam.sdk.values.KV<K,InputT>> element)

Every time a single element arrives, this method invokes runner to process a single element.

Parameters:

element - input data element.

onWatermark

public void onWatermark(Watermark watermark)
                 throws java.lang.RuntimeException

Trigger timers that need to be fired at watermark and emit output watermark.

Parameters:

watermark - watermark

Throws:

java.lang.RuntimeException

beforeClose

protected void beforeClose()

This advances the input watermark and processing time to the timestamp max value in order to emit all data.

getIsPartialCombining

public boolean getIsPartialCombining()

Accessor for isPartialCombining.

Returns:

whether it is partial combining.