Package org.apache.nemo.compiler.frontend.beam.source

Class BeamUnboundedSourceVertex<O,​M extends org.apache.beam.sdk.io.UnboundedSource.CheckpointMark>

  • Type Parameters:
    O - output type.
    M - checkpoint mark type.
    All Implemented Interfaces:
    java.io.Serializable, Cloneable<IRVertex>
    public final class BeamUnboundedSourceVertex<O,​M extends org.apache.beam.sdk.io.UnboundedSource.CheckpointMark>
extends SourceVertex<java.lang.Object>
    SourceVertex implementation for UnboundedSource.
    See Also:
    Serialized Form

    • Constructor Detail

      • BeamUnboundedSourceVertex

        public BeamUnboundedSourceVertex​(org.apache.beam.sdk.io.UnboundedSource<O,​M> source,
                                 org.apache.beam.sdk.transforms.display.DisplayData displayData)
        The default constructor for beam unbounded source.
        Parameters:
        source - unbounded source.
        displayData - static display data associated with a pipeline component.

    • Method Detail

      • getClone

        public IRVertex getClone()
        Description copied from interface: Cloneable
        Creates and returns a copy of this object.

        The precise meaning of "copy" may depend on the class of the object. The general intent is that, all fields of the object are copied.

        Returns:
        a clone of this object.

      • isBounded

        public boolean isBounded()
        Specified by:
        isBounded in class SourceVertex<java.lang.Object>
        Returns:
        true if it is bounded source

      • getReadables

        public java.util.List<Readable<java.lang.Object>> getReadables​(int desiredNumOfSplits)
                                                        throws java.lang.Exception
        Description copied from class: SourceVertex
        Gets parallel readables.
        Specified by:
        getReadables in class SourceVertex<java.lang.Object>
        Parameters:
        desiredNumOfSplits - number of splits desired.
        Returns:
        the list of readables.
        Throws:
        java.lang.Exception - if fail to get.

      • getEstimatedSizeBytes

        public long getEstimatedSizeBytes()
        Description copied from class: SourceVertex
        Gets the estimated size of bytes. Returns 0L if not applicable.
        Specified by:
        getEstimatedSizeBytes in class SourceVertex<java.lang.Object>
        Returns:
        size of input bytes.

      • clearInternalStates

        public void clearInternalStates()
        Description copied from class: SourceVertex
        Clears internal states, must be called after getReadables(). Concretely, this clears the huge list of input splits held by objects like BeamBoundedSourceVertex before sending the vertex to remote executors. Between clearing states of an existing vertex, and creating a new vertex, we've chosen the former approach to ensure consistent use of the same IRVertex object across the compiler, the master, and the executors.
        Specified by:
        clearInternalStates in class SourceVertex<java.lang.Object>

      • getPropertiesAsJsonNode

        public com.fasterxml.jackson.databind.node.ObjectNode getPropertiesAsJsonNode()
        Overrides:
        getPropertiesAsJsonNode in class Vertex
        Returns:
        JSON representation of additional properties