Package org.apache.nemo.common.ir.vertex

Class SourceVertex<O>

java.lang.Object

org.apache.nemo.common.dag.Vertex

org.apache.nemo.common.ir.vertex.IRVertex

org.apache.nemo.common.ir.vertex.SourceVertex<O>

Type Parameters:

O - output type.

All Implemented Interfaces:

java.io.Serializable, Cloneable<IRVertex>

Direct Known Subclasses:

BeamBoundedSourceVertex, BeamUnboundedSourceVertex, CachedSourceVertex, EmptyComponents.EmptySourceVertex, InMemorySourceVertex

public abstract class SourceVertex<O>
extends IRVertex

IRVertex that reads data from an external source. It is to be implemented in the compiler frontend with source-specific data fetching logic.

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor	Description
SourceVertex()	Constructor for SourceVertex.
SourceVertex(SourceVertex that)	Copy Constructor for SourceVertex.

Method Summary

Modifier and Type	Method	Description
abstract void	clearInternalStates()	Clears internal states, must be called after getReadables().
abstract long	getEstimatedSizeBytes()	Gets the estimated size of bytes.
abstract java.util.List<Readable<O>>	getReadables(int desiredNumOfSplits)	Gets parallel readables.
abstract boolean	isBounded()

Methods inherited from class org.apache.nemo.common.ir.vertex.IRVertex

copyExecutionPropertiesTo, getExecutionProperties, getIRVertexPropertiesAsJsonNode, getPropertyValue, isUtilityVertex, setProperty, setPropertyPermanently

Methods inherited from class org.apache.nemo.common.dag.Vertex

getId, getNumericId, getPropertiesAsJsonNode

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.nemo.common.Cloneable

getClone

Constructor Detail

SourceVertex

public SourceVertex()

Constructor for SourceVertex.

SourceVertex

public SourceVertex(SourceVertex that)

Copy Constructor for SourceVertex.

Parameters:

that - the source object for copying

Method Detail

isBounded

public abstract boolean isBounded()

Returns:

true if it is bounded source

getReadables

public abstract java.util.List<Readable<O>> getReadables(int desiredNumOfSplits)
                                                  throws java.lang.Exception

Gets parallel readables.

Parameters:

desiredNumOfSplits - number of splits desired.

Returns:

the list of readables.

Throws:

java.lang.Exception - if fail to get.

getEstimatedSizeBytes

public abstract long getEstimatedSizeBytes()

Gets the estimated size of bytes. Returns 0L if not applicable.

Returns:

size of input bytes.

clearInternalStates

public abstract void clearInternalStates()

Clears internal states, must be called after getReadables(). Concretely, this clears the huge list of input splits held by objects like BeamBoundedSourceVertex before sending the vertex to remote executors. Between clearing states of an existing vertex, and creating a new vertex, we've chosen the former approach to ensure consistent use of the same IRVertex object across the compiler, the master, and the executors.