All Classes Interface Summary Class Summary Enum Summary Exception Summary Annotation Types Summary
Class |
Description |
AbsentBlockException |
An exception which represents the requested block is neither AVAILABLE nor IN_PROGRESS.
|
AbstractBlockStore |
This abstract class represents a default BlockStore ,
which contains other components used in each implementation of BlockStore .
|
AbstractDoFnTransform<InputT,InterT,OutputT> |
This is a base class for Beam DoFn Transforms.
|
AdditionalOutputTagProperty |
Additional Output Tag Execution Property for edge that contains tag for additional outputs.
|
AggressiveSpeculativeCloningPass |
Speculative execution.
|
AllMetricServlet |
Servlet which handles total metric request.
|
AlternatingLeastSquare |
Sample Alternating Least Square application.
|
AlternatingLeastSquare.CalculateNextMatrix |
DoFn for calculating next matrix at each iteration.
|
AlternatingLeastSquare.CreateInitialMatrix |
A DoFn that creates an initial matrix.
|
AlternatingLeastSquare.ParseLine |
Method for parsing the input line.
|
AlternatingLeastSquare.TrainingDataCombiner |
Combiner for the training data.
|
AlternatingLeastSquare.UngroupSingleVectorList |
A DoFn that relays a single vector list.
|
AlternatingLeastSquare.UpdateUserAndItemMatrix |
Composite transform that wraps the transforms inside the loop.
|
AlternatingLeastSquareInefficient |
Sample Alternating Least Square application.
|
AlternatingLeastSquareInefficient.UpdateUserAndItemMatrix |
Composite transform that wraps the transforms inside the loop.
|
Annotates |
Annotation used to indicate which execution properties the class annotates.
|
AnnotatingPass |
A compile-time pass that annotates the IR DAG with execution properties.
|
AntiAffinitySchedulingConstraint |
Check if one of the tasks running on the executor, and the task to schedule are both in the anti-affinity group.
|
ArgBuilder |
Argument builder.
|
AssociatedProperty |
|
Backend<Plan> |
Interface for backend components.
|
BasicPullPolicy |
Basic pull policy.
|
BasicPushPolicy |
Basic push policy.
|
BatchScheduler |
(CONCURRENCY) Only a single dedicated thread should use the public methods of this class.
|
BatchSchedulerUtils |
Utlity methods regarding schedulers.
|
BeamBoundedSourceVertex<O> |
SourceVertex implementation for BoundedSource.
|
BeamDecoderFactory<T> |
|
BeamEncoderFactory<T> |
|
BeamStateTranslator |
|
BeamUnboundedSourceVertex<O,M extends org.apache.beam.sdk.io.UnboundedSource.CheckpointMark> |
SourceVertex implementation for UnboundedSource.
|
BeamWordCount |
An example that counts words in Shakespeare and includes Beam best practices.
|
BeamWordCount.CountWords |
A PTransform that converts a PCollection containing lines of text into a PCollection of
formatted word counts.
|
BeamWordCount.FormatAsTextFn |
A SimpleFunction that converts a Word and Count into a printable string.
|
BeamWordCount.WordCountOptions |
|
Block<K extends java.io.Serializable> |
This interface represents a block, which is the output of a specific task.
|
BlockFetchException |
BlockFetchException.
|
BlockFetchFailureProperty |
Decides how to react to a data block fetch failure.
|
BlockFetchFailureProperty.Value |
Possible values of DataFlowModel ExecutionProperty.
|
BlockInputReader |
Represents the input data transfer to a task.
|
BlockManagerMaster |
Master-side block manager.
|
BlockManagerMaster.BlockRequestHandler |
The handler of block location requests.
|
BlockManagerWorker |
Executor-side block manager.
|
BlockOutputWriter |
Represents the output data transfer from a task.
|
BlockState |
Represents the states of a whole data(a task output).
|
BlockState.State |
BlockState.
|
BlockStore |
Interface for Block placement.
|
BlockTransferThrottler |
A class to restrict parallel connection per runtime edge.
|
BlockWriteException |
BlockWriteException.
|
BoundedIteratorReadable<O> |
An abstract readable class that retrieves data from iterator.
|
Broadcast |
Sample Broadcast application.
|
BroadcastManagerMaster |
Broadcast variables saved in the master.
|
BroadcastManagerWorker |
Used by tasks to get/fetch (probably remote) broadcast variables.
|
ByteBufferInputStream |
This class is a customized input stream implementation which reads data from
list of ByteBuffer .
|
ByteInputContext |
Container for multiple input streams.
|
ByteOutputContext |
Container for multiple output streams.
|
BytesDecoderFactory |
|
BytesEncoderFactory |
|
ByteTransfer |
|
ByteTransferContext |
|
CachedSourceVertex<T> |
Bounded source vertex for cached data.
|
CacheIDProperty |
Cache ID ExecutionProperty.
|
ClientEndpoint |
A request endpoint in client side of a plan.
|
ClientRPC |
Driver-side RPC implementation for communication from/to Nemo Client.
|
ClientUtils |
Utility class for the Client.
|
Cloneable<T extends Cloneable<T>> |
This interface is implemented by objects that can be cloned.
|
ClonedSchedulingProperty |
Specifies cloned execution of a vertex.
|
ClonedSchedulingProperty.CloneConf |
Configurations for cloning.
|
ClosableBlockingQueue<T> |
A blocking queue implementation which is capable of closing.
|
CombineFnFinalTransform<K,A,O> |
Accumulates all of the partially accumulated KVs(Key, Accum) into KVs(Key, Output).
|
CombineFnPartialTransform<K,I,A> |
Partially accumulates the given KVs(Key, Input) into KVs(Key, Accum).
|
CommonEventHandler<T> |
Class for handling common events.
|
CommonSubexpressionEliminationPass |
Pass for Common Subexpression Elimination optimization.
|
CommunicationPatternProperty |
DataCommunicationPattern ExecutionProperty.
|
CommunicationPatternProperty.Value |
Possible values of DataCommunicationPattern ExecutionProperty.
|
CompilerEvent |
An empty interface to indicate events that are sent from Compiler.
|
CompilerEventHandler<T extends CompilerEvent> |
Class for handling events sent from Compiler.
|
CompilerTestUtil |
Utility methods for tests.
|
CompileTimeOptimizationException |
CompileTimeOptimizationException.
|
CompileTimePass |
Abstract class for compile-time optimization passes that processes the DAG.
|
CompositePass |
A compile-time pass composed of multiple compile-time passes, which each modifies an IR DAG.
|
CompressionPass |
A pass for applying compression algorithm for data flowing between vertices.
|
CompressionProperty |
Compression ExecutionProperty.
|
CompressionProperty.Value |
Possible values of Compression ExecutionProperty.
|
CompressionStreamChainer |
|
ConditionalLargeShufflePolicy |
A policy to demonstrate the large shuffle optimization, witch batches disk seek during data shuffle, conditionally.
|
ContainerException |
ContainerException.
|
ContainerManager |
(WARNING) This class is not thread-safe.
|
ContainerTypeAwareSchedulingConstraint |
This policy find executors which has corresponding container type.
|
ControlMessage |
|
ControlMessage.BlockLocationInfoMsg |
Protobuf type protobuf.BlockLocationInfoMsg
|
ControlMessage.BlockLocationInfoMsg.Builder |
Protobuf type protobuf.BlockLocationInfoMsg
|
ControlMessage.BlockLocationInfoMsgOrBuilder |
|
ControlMessage.BlockStateChangedMsg |
Protobuf type protobuf.BlockStateChangedMsg
|
ControlMessage.BlockStateChangedMsg.Builder |
Protobuf type protobuf.BlockStateChangedMsg
|
ControlMessage.BlockStateChangedMsgOrBuilder |
|
ControlMessage.BlockStateFromExecutor |
Protobuf enum protobuf.BlockStateFromExecutor
|
ControlMessage.BlockStore |
Protobuf enum protobuf.BlockStore
|
ControlMessage.BlockTransferContextDescriptor |
Protobuf type protobuf.BlockTransferContextDescriptor
|
ControlMessage.BlockTransferContextDescriptor.Builder |
Protobuf type protobuf.BlockTransferContextDescriptor
|
ControlMessage.BlockTransferContextDescriptorOrBuilder |
|
ControlMessage.ByteTransferContextSetupMessage |
Protobuf type protobuf.ByteTransferContextSetupMessage
|
ControlMessage.ByteTransferContextSetupMessage.Builder |
Protobuf type protobuf.ByteTransferContextSetupMessage
|
ControlMessage.ByteTransferContextSetupMessageOrBuilder |
|
ControlMessage.ByteTransferDataDirection |
Messages between Executors
|
ControlMessage.ClientToDriverMessage |
Protobuf type protobuf.ClientToDriverMessage
|
ControlMessage.ClientToDriverMessage.Builder |
Protobuf type protobuf.ClientToDriverMessage
|
ControlMessage.ClientToDriverMessageOrBuilder |
|
ControlMessage.ClientToDriverMessageType |
Messages from client to driver
|
ControlMessage.ContainerFailedMsg |
Protobuf type protobuf.ContainerFailedMsg
|
ControlMessage.ContainerFailedMsg.Builder |
Protobuf type protobuf.ContainerFailedMsg
|
ControlMessage.ContainerFailedMsgOrBuilder |
|
ControlMessage.DataCollectMessage |
Protobuf type protobuf.DataCollectMessage
|
ControlMessage.DataCollectMessage.Builder |
Protobuf type protobuf.DataCollectMessage
|
ControlMessage.DataCollectMessageOrBuilder |
|
ControlMessage.DriverToClientMessage |
Protobuf type protobuf.DriverToClientMessage
|
ControlMessage.DriverToClientMessage.Builder |
Protobuf type protobuf.DriverToClientMessage
|
ControlMessage.DriverToClientMessageOrBuilder |
|
ControlMessage.DriverToClientMessageType |
Messages from driver to client
|
ControlMessage.ExecutorFailedMsg |
Protobuf type protobuf.ExecutorFailedMsg
|
ControlMessage.ExecutorFailedMsg.Builder |
Protobuf type protobuf.ExecutorFailedMsg
|
ControlMessage.ExecutorFailedMsgOrBuilder |
|
ControlMessage.InMasterBroadcastVariableMessage |
Protobuf type protobuf.InMasterBroadcastVariableMessage
|
ControlMessage.InMasterBroadcastVariableMessage.Builder |
Protobuf type protobuf.InMasterBroadcastVariableMessage
|
ControlMessage.InMasterBroadcastVariableMessageOrBuilder |
|
ControlMessage.LaunchDAGMessage |
Protobuf type protobuf.LaunchDAGMessage
|
ControlMessage.LaunchDAGMessage.Builder |
Protobuf type protobuf.LaunchDAGMessage
|
ControlMessage.LaunchDAGMessageOrBuilder |
|
ControlMessage.Message |
Protobuf type protobuf.Message
|
ControlMessage.Message.Builder |
Protobuf type protobuf.Message
|
ControlMessage.MessageOrBuilder |
|
ControlMessage.MessageType |
Protobuf enum protobuf.MessageType
|
ControlMessage.Metric |
Common messages
|
ControlMessage.Metric.Builder |
Common messages
|
ControlMessage.MetricMsg |
Protobuf type protobuf.MetricMsg
|
ControlMessage.MetricMsg.Builder |
Protobuf type protobuf.MetricMsg
|
ControlMessage.MetricMsgOrBuilder |
|
ControlMessage.MetricOrBuilder |
|
ControlMessage.NotificationMessage |
Protobuf type protobuf.NotificationMessage
|
ControlMessage.NotificationMessage.Builder |
Protobuf type protobuf.NotificationMessage
|
ControlMessage.NotificationMessageOrBuilder |
|
ControlMessage.OptimizationType |
Protobuf enum protobuf.OptimizationType
|
ControlMessage.PipeInitMessage |
Protobuf type protobuf.PipeInitMessage
|
ControlMessage.PipeInitMessage.Builder |
Protobuf type protobuf.PipeInitMessage
|
ControlMessage.PipeInitMessageOrBuilder |
|
ControlMessage.PipeLocationInfoMessage |
Protobuf type protobuf.PipeLocationInfoMessage
|
ControlMessage.PipeLocationInfoMessage.Builder |
Protobuf type protobuf.PipeLocationInfoMessage
|
ControlMessage.PipeLocationInfoMessageOrBuilder |
|
ControlMessage.PipeTransferContextDescriptor |
Protobuf type protobuf.PipeTransferContextDescriptor
|
ControlMessage.PipeTransferContextDescriptor.Builder |
Protobuf type protobuf.PipeTransferContextDescriptor
|
ControlMessage.PipeTransferContextDescriptorOrBuilder |
|
ControlMessage.RecoverableFailureCause |
Protobuf enum protobuf.RecoverableFailureCause
|
ControlMessage.RequestBlockLocationMsg |
Protobuf type protobuf.RequestBlockLocationMsg
|
ControlMessage.RequestBlockLocationMsg.Builder |
Protobuf type protobuf.RequestBlockLocationMsg
|
ControlMessage.RequestBlockLocationMsgOrBuilder |
|
ControlMessage.RequestBroadcastVariableMessage |
Protobuf type protobuf.RequestBroadcastVariableMessage
|
ControlMessage.RequestBroadcastVariableMessage.Builder |
Protobuf type protobuf.RequestBroadcastVariableMessage
|
ControlMessage.RequestBroadcastVariableMessageOrBuilder |
|
ControlMessage.RequestPipeLocationMessage |
Protobuf type protobuf.RequestPipeLocationMessage
|
ControlMessage.RequestPipeLocationMessage.Builder |
Protobuf type protobuf.RequestPipeLocationMessage
|
ControlMessage.RequestPipeLocationMessageOrBuilder |
|
ControlMessage.RunTimePassMessageEntry |
Protobuf type protobuf.RunTimePassMessageEntry
|
ControlMessage.RunTimePassMessageEntry.Builder |
Protobuf type protobuf.RunTimePassMessageEntry
|
ControlMessage.RunTimePassMessageEntryOrBuilder |
|
ControlMessage.RunTimePassMessageMsg |
Protobuf type protobuf.RunTimePassMessageMsg
|
ControlMessage.RunTimePassMessageMsg.Builder |
Protobuf type protobuf.RunTimePassMessageMsg
|
ControlMessage.RunTimePassMessageMsgOrBuilder |
|
ControlMessage.ScheduleTaskMsg |
Messages from Master to Executors
|
ControlMessage.ScheduleTaskMsg.Builder |
Messages from Master to Executors
|
ControlMessage.ScheduleTaskMsgOrBuilder |
|
ControlMessage.TaskStateChangedMsg |
Messages from Executors to Master
|
ControlMessage.TaskStateChangedMsg.Builder |
Messages from Executors to Master
|
ControlMessage.TaskStateChangedMsgOrBuilder |
|
ControlMessage.TaskStateFromExecutor |
Protobuf enum protobuf.TaskStateFromExecutor
|
CreateViewTransform<I,O> |
This transforms emits materialized data for each window.
|
CreateViewTransform.MultiView<T> |
Represents PrimitiveViewT supplied to the ViewFn .
|
DAG<V extends Vertex,E extends Edge<V>> |
DAG implementation.
|
DAGBuilder<V extends Vertex,E extends Edge<V>> |
DAG Builder.
|
DAGInterface<V extends Vertex,E extends Edge<V>> |
Methods for querying a directed-acyclic graph (DAG).
|
DAGInterface.TraversalOrder |
Indicates the traversal order of this DAG.
|
DataFetcherOutputCollector<O> |
This collector receives data from DataFetcher and forwards it to the next operator.
|
DataFlowProperty |
DataFlowModel ExecutionProperty.
|
DataFlowProperty.Value |
Possible values of DataFlowModel ExecutionProperty.
|
DataPersistenceProperty |
Data persistence ExecutionProperty.
|
DataPersistenceProperty.Value |
Possible options for the data persistence strategy.
|
DataPlaneConf |
Data plane Configuration for Executors.
|
DataSkewPolicy |
A policy to perform data skew dynamic optimization.
|
DataSourceException |
DataSourceException
Thrown when any exception occurs in data sources.
|
DataStoreProperty |
DataStore ExecutionProperty.
|
DataStoreProperty.Value |
Possible values of DataStore ExecutionProperty.
|
DataTransferEvent |
Event for data transfer, such as data read or write.
|
DataTransferEvent.TransferType |
Enum of transfer types.
|
DataUtil |
Utility methods for data handling (e.g., (de)serialization).
|
DataUtil.InputStreamIterator<T> |
An iterator that emits objects from InputStream using the corresponding DecoderFactory .
|
DataUtil.IteratorWithNumBytes<T> |
Iterator with interface to access to the number of bytes.
|
DataUtil.IteratorWithNumBytes.NumBytesNotSupportedException |
|
DecoderFactory<T> |
A decoder factory object which generates decoders that decode byte streams into values of type T .
|
DecoderFactory.Decoder<T> |
Interface of Decoder.
|
DecoderFactory.DummyDecoderFactory |
Dummy coder factory implementation which is not supposed to be used.
|
DecoderProperty |
Decoder ExecutionProperty.
|
DecodeStreamChainer |
|
DecompressionProperty |
Decompression ExecutionProperty.
|
DecompressionStreamChainer |
|
DedicatedKeyPerElement |
Declares that all of the designated keys for each element in a Partitioner is dedicated for the element.
|
DedicatedKeyPerElementPartitioner |
An implementation of Partitioner which assigns a dedicated key per an output data from a task.
|
DefaultCompositePass |
A series of primitive passes that is applied commonly to all policies.
|
DefaultDataPersistencePass |
Pass for initiating IREdge data persistence ExecutionProperty with default values.
|
DefaultDataStorePass |
Edge data store pass to process inter-stage memory store edges.
|
DefaultEdgeDecoderPass |
Pass for initiating IREdge Decoder ExecutionProperty with default dummy coder.
|
DefaultEdgeEncoderPass |
Pass for initiating IREdge Encoder ExecutionProperty with default dummy coder.
|
DefaultExecutorRepresenter |
(WARNING) This class is not thread-safe, and thus should only be accessed through ExecutorRegistry.
|
DefaultOutputManager<OutputT> |
Default output emitter that uses outputCollector.
|
DefaultParallelismPass |
Optimization pass for tagging parallelism execution property.
|
DefaultPolicy |
A basic default policy, that performs the minimum amount of optimization to be done to a specific DAG.
|
DefaultPolicyWithSeparatePass |
A simple example policy to demonstrate a policy with a separate, refactored pass.
|
DefaultPolicyWithSeparatePass.RefactoredPass |
A simple custom pass consisted of the two passes at the end of the default pass.
|
DefaultScheduleGroupPass |
A pass for assigning each stages in schedule groups.
|
DirectByteBufferOutputStream |
This class is a customized output stream implementation backed by
ByteBuffer , which utilizes off heap memory when writing the data via MemoryPoolAssigner.
|
DisaggregationEdgeDataStorePass |
A pass to support Disaggregated Resources by tagging edges.
|
DisaggregationPolicy |
A policy to demonstrate the disaggregation optimization, that uses GlusterFS as file storage.
|
DoFnTransform<InputT,OutputT> |
DoFn transform implementation when there is no side input.
|
DriverEndpoint |
A request endpoint in driver side of a plan.
|
DriverRPCServer |
Client-side RPC implementation for communication from/to Nemo Driver.
|
DuplicateEdgeGroupProperty |
Invariant data ExecutionProperty.
|
DuplicateEdgeGroupPropertyValue |
Value of DuplicateEdgeGroupProperty.
|
DuplicateEdgeGroupSizePass |
A pass for annotate duplicate data for each edge.
|
DynamicOptimizationException |
DynamicOptimizationException.
|
DynamicTaskSizingPolicy |
A policy to dynamically resize tasks in runtime.
|
DynamicTaskSizingRuntimePass |
Runtime pass for Dynamic Task Sizing policy.
|
EDGARAvgDocSize |
Application for EDGAR dataset.
|
EDGARDocumentSuccessRate |
Application for EDGAR dataset.
|
EDGARRequestsByCIK |
Application for EDGAR dataset.
|
EDGARTop10BadRefererDocs |
Application for EDGAR dataset.
|
EDGARTop10BadRefererDocs.ValueComparator<K> |
Value comparator comparing the double value.
|
EDGARTop10Documents |
Application for EDGAR dataset.
|
EDGARTop10Documents.ValueComparator<K> |
Value comparator comparing the long value.
|
Edge<V extends Vertex> |
Connects two vertices of a DAG.
|
EdgeExecutionProperty<T extends java.io.Serializable> |
|
EmptyComponents |
Empty components to mock transform and source, for tests and examples.
|
EmptyComponents.EmptySourceVertex<T> |
An empty Source Vertex.
|
EmptyComponents.EmptyTransform<I,O> |
An empty transform.
|
EnableDynamicTaskSizingProperty |
Property to enable dynamic task sizing policy.
|
EncoderFactory<T> |
A encoder factory object which generates encoders that encode values of type T into byte streams.
|
EncoderFactory.DummyEncoderFactory |
Dummy encoder factory implementation which is not supposed to be used.
|
EncoderFactory.Encoder<T> |
Interface of Encoder.
|
EncoderProperty |
Encoder ExecutionProperty.
|
EncodeStreamChainer |
|
Event |
Class for all generic event that contains timestamp at the moment.
|
ExampleTestArgs |
Test Arguments for Examples.
|
ExampleTestUtil |
Test Utils for Examples.
|
ExecutionProperty<T extends java.io.Serializable> |
An abstract class for each execution factors.
|
ExecutionPropertyMap<T extends ExecutionProperty> |
ExecutionPropertyMap Class, which uses HashMap for keeping track of ExecutionProperties for vertices and edges.
|
Executor |
Executor.
|
ExecutorRegistry |
(WARNING) This class must be thread-safe.
|
ExecutorRepresenter |
Contains information/state regarding an executor
Such information may include
a) The executor's resource type.
|
FailedMessageSender |
A message sender that failed.
|
FileArea |
A file area descriptor.
|
FileBlock<K extends java.io.Serializable> |
This class represents a block which is stored in (local or remote) file.
|
FileMetadata<K extends java.io.Serializable> |
This class represents a metadata for a Block .
|
FinalCombineFn<AccumT,Output> |
Wrapper class for Combine.CombineFn .
|
Finishmark |
Finish mark that notifies the data fetching is finished.
|
FlattenTransform<T> |
Flatten transform implementation.
|
FloatArrayCoder |
EncoderFactory for float[].
|
FreeSlotSchedulingConstraint |
This policy finds executor that has free slot for a Task.
|
GBKTransform<K,InputT,OutputT> |
This transform executes GroupByKey transformation and CombinePerKey transformation when input data is unbounded
or is not in a global window.
|
GlusterFileStore |
Stores blocks in a mounted GlusterFS volume.
|
GroupByKeyTransform<I> |
Group Beam KVs.
|
GrpcMessageEnvironment |
This class is installed when every node, containing both driver and evaluator, has been started, and setup
grpc environment to implement RPC semantics defined in org.apache.nemo.runtime.common.message package.
|
GrpcMessageService |
|
GrpcMessageService.Void |
Protobuf type protobuf.Void
|
GrpcMessageService.Void.Builder |
Protobuf type protobuf.Void
|
GrpcMessageService.VoidOrBuilder |
|
HashPartitioner |
An implementation of Partitioner which hashes output data from a source task
according to the key of elements.
|
HashRange |
Descriptor for hash range.
|
IdManager |
ID manager.
|
IgnoreSchedulingTempDataReceiverProperty |
A property represents that a vertex annotated with this property doesn't process any data and
should be regarded as a kind of "marker" to construct a temporary edge that contains some data that
have to be written before it's usage is not determined (e.g., for caching).
|
IllegalEdgeOperationException |
IllegalEdgeOperationException.
|
IllegalMessageException |
IllegalMessageException.
|
IllegalStateTransitionException |
IllegalStateTransitionException.
|
IllegalVertexOperationException |
IllegalVertexOperationException.
|
InMemorySideInputReader |
Accumulates and provides side inputs in memory.
|
InMemorySourceVertex<T> |
Source vertex with the data in memory.
|
InMemoryStateInternalsFactory<K> |
InMemoryStateInternalsFactory.
|
InMemoryTimerInternalsFactory<K> |
InMemoryTimerInternalsFactory.
|
InputReader |
Represents the input data transfer to a task.
|
InputWatermarkManager |
An interface for tracking input watermarks among multiple input streams.
|
IntactPartitioner |
An implementation of Partitioner which makes an output data
from a source task to a single partition.
|
IntArrayCoder |
EncoderFactory for int[].
|
IntDecoderFactory |
|
IntEncoderFactory |
|
IntermediateDataIOFactory |
|
InvalidParameterException |
InvalidParameterException.
|
InvalidUserMainException |
InvalidUserMainException.
|
IRDAG |
An IRDAG object captures a high-level data processing application (e.g., Spark/Beam application).
|
IRDAGChecker |
Checks the integrity of an IR DAG.
|
IRDAGChecker.GlobalDAGChecker |
Checks the entire DAG.
|
IREdge |
Physical execution plan of intermediate data movement.
|
IRVertex |
The basic unit of operation in a dataflow program, as well as the most important data structure in Nemo.
|
JavaMapReduce |
Java MapReduce example.
|
JavaSparkPi |
Computes an approximation to pi.
|
JavaSparkSQLExample |
Java Spark SQL Example program.
|
JavaSparkSQLExample.Person |
Simple person class.
|
JavaUserDefinedTypedAggregation |
Java SparkSQL example: User-defined Typed Aggregation.
|
JavaUserDefinedTypedAggregation.Average |
Average class.
|
JavaUserDefinedTypedAggregation.Employee |
Employee class.
|
JavaUserDefinedTypedAggregation.MyAverage |
MyAverage class.
|
JavaUserDefinedUntypedAggregation |
Java SparkSQL example: User-defined Untyped Aggregation.
|
JavaUserDefinedUntypedAggregation.MyAverage |
MyAverage class.
|
JavaWordAndLineCount |
Java Spark word-count and line-count examples in one.
|
JavaWordCount |
Java Wordcount example.
|
JobConf |
Job Configurations.
|
JobLauncher |
Job launcher.
|
JobMetric |
|
JobMetricServlet |
Servlet which handles JobMetric metric request.
|
JsonParseException |
JsonParseException.
|
KeyDecoderProperty |
KeyDecoder ExecutionProperty.
|
KeyEncoderProperty |
KeyEncoder ExecutionProperty.
|
KeyExtractor |
Extracts a key from an element.
|
KeyExtractorProperty |
KeyExtractor ExecutionProperty.
|
KeyRange<K extends java.io.Serializable> |
Represents the key range of data partitions within a block.
|
LambdaPass |
Lambda Pass.
|
LambdaPolicy |
Lambda Policy
Description: A part of lambda executor, to support Lambda policy.
|
LargeShuffleAnnotatingPass |
This pass assumes that a RelayVertex was previously inserted to receive each shuffle edge.
|
LargeShuffleCompositePass |
A series of passes to optimize large shuffle with disk seek batching techniques.
|
LargeShufflePolicy |
A policy to demonstrate the large shuffle optimization, witch batches disk seek during data shuffle.
|
LargeShuffleReshapingPass |
Inserts the RelayVertex for each shuffle edge.
|
LatencyMark |
Latency mark is a watermark with the data related to stream data latencies.
|
LatencymarkEmitTransform<I,O> |
|
LatencyMetric |
Metric class for recording latencymark and the time when the latencymark is recorded.
|
LimitedInputStream |
This class provide functionality to limit bytes read from InputStream .
|
LocalBlockStore |
This abstract class represents BlockStore
which contains the (meta)data of the Block s in local.
|
LocalFileMetadata<K extends java.io.Serializable> |
This class represents a metadata for a local file Block .
|
LocalFileStore |
Stores blocks in local files.
|
LocalInputContext |
This class provides a data transfer interface to the receiver side when both the sender and the receiver are in the
same executor.
|
LocalitySchedulingConstraint |
This policy tries to pick the executors where the corresponding source or intermediate data for a task reside.
|
LocalMessageDispatcher |
Dispatch messages on a single machine.
|
LocalMessageEnvironment |
|
LocalMessageSender<T> |
A simple MessageSender implementation that works on a single node.
|
LocalOutputContext |
This class provides a data transfer interface to the sender side when both the sender and the receiver are
in the same executor.
|
LocalTransferContext |
This class represents the data transfer context when both the sender and the receiver are in the same executor.
|
LongDecoderFactory |
|
LongEncoderFactory |
|
LoopCompositeTransform<inputT extends org.apache.beam.sdk.values.PInput,outputT extends org.apache.beam.sdk.values.POutput> |
A composite transform for wrapping transforms inside a loop to create loop operators in the IR.
|
LoopExtractionPass |
Pass for extracting and grouping each loops together using the LoopVertex.
|
LoopOptimizationCompositePass |
A series of passes to perform LoopOptimization.
|
LoopOptimizations |
Loop Optimization.
|
LoopOptimizations.LoopFusionPass |
Pass for Loop Fusion optimization.
|
LoopOptimizations.LoopInvariantCodeMotionPass |
Pass for Loop Invariant Code Motion optimization.
|
LoopUnrollingPass |
|
LoopVertex |
IRVertex that contains a partial DAG that is iterative.
|
MapReduceDisaggregationOptimization |
A sample MapReduceDisaggregationOptimization application.
|
MemoryAllocationException |
An exception to be thrown when a memory allocation operation is not successful.
|
MemoryChunk |
This class represents chunk of memory residing in off-heap region
managed by MemoryPoolAssigner , which is backed by ByteBuffer .
|
MemoryPoolAssigner |
The MemoryPoolAssigner assigns the memory that Nemo uses for writing data blocks from the MemoryPoolAssigner.MemoryPool .
|
MemoryStore |
Store data in local memory.
|
Message<T> |
|
MessageAggregatorTransform<K,V,O> |
|
MessageAggregatorVertex<K,V,O> |
Aggregates upstream messages.
|
MessageAggregatorVertex.InitialStateSupplier<O> |
Creates the initial aggregated message.
|
MessageAggregatorVertex.MessageAggregatorFunction<K,V,O> |
Aggregates incoming messages.
|
MessageContext |
|
MessageEnvironment |
Set up MessageListener s to handle incoming messages on this node, and connect to remote nodes and return
MessageSender s to send message to them.
|
MessageGeneratorTransform<I,K,V> |
A Transform for the message generator vertex.
|
MessageGeneratorVertex<I,K,V> |
Produces a message for run-time pass.
|
MessageGeneratorVertex.MessageGeneratorFunction<I,K,V> |
Applied on the input data elements to produce a message.
|
MessageIdEdgeProperty |
Vertices and edges with the same MessageId are subject to the same run-time optimization.
|
MessageIdVertexProperty |
Vertices and edges with the same MessageId belong to the same run-time optimization.
|
MessageListener<T> |
|
MessageParameters |
Parameters for message components.
|
MessageParameters.SenderId |
Id of the sender.
|
MessageSender<T> |
|
MessageServiceGrpc |
A grpc service for implementing org.apache.nemo.runtime.common.message components using grpc
|
MessageServiceGrpc.MessageServiceBlockingStub |
A grpc service for implementing org.apache.nemo.runtime.common.message components using grpc
|
MessageServiceGrpc.MessageServiceFutureStub |
A grpc service for implementing org.apache.nemo.runtime.common.message components using grpc
|
MessageServiceGrpc.MessageServiceImplBase |
A grpc service for implementing org.apache.nemo.runtime.common.message components using grpc
|
MessageServiceGrpc.MessageServiceStub |
A grpc service for implementing org.apache.nemo.runtime.common.message components using grpc
|
MessageUtils |
Utility class for messages.
|
Metric |
Interface for all metrics.
|
MetricBroadcaster |
MetricBroadcaster broadcast metric changes to the currently active WebSocket sessions.
|
MetricException |
MetricException.
|
MetricManagerMaster |
A default metric message handler.
|
MetricManagerWorker |
Metric sender that periodically flushes the collected metrics to Driver.
|
MetricMessageHandler |
Metric message handler.
|
MetricMessageSender |
Interface for metric sender.
|
MetricStore |
MetricStore stores metric data which will be used by web visualize interface, logging, and so on.
|
MetricUtils |
Utility class for metrics.
|
MinimalWordCount |
MinimalWordCount program from BEAM.
|
MinOccupancyFirstSchedulingPolicy |
This policy chooses a set of Executors, on which have minimum running Tasks.
|
MultiInputWatermarkManager |
This tracks the minimum input watermark among multiple input streams.
|
MultinomialLogisticRegression |
Sample Multinomial Logistic Regression application.
|
MultinomialLogisticRegression.ApplyGradient |
DoFn class that applies the gradient to the model.
|
MultinomialLogisticRegression.CalculateGradient |
Calculate Gradient.
|
MultinomialLogisticRegression.CombineFunction |
Combine Function for two Double arrays.
|
MultinomialLogisticRegression.CombineFunctionForIterable |
Combine Function for Iterable of gradients.
|
MultinomialLogisticRegression.UpdateModel |
+ Composite transform that wraps the transforms inside the loop.
|
NativeChannelImplementationSelector |
|
NcsMessageEnvironment |
Message environment for NCS.
|
NemoBackend |
Backend component for Nemo Runtime.
|
NemoClient |
A wrapper class that contains client handlers.
|
NemoContext |
REEF Context for the Executor.
|
NemoDriver |
REEF Driver for Nemo.
|
NemoEventDecoderFactory |
A factory for NemoEventDecoder.
|
NemoEventEncoderFactory |
A factory for NemoEventEncoder.
|
NemoOptimizer |
An interface for optimizer, which manages the optimization over submitted IR DAGs through Policy s.
|
NemoPipelineOptions |
NemoPipelineOptions.
|
NemoPipelineResult |
Beam result.
|
NemoPlanRewriter |
Rewrites the physical plan during execution, to enforce the optimizations of Nemo RunTimePasses.
|
NemoRunner |
Runner class for BEAM programs.
|
NemoRunnerRegistrar |
Contains the PipelineRunnerRegistrar and PipelineOptionsRegistrar for the NemoRunner .
|
NemoRunnerRegistrar.Options |
|
NemoRunnerRegistrar.Runner |
|
NettyChannelImplementationSelector |
Selects appropriate Channel implementation.
|
NetworkTraceAnalysis |
An app that analyzes data flow from network trace.
|
NextIntraTaskOperatorInfo |
Contains information for next operator:
-- edgeIndex: the index of edge to next operator.
|
NioChannelImplementationSelector |
|
NodeConnectionException |
NodeConnectionException.
|
NodeShareSchedulingConstraint |
|
NonSerializedMemoryBlock<K extends java.io.Serializable> |
This class represents a block which is stored in local memory and not serialized.
|
NonSerializedPartition<K> |
A collection of data elements.
|
NoWatermarkEmitTransform<I,O> |
This transform does not emit watermarks.
|
ObjectCoderForString |
This coder acts like an object coder to conform the type, but is actually just a StringUtf8Coder .
|
OperatorVertex |
IRVertex that transforms input data.
|
OperatorVertexOutputCollector<O> |
OutputCollector implementation.
|
OperatorWatermarkCollector |
This class is used for collecting watermarks for an OperatorVertex.
|
Optimizer |
An interface for optimizer, which manages the optimization over submitted IR DAGs through
Policy s.
|
OptimizerUtils |
Utility class for optimizer.
|
OutputCollector<O> |
Interface through which Transform emits outputs.
|
OutputContext |
Represents the output context during the data transfer between two tasks.
|
OutputMismatchException |
OutputMismatchException.
|
OutputWriter |
Represents the output data transfer from a task.
|
Pair<A,B> |
Pair class.
|
PairDecoderFactory<A extends java.io.Serializable,B extends java.io.Serializable> |
An DecoderFactory for Pair .
|
PairEncoderFactory<A extends java.io.Serializable,B extends java.io.Serializable> |
An EncoderFactory for Pair .
|
PairKeyExtractor |
A KeyExtractor for Pair class.
|
ParallelismProperty |
This property decides the number of parallel tasks to use for executing the corresponding IRVertex.
|
ParallelismProphet |
A prophet for Parallelism.
|
PartialCombineFn<InputT,AccumT> |
Wrapper class for Combine.CombineFn .
|
Partition<T,K> |
A collection of data elements.
|
Partitioner<K extends java.io.Serializable> |
This interface represents the way of partitioning output data from a source task.
|
PartitionerProperty |
Partitioner ExecutionProperty.
|
PartitionerProperty.Type |
Partitioning types.
|
PartitionMetadata<K extends java.io.Serializable> |
This class represents a metadata for a partition.
|
PartitionSetProperty |
This property decides which partitions the tasks of the destination IRVertex should fetch.
|
PartitionWordsByLength |
Partition words by length example.
|
Pass |
Abstract class for optimization passes.
|
PendingTaskCollectionPointer |
Points to a collection of pending tasks eligible for scheduling.
|
PerKeyMedian |
IP stat analysis example used in skew experiment.
|
PerPercentileAverage |
Per percentile statistics application.
|
PersistentConnectionToMasterMap |
Persistent connection for sending messages to master.
|
PhysicalPlan |
A job's physical plan, to be executed by the Runtime.
|
PhysicalPlanGenerationException |
PhysicalPlanGenerationException.
|
PhysicalPlanGenerator |
A function that converts an IR DAG to physical DAG.
|
PipeContainer |
|
PipeInputReader |
Represents the input data transfer to a task.
|
PipelineVisitor |
Uses the translator and the context to build a Nemo IR DAG.
|
PipeManagerMaster |
Master-side pipe manager.
|
PipeManagerWorker |
Two threads use this class
- Network thread: Saves pipe connections created from destination tasks.
|
PipeOutputWriter |
Represents the output data transfer from a task.
|
PipeTransferForAllEdgesPass |
Annotate 'Pipe' on all edges.
|
PlanAppender |
An implementation of plan appender which append two plans regarding to caching.
|
PlanAppenderException |
An exception which represents exception during appending plans.
|
PlanRewriter |
PhysicalPlan rewriter.
|
PlanState |
Represents the states and their transitions of a physical plan.
|
PlanState.State |
PlanState.
|
PlanStateManager |
Maintains three levels of state machines (PlanState, StageState, and TaskState) of a physical plan.
|
Policy |
An interface for policies, each of which is composed of a list of static optimization passes.
|
PolicyBuilder |
A builder for policies.
|
PolicyImpl |
Implementation of the Policy interface.
|
Prophet<K,V> |
A prophet class for dynamic optimization.
|
PubSubEventHandlerWrapper |
Wrapper class with a pub/sub event handler.
|
PushBackDoFnTransform<InputT,OutputT> |
DoFn transform implementation with push backs for side inputs.
|
Readable<O> |
Interface for readable.
|
RelayVertex |
Relays input data from upstream vertex to downstream vertex promptly.
|
RemoteClientMessageLoggingHandler |
A simple logging handler that forwards messages to REEF client.
|
RemoteFileMetadata<K extends java.io.Serializable> |
This class represents a metadata for a remote file block.
|
RemoteFileStore |
Interface for remote block stores (e.g., GlusterFS, ...).
|
ReplyFutureMap<T> |
Orchestrate message sender and receiver using CompletableFuture for asynchronous request-reply communication.
|
Requires |
Annotation used to indicate which execution properties the class requires as prerequisites.
|
ReshapingPass |
A compile-time pass that reshapes the structure of the IR DAG.
|
ResourceAntiAffinityProperty |
Indices of tasks that must not concurrently run on the same executor.
|
ResourceLambdaProperty |
Resource property supporting Lambda Pass.
|
ResourceLambdaProperty.Value |
Possible values of DataStore ExecutionProperty.
|
ResourceLocalityPass |
|
ResourceLocalityProperty |
This property decides whether or not to schedule this vertex only on executors where
source (including intermediate) data reside.
|
ResourcePriorityProperty |
ExecutionPlacement ExecutionProperty.
|
ResourceSitePass |
Computes and assigns appropriate share of nodes to each irVertex to minimize shuffle time,
with respect to bandwidth restrictions of nodes.
|
ResourceSiteProperty |
Map between node name and the number of parallelism which will run on the node.
|
ResourceSlotPass |
|
ResourceSlotProperty |
This property decides whether or not to comply to slot restrictions when scheduling this vertex.
|
ResourceSpecification |
Represents the specifications of a resource.
|
RuntimeEdge<V extends Vertex> |
Represents the edge between vertices in a logical/physical plan in runtime.
|
RuntimeEvent |
An empty interface to indicate events that are sent from Runtime.
|
RuntimeEventHandler<T extends RuntimeEvent> |
Class for handling events sent from Runtime.
|
RuntimeIdManager |
ID Generator.
|
RuntimeMaster |
(WARNING) Use runtimeMasterThread for all public methods to avoid race conditions.
|
RunTimeMessageOutputCollector<O> |
OutputCollector for dynamic optimization data.
|
RuntimeOptimizationException |
CompileTimeOptimizationException.
|
RunTimePass<T> |
Abstract class for dynamic optimization passes, for dynamically optimizing the IRDAG.
|
RuntimeTestUtil |
Utility class for runtime unit tests.
|
SamplingLargeShuffleSkewPolicy |
A policy to demonstrate the large shuffle optimization, witch batches disk seek during data shuffle.
|
SamplingSkewReshapingPass |
Optimizes the PartitionSet property of shuffle edges to handle data skews using the SamplingVertex.
|
SamplingTaskSizingPass |
Compiler pass for dynamic task size optimization.
|
SamplingVertex |
Executes the original IRVertex using a subset of input data partitions.
|
ScheduleGroupProperty |
ScheduleGroup ExecutionProperty.
|
Scheduler |
Only two threads call scheduling code: RuntimeMaster thread (RMT), and SchedulerThread(ST).
|
SchedulingConstraint |
Functions to test schedulability with a pair of an executor and a task.
|
SchedulingConstraintRegistry |
|
SchedulingException |
SchedulingException.
|
SchedulingPolicy |
A function to select an executor from collection of available executors.
|
SerializedMemoryBlock<K extends java.io.Serializable> |
This class represents a block which is serialized and stored in local memory.
|
SerializedMemoryStore |
Serialize and store data in local memory.
|
SerializedPartition<K> |
A collection of data elements.
|
Serializer<E,D> |
|
SerializerManager |
|
ShuffleEdgePushPass |
A pass for tagging shuffle edges different from the default ones.
|
SideInputCoder<T> |
EncoderFactory for side inputs.
|
SideInputElement<T> |
|
SideInputTransform<T> |
Side input transform implementation.
|
SignalTransform |
This class evokes run-time pass when there's no need to transfer any run-time information.
|
SignalVertex |
Signal vertex holding signal transform.
|
SimpleSumSQL |
A simple SQL application.
|
SimulatedTaskExecutor |
Class for simulated task execution.
|
SimulationException |
SimulationException.
|
SimulationScheduler |
Scheduler for simulating an execution not controlled by the runtime master.
|
SingleInputWatermarkManager |
This is a special implementation for single input data stream for optimization.
|
SkewAnnotatingPass |
For each shuffle edge, set the number of partitions to (dstParallelism * HASH_RANGE_MULTIPLIER).
|
SkewCompositePass |
Pass to modify the DAG for a job to perform data skew.
|
SkewProphet |
Prophet class for skew handling.
|
SkewReshapingPass |
Pass to reshape the IR DAG for skew handling.
|
SkewRunTimePass |
Dynamic optimization pass for handling data skew.
|
SourceVertex<O> |
IRVertex that reads data from an external source.
|
SparkBroadcastVariables |
Broadcast variables of Spark.
|
SparkKeyExtractor |
Extracts the key from a KV element.
|
Stage |
Stage.
|
StageEdge |
Edge of a stage that connects an IRVertex of the source stage to an IRVertex of the destination stage.
|
StageMetric |
|
StageMetricServlet |
|
StagePartitioner |
A function that is responsible for stage partitioning on IR DAG.
|
StageState |
Represents the states and their transitions of a stage.
|
StageState.State |
StageState.
|
StateMachine |
A finite state machine that can be created with user defined states and transitions.
|
StateMachine.Builder |
Builder that builds a StateMachine.
|
StateMetric<T extends java.io.Serializable> |
Interface for metric which contians its state.
|
StateTransitionEvent<T extends java.io.Serializable> |
Event of state transition.
|
StateTranslator |
A class provides the translation between the state of plan and corresponding ClientEndpoint .
|
StreamingPolicy |
Streaming policy.
|
StreamingScheduler |
A simple scheduler for streaming workloads.
|
StreamMetric |
Metrics associated with stream.
|
StreamTransform<T> |
A Transform relays input data from upstream vertex to downstream vertex promptly.
|
SubPartitionSetProperty |
This property decides the hash partition set of sampled and optimized tasks in Dynamic Task Sizing Policy.
|
Task |
A Task (attempt) is a self-contained executable that can be executed on a machine.
|
TaskExecutor |
Executes a task.
|
TaskMetric |
|
TaskMetricServlet |
|
TaskSizeSplitterVertex |
This vertex works as a partition-based sampling vertex of dynamic task sizing pass.
|
TaskState |
Represents the states and their transitions of a task.
|
TaskState.RecoverableTaskFailureCause |
Causes of a recoverable failure.
|
TaskState.State |
TaskState.
|
TaskStateManager |
Manages the states related to a task.
|
TestPlanGenerator |
Generates physical plans for testing purposes.
|
TestPlanGenerator.PlanType |
Type of the plan to generate.
|
TestPolicy |
A policy for tests.
|
TransferOutputStream |
Represents the output stream to which the sender sends its data during the data transfer.
|
Transform<I,O> |
Interface for specifying 'What' to do with data.
|
Transform.Context |
Context of the transform.
|
TransformContextImpl |
Transform Context Implementation.
|
TransientResourceCompositePass |
A series of passes to harness transient resources.
|
TransientResourceDataTransferPass |
Optimizes IREdges between transient resources and reserved resources.
|
TransientResourcePolicy |
A policy to perform optimization that uses transient resources in data centers.
|
TransientResourcePriorityPass |
Place valuable computations on reserved resources, and the rest on transient resources.
|
UnknownExecutionStateException |
UnknownExecutionStateException.
|
UnknownFailureCauseException |
UnknownFailureCauseException.
|
UnrecoverableFailureException |
UnrecoverableFailureException.
|
UnsupportedBlockStoreException |
UnsupportedBlockStoreException.
|
UnsupportedCommPatternException |
UnsupportedCommPatternException.
|
UnsupportedCompressionException |
UnsupportedCompressionException
Thrown when the compression method is not supported.
|
UnsupportedExecutionPropertyException |
UnsupportedExecutionPropertyException.
|
UnsupportedMethodException |
UnsupportedMethodException.
|
UnsupportedMetricException |
UnsupportedMetricException.
|
UnsupportedPartitionerException |
UnsupportedPartitionerException.
|
UpfrontCloningPass |
Set the ClonedScheduling property of source vertices, in an upfront manner.
|
UserApplicationRunner |
Compiles and runs User application.
|
Util |
Class to hold the utility methods.
|
Vertex |
A vertex in DAG.
|
VertexExecutionProperty<T extends java.io.Serializable> |
|
Watermark |
Watermark event.
|
WatermarkWithIndex |
This contains a watermark and the src task index.
|
WebSocketMetricAdapter |
Adapter for WebSocket metric request and response.
|
WebSocketMetricServlet |
Servlet which handles WebSocket HTTP request.
|
WindowedBroadcast |
A Windowed WordCount application.
|
WindowedWordCount |
A Windowed WordCount application.
|
WindowFnTransform<T,W extends org.apache.beam.sdk.transforms.windowing.BoundedWindow> |
Windowing transform implementation.
|
WordCount |
WordCount application.
|
WordCountTimeOut1Sec |
WordCount application, but with a timeout of 1 second.
|
WriteOneFilePerWindow |
This class is brought from beam/examples/common/WriteOneFilePerWindow.java.
|
WriteOneFilePerWindow.PerWindowFiles |
A FileBasedSink.FilenamePolicy produces a base file name for a write based on metadata about the data
being written.
|
XGBoostPass |
Pass for applying XGBoost optimizations.
|
XGBoostPolicy |
A policy that enforces values retrieved by an optimization by XGBoost.
|