Package org.apache.nemo.runtime.executor.data

Class DataUtil

java.lang.Object

org.apache.nemo.runtime.executor.data.DataUtil

public final class DataUtil
extends java.lang.Object

Utility methods for data handling (e.g., (de)serialization).

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	DataUtil.InputStreamIterator<T>	An iterator that emits objects from InputStream using the corresponding DecoderFactory.
static interface	DataUtil.IteratorWithNumBytes<T>	Iterator with interface to access to the number of bytes.

Method Summary

Modifier and Type	Method	Description
static java.lang.String	blockIdToFilePath(java.lang.String blockId, java.lang.String fileDirectory)	Converts a block id to the corresponding file path.
static java.lang.String	blockIdToMetaFilePath(java.lang.String blockId, java.lang.String fileDirectory)	Converts a block id to the corresponding metadata file path.
static java.io.InputStream	buildInputStream(java.io.InputStream in, java.util.List<DecodeStreamChainer> decodeStreamChainers)	Chain InputStream with DecodeStreamChainers.
static java.io.OutputStream	buildOutputStream(java.io.OutputStream out, java.util.List<EncodeStreamChainer> encodeStreamChainers)	Chain OutputStream with EncodeStreamChainers.
static java.lang.Iterable	concatNonSerPartitions(java.lang.Iterable<NonSerializedPartition> partitionsToConcat)	Concatenates an iterable of non-serialized Partitions into a single iterable of elements.
static <K extends java.io.Serializable> java.lang.Iterable<NonSerializedPartition<K>>	convertToNonSerPartitions(Serializer serializer, java.lang.Iterable<SerializedPartition<K>> partitionsToConvert)	Converts the serialized Partitions in an iterable to non-serialized partitions.
static <K extends java.io.Serializable> java.lang.Iterable<SerializedPartition<K>>	convertToSerPartitions(Serializer serializer, java.lang.Iterable<NonSerializedPartition<K>> partitionsToConvert, MemoryPoolAssigner memoryPoolAssigner)	Converts the non-serialized Partitions in an iterable to serialized partitions.
static <K extends java.io.Serializable> NonSerializedPartition	deserializePartition(int partitionSize, Serializer serializer, K key, java.io.InputStream inputStream)	Reads the data of a partition from an input stream and deserializes it.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

deserializePartition

public static <K extends java.io.Serializable> NonSerializedPartition deserializePartition(int partitionSize,
                                                                                           Serializer serializer,
                                                                                           K key,
                                                                                           java.io.InputStream inputStream)
                                                                                    throws java.io.IOException

Reads the data of a partition from an input stream and deserializes it.

Type Parameters:

K - the key type of the partitions.

Parameters:

partitionSize - the size of the partition to deserialize.

serializer - the serializer to decode the bytes.

key - the key value of the result partition.

inputStream - the input stream which will return the data in the partition as bytes.

Returns:

the list of deserialized elements.

Throws:

java.io.IOException - if fail to deserialize.

convertToSerPartitions

public static <K extends java.io.Serializable> java.lang.Iterable<SerializedPartition<K>> convertToSerPartitions(Serializer serializer,
                                                                                                                 java.lang.Iterable<NonSerializedPartition<K>> partitionsToConvert,
                                                                                                                 MemoryPoolAssigner memoryPoolAssigner)
                                                                                                          throws java.io.IOException,
                                                                                                                 MemoryAllocationException

Converts the non-serialized Partitions in an iterable to serialized partitions.

Type Parameters:

K - the key type of the partitions.

Parameters:

serializer - the serializer for serialization.

partitionsToConvert - the partitions to convert.

memoryPoolAssigner - the memory pool assigner for DirectByteBufferOutputStream.

Returns:

the converted SerializedPartitions.

Throws:

java.io.IOException - if fail to convert.

MemoryAllocationException - if fail to allocate memory.

convertToNonSerPartitions

public static <K extends java.io.Serializable> java.lang.Iterable<NonSerializedPartition<K>> convertToNonSerPartitions(Serializer serializer,
                                                                                                                       java.lang.Iterable<SerializedPartition<K>> partitionsToConvert)
                                                                                                                throws java.io.IOException

Converts the serialized Partitions in an iterable to non-serialized partitions.

Type Parameters:

K - the key type of the partitions.

Parameters:

serializer - the serializer for deserialization.

partitionsToConvert - the partitions to convert.

Returns:

the converted NonSerializedPartitions.

Throws:

java.io.IOException - if fail to convert.

blockIdToFilePath

public static java.lang.String blockIdToFilePath(java.lang.String blockId,
                                                 java.lang.String fileDirectory)

Converts a block id to the corresponding file path.

Parameters:

blockId - the ID of the block.

fileDirectory - the directory of the target block file.

Returns:

the file path of the partition.

blockIdToMetaFilePath

public static java.lang.String blockIdToMetaFilePath(java.lang.String blockId,
                                                     java.lang.String fileDirectory)

Converts a block id to the corresponding metadata file path.

Parameters:

blockId - the ID of the block.

fileDirectory - the directory of the target block file.

Returns:

the metadata file path of the partition.

concatNonSerPartitions

public static java.lang.Iterable concatNonSerPartitions(java.lang.Iterable<NonSerializedPartition> partitionsToConcat)
                                                 throws java.io.IOException

Concatenates an iterable of non-serialized Partitions into a single iterable of elements.

Parameters:

partitionsToConcat - the partitions to concatenate.

Returns:

the concatenated iterable of all elements.

Throws:

java.io.IOException - if fail to concatenate.

buildInputStream

public static java.io.InputStream buildInputStream(java.io.InputStream in,
                                                   java.util.List<DecodeStreamChainer> decodeStreamChainers)
                                            throws java.io.IOException

Chain InputStream with DecodeStreamChainers.

Parameters:

in - the InputStream.

decodeStreamChainers - the list of DecodeStreamChainer to be applied on the stream.

Returns:

chained InputStream.

Throws:

java.io.IOException - if fail to create new stream.

buildOutputStream

public static java.io.OutputStream buildOutputStream(java.io.OutputStream out,
                                                     java.util.List<EncodeStreamChainer> encodeStreamChainers)
                                              throws java.io.IOException

Chain OutputStream with EncodeStreamChainers.

Parameters:

out - the OutputStream.

encodeStreamChainers - the list of EncodeStreamChainer to be applied on the stream.

Returns:

chained OutputStream.

Throws:

java.io.IOException - if fail to create new stream.