Package org.apache.nemo.runtime.executor.data.partition

Class SerializedPartition<K>

java.lang.Object

org.apache.nemo.runtime.executor.data.partition.SerializedPartition<K>

Type Parameters:

K - the key type of its partitions.

All Implemented Interfaces:

Partition<byte[],K>

public final class SerializedPartition<K>
extends java.lang.Object
implements Partition<byte[],K>

A collection of data elements. The data is stored as an array of bytes. This is a unit of read / write towards Blocks. Releasing the memory(either off-heap or on-heap) occurs on block deletion. TODO #396: Refactoring SerializedPartition into multiple classes

Constructor Summary

Constructors

Constructor	Description
SerializedPartition(K key, byte[] serializedData, int length, MemoryPoolAssigner memoryPoolAssigner)	Creates a serialized Partition with actual data residing in on-heap region.
SerializedPartition(K key, java.util.List<MemoryChunk> serializedChunkList, int length, MemoryPoolAssigner memoryPoolAssigner)	Creates a serialized Partition with actual data residing in off-heap region.
SerializedPartition(K key, Serializer serializer, MemoryPoolAssigner memoryPoolAssigner)	Creates a serialized Partition without actual data.

Method Summary

Modifier and Type	Method	Description
void	commit()	Commits a partition to prevent further data write.
byte[]	getData()	This method should only be used when this partition is residing in on-heap region.
java.util.List<java.nio.ByteBuffer>	getDirectBufferList()	This method is used to emit the output as SerializedPartition.
K	getKey()
int	getLength()
boolean	isOffheap()
boolean	isSerialized()
void	release()	Releases the off-heap memory that this SerializedPartition holds.
void	write(java.lang.Object element)	Writes an element to non-committed partition.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SerializedPartition

public SerializedPartition(K key,
                           Serializer serializer,
                           MemoryPoolAssigner memoryPoolAssigner)
                    throws java.io.IOException,
                           MemoryAllocationException

Creates a serialized Partition without actual data. Data can be written to this partition until it is committed.

Parameters:

key - the key of this partition.

serializer - the serializer to be used to serialize data.

memoryPoolAssigner - the memory pool assigner for memory allocation.

Throws:

java.io.IOException - if fail to chain the output stream.

MemoryAllocationException - if fail to allocate memory.

SerializedPartition

public SerializedPartition(K key,
                           byte[] serializedData,
                           int length,
                           MemoryPoolAssigner memoryPoolAssigner)

Creates a serialized Partition with actual data residing in on-heap region. Data cannot be written to this partition after the construction.

Parameters:

key - the key.

serializedData - the serialized data.

length - the length of the actual serialized data. (It can be different with serializedData.length)

memoryPoolAssigner - the memory pool assigner.

SerializedPartition

public SerializedPartition(K key,
                           java.util.List<MemoryChunk> serializedChunkList,
                           int length,
                           MemoryPoolAssigner memoryPoolAssigner)

Creates a serialized Partition with actual data residing in off-heap region. Data cannot be written to this partition after the construction.

Parameters:

key - the key.

serializedChunkList - the serialized data in list list of MemoryChunks.

length - the length of the actual serialized data.(It can be different with serializedData.length)

memoryPoolAssigner - the memory pool assigner.

Method Detail

write

public void write(java.lang.Object element)
           throws java.io.IOException

Writes an element to non-committed partition.

Specified by:

write in interface Partition<byte[],K>

Parameters:

element - element to write.

Throws:

java.io.IOException - if the partition is already committed.

commit

public void commit()
            throws java.io.IOException

Commits a partition to prevent further data write.

Specified by:

commit in interface Partition<byte[],K>

Throws:

java.io.IOException - if fail to commit partition.

getKey

public K getKey()

Specified by:

getKey in interface Partition<byte[],K>

Returns:

the key value.

isSerialized

public boolean isSerialized()

Specified by:

isSerialized in interface Partition<byte[],K>

Returns:

whether the data in this Partition is serialized or not.

getData

public byte[] getData()
               throws java.io.IOException

This method should only be used when this partition is residing in on-heap region.

Specified by:

getData in interface Partition<byte[],K>

Returns:

the serialized data.

Throws:

java.io.IOException - if the partition is not committed yet.

getDirectBufferList

public java.util.List<java.nio.ByteBuffer> getDirectBufferList()
                                                        throws java.io.IOException

This method is used to emit the output as SerializedPartition.

Returns:

the serialized data in list of ByteBuffers

Throws:

java.io.IOException - if the partition is not committed yet.

getLength

public int getLength()
              throws java.io.IOException

Returns:

the length of the actual data.

Throws:

java.io.IOException - if the partition is not committed yet.

isOffheap

public boolean isOffheap()

Returns:

whether this SerializedPartition is residing in off-heap region.

release

public void release()

Releases the off-heap memory that this SerializedPartition holds. TODO #403: Remove 'transient' uses of SerializedPartition.