Package org.apache.nemo.compiler.optimizer.pass.compiletime.annotating

Class LargeShuffleAnnotatingPass

  • All Implemented Interfaces:
    java.util.function.Function<IRDAG,​IRDAG>
    public final class LargeShuffleAnnotatingPass
extends AnnotatingPass
    This pass assumes that a RelayVertex was previously inserted to receive each shuffle edge.

    src - shuffle-edge - streamvertex - one-to-one-edge - dst

    (1) shuffle-edge Encode/compress into byte[], and have the receiver read data as the same byte[], rather than decompressing/decoding. Perform a push-based in-memory shuffle with discarding on.

    (2) streamvertex Ignore resource slots, such that all tasks fetch the in-memory input data blocks as soon as they become available.

    (3) one-to-one-edge Do not encode/compress the byte[] Perform a pull-based and on-disk data transfer with the DedicatedKeyPerElement.

    • Constructor Detail

      • LargeShuffleAnnotatingPass

        public LargeShuffleAnnotatingPass()
        Default constructor.