Nemo Release 0.2

Mar 9, 2020 • Wooyeon Lee

Release Notes - Apache Nemo - Version 0.2

Sub-task

  • [NEMO-350] - Implement Off-heap SerializedMemoryStore
  • [NEMO-385] - Support Lambda Pass with lambda policy and lambda resource property
  • [NEMO-387] - Support Lambda scheduler

Bug

  • [NEMO-80] - SLF4J: Failed to load class “org.slf4j.impl.StaticLoggerBinder”
  • [NEMO-304] - Fail-fast for mis-configuration in user application
  • [NEMO-321] - Fix the data skew pass metric mismatch
  • [NEMO-327] - Fix skew handling for multi shuffle edge receiver
  • [NEMO-357] - Fix broken link on README
  • [NEMO-368] - NEMO-353 breaks the application from running in YARN environments
  • [NEMO-402] - Broken guava version conflicts cause ERROR: Trying to remove a RunningJob that is unknown
  • [NEMO-413] - Fix index checking for byte access of MemoryChunk using UNSAFE
  • [NEMO-414] - Command-line specified runtime data plane configurations not applied
  • [NEMO-416] - Guava vendor version conflict when deserializing Task object
  • [NEMO-420] - OffHeapMemory configuration only supports a single type of executor

New Feature

  • [NEMO-335] - Using a database for recording metric data
  • [NEMO-336] - Cost prediction using the metric data
  • [NEMO-359] - implementation of getEstimatedSizeBytes in SourceVertex
  • [NEMO-360] - Implementing an 'XGBoostPolicy'
  • [NEMO-388] - Off-heap memory management (reuse ByteBuffer)

Improvement

  • [NEMO-11] - Generalize Equality of Int Predicates for Loops
  • [NEMO-162] - Add insertVertex() API in optimization pass
  • [NEMO-182] - Consider reshaping in run-time optimization
  • [NEMO-253] - Refactor getInternal(Main/Additional)OutputMap in TaskExecutor
  • [NEMO-275] - Eager Garbage Collection for GroupByKey
  • [NEMO-319] - Fix path to beam resources in examples in README
  • [NEMO-320] - Make WebUI scale to big workloads
  • [NEMO-323] - Upgrade current working version to 0.2-SNAPSHOT
  • [NEMO-328] - Refactor IRDAG
  • [NEMO-332] - Refactor RunTimePass
  • [NEMO-337] - IRDAG Unit Tests
  • [NEMO-338] - SkewSamplingPass
  • [NEMO-340] - SonarCloud for PRs
  • [NEMO-353] - Launch NEXMark applications
  • [NEMO-356] - Visualize the name of beam transform in DAG
  • [NEMO-358] - Recycling vertex ids while cloning a vertex
  • [NEMO-369] - DirectByteArrayOutputStream usage refactoring
  • [NEMO-390] - Address SonarCloud issues for the IR package
  • [NEMO-391] - Set GrpcMessageEnvironment as a default implementation
  • [NEMO-395] - Address SonarCloud issues for the scheduler package
  • [NEMO-404] - Provide user argument to use lambda executor representer
  • [NEMO-412] - Address Sonar Cloud issue for MemoryChunk
  • [NEMO-418] - BlockFetchFailureProperty
  • [NEMO-424] - Fix Sonarcloud bugs regarding Optional
  • [NEMO-428] - Ignore .factorypath for rat check and version control

Task

  • [NEMO-25] - Improve WebUI to use RESTful APIs by Nemo
  • [NEMO-153] - IR-based dynamic optimization for WordCount application
  • [NEMO-154] - Handle skewness information in SchedulingConstraint
  • [NEMO-193] - Revised version of IR-based dynamic optimization
  • [NEMO-322] - Committer's guide
  • [NEMO-361] - Consistency on indentations
  • [NEMO-362] - Upgrade of checkstyle version
  • [NEMO-364] - Upgrade Beam
  • [NEMO-375] - Add option to turn off metric collection to DB
  • [NEMO-379] - Change javadoc goal to a proper one
  • [NEMO-383] - Implement DirectByteBufferOutputStream for Off-heap SerializedMemoryStore
  • [NEMO-384] - Implement DirectByteBufferInputStream for Off-heap SerializedMemoryStore
  • [NEMO-397] - Separation of JVM heap region and off-heap memory region
  • [NEMO-399] - Include the official WordCount example on the Beam website
  • [NEMO-400] - Javadoc compile error
  • [NEMO-421] - Release v0.2
  • [NEMO-437] - Support Java version 11

Umbrella

List of Contributors

Nemo 0.2 was the work of many contributors:

  • Arun Lakshman R
  • Byung-Gon Chun
  • Davor Bonaci
  • Eunji Jeong
  • Geon Woo Kim
  • Gyewon Lee
  • Haeyoon Cho
  • Jae Hyeon Park
  • JangHo Seo
  • Jangho Seo
  • Jean-Baptiste Onofré
  • Jeongyoon Eo
  • John Yang
  • Joo Yeon Kim
  • Kenn Knowles
  • Markus Weimer
  • Minhyeok Kweun
  • Sanha Lee
  • Seonghyun Park
  • Soojeong Kim
  • Taegeon Um
  • Won Wook SONG
  • Wooyeon Lee
  • Yunseong Lee