com.cloudera.oryx.lambda.batch

Class ScalaBatchLayerUpdateAdapter<K,M,U>

java.lang.Object

com.cloudera.oryx.lambda.batch.ScalaBatchLayerUpdateAdapter<K,M,U>

Type Parameters:

K - type of key read from input topic

M - type of message read from input topic

U - type of model message written

All Implemented Interfaces:

BatchLayerUpdate<K,M,U>, Serializable

public final class ScalaBatchLayerUpdateAdapter<K,M,U>
extends Object
implements BatchLayerUpdate<K,M,U>

Adapts a ScalaBatchLayerUpdate to be a Java BatchLayerUpdate.

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
ScalaBatchLayerUpdateAdapter(com.cloudera.oryx.api.batch.ScalaBatchLayerUpdate<K,M,U> scalaUpdate)

Method Summary

Modifier and Type	Method and Description
void	runUpdate(org.apache.spark.api.java.JavaSparkContext sparkContext, long timestamp, org.apache.spark.api.java.JavaPairRDD<K,M> newData, org.apache.spark.api.java.JavaPairRDD<K,M> pastData, String modelDirString, TopicProducer<String,U> modelUpdateTopic)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ScalaBatchLayerUpdateAdapter

public ScalaBatchLayerUpdateAdapter(com.cloudera.oryx.api.batch.ScalaBatchLayerUpdate<K,M,U> scalaUpdate)

Method Detail

runUpdate

public void runUpdate(org.apache.spark.api.java.JavaSparkContext sparkContext,
                      long timestamp,
                      org.apache.spark.api.java.JavaPairRDD<K,M> newData,
                      org.apache.spark.api.java.JavaPairRDD<K,M> pastData,
                      String modelDirString,
                      TopicProducer<String,U> modelUpdateTopic)

Specified by:

runUpdate in interface BatchLayerUpdate<K,M,U>

Parameters:

sparkContext - Spark context

timestamp - timestamp of current interval

newData - data that has arrived in current interval

pastData - all previously-known data (may be null)

modelDirString - String representation of path where models should be output, if desired

modelUpdateTopic - topic to push models onto, if desired. Note that this may be null if the application is configured to not produce updates to a topic