com.cloudera.oryx.api.batch

Interface BatchLayerUpdate<K,M,U>

Type Parameters:

K - type of key read from input topic

M - type of message read from input topic

U - type of model message written

All Superinterfaces:

Serializable

All Known Implementing Classes:

ALSUpdate, ExampleBatchLayerUpdate, KMeansUpdate, MLUpdate, RDFUpdate, ScalaBatchLayerUpdateAdapter

Functional Interface:

This is a functional interface and can therefore be used as the assignment target for a lambda expression or method reference.

@FunctionalInterface
public interface BatchLayerUpdate<K,M,U>
extends Serializable

Implementations of this interface define the update process for an instance of the batch layer. It specifies what happens in one batch update cycle. Given the time, and access to both past and current data, it defines some update process in Spark, which produces some output (e.g. a machine learning model in PMML).

Since:

2.0.0

Method Summary

Modifier and Type	Method and Description
void	runUpdate(org.apache.spark.api.java.JavaSparkContext sparkContext, long timestamp, org.apache.spark.api.java.JavaPairRDD<K,M> newData, org.apache.spark.api.java.JavaPairRDD<K,M> pastData, String modelDirString, TopicProducer<String,U> modelUpdateTopic)

Method Detail

runUpdate

void runUpdate(org.apache.spark.api.java.JavaSparkContext sparkContext,
               long timestamp,
               org.apache.spark.api.java.JavaPairRDD<K,M> newData,
               org.apache.spark.api.java.JavaPairRDD<K,M> pastData,
               String modelDirString,
               TopicProducer<String,U> modelUpdateTopic)
        throws IOException,
               InterruptedException

Parameters:

sparkContext - Spark context

timestamp - timestamp of current interval

newData - data that has arrived in current interval

pastData - all previously-known data (may be null)

modelDirString - String representation of path where models should be output, if desired

modelUpdateTopic - topic to push models onto, if desired. Note that this may be null if the application is configured to not produce updates to a topic

Throws:

IOException - if an error occurs during execution of the update function

InterruptedException - if the caller is interrupted waiting for parallel tasks to complete

Since:

2.0.0