com.cloudera.oryx.api.speed

Interface SpeedModelManager<K,M,U>

Type Parameters:

K - type of key read from input topic

M - type of message read from input topic

U - type of update message read/written

All Superinterfaces:

AutoCloseable, Closeable

All Known Implementing Classes:

AbstractSpeedModelManager, ALSSpeedModelManager, ExampleSpeedModelManager, KMeansSpeedModelManager, RDFSpeedModelManager, ScalaSpeedModelManagerAdapter

public interface SpeedModelManager<K,M,U>
extends Closeable

Implementations of this interface maintain, in memory, the current state of a model in the speed layer. It is given a reference to stream of updates during initialization, and consumes models and new input, updates in-memory state, and produces updates accordingly.

Since:

2.0.0

Method Summary

Modifier and Type	Method and Description
Iterable<U>	buildUpdates(org.apache.spark.api.java.JavaPairRDD<K,M> newData)
default void	close()
void	consume(Iterator<KeyMessage<String,U>> updateIterator, org.apache.hadoop.conf.Configuration hadoopConf) Called by the framework to initiate a continuous process of reading models, and reading from the input topic and updating model state in memory, and issuing updates to the update topic.

Method Detail

consume

void consume(Iterator<KeyMessage<String,U>> updateIterator,
             org.apache.hadoop.conf.Configuration hadoopConf)
      throws IOException

Called by the framework to initiate a continuous process of reading models, and reading from the input topic and updating model state in memory, and issuing updates to the update topic. This will be executed asynchronously and may block. Note that an exception or error thrown from this method is fatal and shuts down processing.

Parameters:

updateIterator - iterator to read models from

hadoopConf - Hadoop context, which may be required for reading from HDFS

Throws:

IOException - if an error occurs while reading updates

Since:

2.0.0

buildUpdates

Iterable<U> buildUpdates(org.apache.spark.api.java.JavaPairRDD<K,M> newData)
                  throws IOException

Parameters:

newData - RDD of raw new data from the topic

Returns:

updates to publish on the update topic

Throws:

IOException - if an error occurs while building updates

Since:

2.0.0

close

default void close()

Specified by:

close in interface AutoCloseable

Specified by:

close in interface Closeable

Since:

2.0.0