com.cloudera.oryx.example.batch

Class ExampleBatchLayerUpdate

java.lang.Object

com.cloudera.oryx.example.batch.ExampleBatchLayerUpdate

All Implemented Interfaces:

BatchLayerUpdate<String,String,String>, Serializable

public final class ExampleBatchLayerUpdate
extends Object
implements BatchLayerUpdate<String,String,String>

Input keys are ignored. Values are treated as lines of space-separated text. The job counts, for each word, the number of distinct other words that co-occur in some line of text in the input. These are written as a "MODEL" update, where the word-count mapping is written as a JSON string.

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
ExampleBatchLayerUpdate()

Method Summary

Modifier and Type	Method and Description
static Map<String,Integer>	countDistinctOtherWords(org.apache.spark.api.java.JavaPairRDD<String,String> data)
void	runUpdate(org.apache.spark.api.java.JavaSparkContext sparkContext, long timestamp, org.apache.spark.api.java.JavaPairRDD<String,String> newData, org.apache.spark.api.java.JavaPairRDD<String,String> pastData, String modelDirString, TopicProducer<String,String> modelUpdateTopic)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ExampleBatchLayerUpdate

public ExampleBatchLayerUpdate()

Method Detail

runUpdate

public void runUpdate(org.apache.spark.api.java.JavaSparkContext sparkContext,
                      long timestamp,
                      org.apache.spark.api.java.JavaPairRDD<String,String> newData,
                      org.apache.spark.api.java.JavaPairRDD<String,String> pastData,
                      String modelDirString,
                      TopicProducer<String,String> modelUpdateTopic)
               throws IOException

Specified by:

runUpdate in interface BatchLayerUpdate<String,String,String>

Parameters:

sparkContext - Spark context

timestamp - timestamp of current interval

newData - data that has arrived in current interval

pastData - all previously-known data (may be null)

modelDirString - String representation of path where models should be output, if desired

modelUpdateTopic - topic to push models onto, if desired. Note that this may be null if the application is configured to not produce updates to a topic

Throws:

IOException - if an error occurs during execution of the update function

countDistinctOtherWords

public static Map<String,Integer> countDistinctOtherWords(org.apache.spark.api.java.JavaPairRDD<String,String> data)