com.cloudera.oryx.lambda

Class AbstractSparkLayer<K,M>

java.lang.Object

com.cloudera.oryx.lambda.AbstractSparkLayer<K,M>

Type Parameters:

K - input topic key type

M - input topic message type

All Implemented Interfaces:

Closeable, AutoCloseable

Direct Known Subclasses:

BatchLayer, SpeedLayer

public abstract class AbstractSparkLayer<K,M>
extends Object
implements Closeable

Encapsulates commonality between Spark-based layer processes, BatchLayer and SpeedLayer

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	AbstractSparkLayer(com.typesafe.config.Config config)

Method Summary

Modifier and Type	Method and Description
protected org.apache.spark.streaming.api.java.JavaInputDStream<org.apache.kafka.clients.consumer.ConsumerRecord<K,M>>	buildInputDStream(org.apache.spark.streaming.api.java.JavaStreamingContext streamingContext)
protected org.apache.spark.streaming.api.java.JavaStreamingContext	buildStreamingContext()
protected com.typesafe.config.Config	getConfig()
protected abstract String	getConfigGroup()
protected String	getGroupID()
protected String	getID()
protected String	getInputTopicLockMaster()
protected Class<K>	getKeyClass()
protected abstract String	getLayerName()
protected Class<M>	getMessageClass()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.io.Closeable

close

Constructor Detail

AbstractSparkLayer

protected AbstractSparkLayer(com.typesafe.config.Config config)

Method Detail

getConfigGroup

protected abstract String getConfigGroup()

Returns:

layer-specific config grouping under "oryx", like "batch" or "speed"

getLayerName

protected abstract String getLayerName()

Returns:

display name for layer like "BatchLayer"

getConfig

protected final com.typesafe.config.Config getConfig()

getID

protected final String getID()

getGroupID

protected final String getGroupID()

getInputTopicLockMaster

protected final String getInputTopicLockMaster()

getKeyClass

protected final Class<K> getKeyClass()

getMessageClass

protected final Class<M> getMessageClass()

buildStreamingContext

protected final org.apache.spark.streaming.api.java.JavaStreamingContext buildStreamingContext()

buildInputDStream

protected final org.apache.spark.streaming.api.java.JavaInputDStream<org.apache.kafka.clients.consumer.ConsumerRecord<K,M>> buildInputDStream(org.apache.spark.streaming.api.java.JavaStreamingContext streamingContext)