com.cloudera.oryx.app.batch.mllib.rdf

Class RDFUpdate

java.lang.Object

com.cloudera.oryx.ml.MLUpdate<String>

com.cloudera.oryx.app.batch.mllib.rdf.RDFUpdate

All Implemented Interfaces:

BatchLayerUpdate<Object,String,String>, Serializable

public final class RDFUpdate
extends MLUpdate<String>

Update function that builds and evaluates random decision forest models in the Batch Layer.

See Also:

Serialized Form

Field Summary

Fields inherited from class com.cloudera.oryx.ml.MLUpdate

MODEL_FILE_NAME

Constructor Summary

Constructors

Constructor and Description
RDFUpdate(com.typesafe.config.Config config)

Method Summary

Modifier and Type	Method and Description
org.dmg.pmml.PMML	buildModel(org.apache.spark.api.java.JavaSparkContext sparkContext, org.apache.spark.api.java.JavaRDD<String> trainData, List<?> hyperParameters, org.apache.hadoop.fs.Path candidatePath)
double	evaluate(org.apache.spark.api.java.JavaSparkContext sparkContext, org.dmg.pmml.PMML model, org.apache.hadoop.fs.Path modelParentPath, org.apache.spark.api.java.JavaRDD<String> testData, org.apache.spark.api.java.JavaRDD<String> trainData)
List<HyperParamValues<?>>	getHyperParameterValues()

Methods inherited from class com.cloudera.oryx.ml.MLUpdate

canPublishAdditionalModelData, getTestFraction, publishAdditionalModelData, runUpdate, splitNewDataToTrainTest

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

RDFUpdate

public RDFUpdate(com.typesafe.config.Config config)

Method Detail

getHyperParameterValues

public List<HyperParamValues<?>> getHyperParameterValues()

Overrides:

getHyperParameterValues in class MLUpdate<String>

Returns:

a list of hyperparameter value ranges to try, one HyperParamValues per hyperparameter. Different combinations of the values derived from the list will be passed back into MLUpdate.buildModel(JavaSparkContext,JavaRDD,List,Path)

buildModel

public org.dmg.pmml.PMML buildModel(org.apache.spark.api.java.JavaSparkContext sparkContext,
                                    org.apache.spark.api.java.JavaRDD<String> trainData,
                                    List<?> hyperParameters,
                                    org.apache.hadoop.fs.Path candidatePath)

Specified by:

buildModel in class MLUpdate<String>

Parameters:

sparkContext - active Spark Context

trainData - training data on which to build a model

hyperParameters - ordered list of hyper parameter values to use in building model

candidatePath - directory where additional model files can be written

Returns:

a PMML representation of a model trained on the given data

evaluate

public double evaluate(org.apache.spark.api.java.JavaSparkContext sparkContext,
                       org.dmg.pmml.PMML model,
                       org.apache.hadoop.fs.Path modelParentPath,
                       org.apache.spark.api.java.JavaRDD<String> testData,
                       org.apache.spark.api.java.JavaRDD<String> trainData)

Specified by:

evaluate in class MLUpdate<String>

Parameters:

sparkContext - active Spark Context

model - model to evaluate

modelParentPath - directory containing model files, if applicable

testData - data on which to test the model performance

trainData - data on which model was trained, which can also be useful in evaluating unsupervised learning problems

Returns:

an evaluation of the model on the test data. Higher should mean "better"