Decision Tree
Note: Make sure you have your training and test data already vectorized and ready to go before you begin trying to fit the machine learning model to unprepped data.
from pyspark.ml.classification import DecisionTreeClassifier
from pyspark.ml.tuning import ParamGridBuilder, CrossValidator
from pyspark.ml.evaluation import BinaryClassificationEvaluator
dt = DecisionTreeClassifier(labelCol="label", featuresCol="features")
dtparamGrid = (ParamGridBuilder()
.addGrid(dt.maxDepth, [2, 5, 10])
.addGrid(dt.maxBins, [10, 20])
.build())
dtevaluator = BinaryClassificationEvaluator(rawPredictionCol="rawPrediction")
# Create 5-fold CrossValidator
dtcv = CrossValidator(estimator = dt,
estimatorParamMaps = dtparamGrid,
evaluator = dtevaluator,
numFolds = 5)
dtcvModel = dtcv.fit(train)
print(dtcvModel)
dtpredictions = dtcvModel.transform(test)
print('Accuracy:', dtevaluator.evaluate(dtpredictions))
print('AUC:', BinaryClassificationMetrics(dtpredictions['label','prediction'].rdd).areaUnderROC)
print('PR:', BinaryClassificationMetrics(dtpredictions['label','prediction'].rdd).areaUnderPR)
Note: When you use the
CrossValidator
function to set up cross-validation of your models, the resulting model object will have all the runs included, but will only use the best model when you interact with the model object using other functions like evaluate
or transform
.Last modified 3yr ago