Random Forest
Note: Make sure you have your training and test data already vectorized and ready to go before you begin trying to fit the machine learning model to unprepped data.
from pyspark.ml.regression import RandomForestRegressor
from pyspark.ml.tuning import ParamGridBuilder, CrossValidator
from pyspark.ml.evaluation import RegressionEvaluator
rf = RandomForestRegressor(labelCol="label", featuresCol="features")
rfparamGrid = (ParamGridBuilder()
#.addGrid(rf.maxDepth, [2, 5, 10, 20, 30])
.addGrid(rf.maxDepth, [2, 5, 10])
#.addGrid(rf.maxBins, [10, 20, 40, 80, 100])
.addGrid(rf.maxBins, [5, 10, 20])
#.addGrid(rf.numTrees, [5, 20, 50, 100, 500])
.addGrid(rf.numTrees, [5, 20, 50])
.build())
rfevaluator = RegressionEvaluator(predictionCol="prediction", labelCol="label", metricName="rmse")
# Create 5-fold CrossValidator
rfcv = CrossValidator(estimator = rf,
estimatorParamMaps = rfparamGrid,
evaluator = rfevaluator,
numFolds = 5)
rfcvModel = rfcv.fit(train)
print(rfcvModel)
rfpredictions = rfcvModel.transform(test)
print('RMSE:', rfevaluator.evaluate(rfpredictions))
Note: When you use the
CrossValidator
function to set up cross-validation of your models, the resulting model object will have all the runs included, but will only use the best model when you interact with the model object using other functions like evaluate
or transform
.Last modified 3yr ago