Naïve Bayes

Setting Up a Naïve Bayes Classifier
Note: Make sure you have your training and test data already vectorized and ready to go before you begin trying to fit the machine learning model to unprepped data.
Load in required libraries
1
from pyspark.ml.classification import NaiveBayes
2
from pyspark.ml.tuning import ParamGridBuilder, CrossValidator
3
from pyspark.ml.evaluation import BinaryClassificationEvaluator
4
from pyspark.mllib.evaluation import BinaryClassificationMetrics
Copied!
Initialize Naïve Bayes object
1
nb = NaiveBayes(labelCol="label", featuresCol="features")
Copied!
Create a parameter grid for tuning the model
1
nbparamGrid = (ParamGridBuilder()
2
               .addGrid(nb.smoothing, [0.0, 0.2, 0.4, 0.6, 0.8, 1.0])
3
               .build())
Copied!
Define how you want the model to be evaluated
1
nbevaluator = BinaryClassificationEvaluator(rawPredictionCol="rawPrediction")
Copied!
Define the type of cross-validation you want to perform
1
# Create 5-fold CrossValidator
2
nbcv = CrossValidator(estimator = nb,
3
                      estimatorParamMaps = nbparamGrid,
4
                      evaluator = nbevaluator,
5
                      numFolds = 5)
Copied!
Fit the model to the data
1
nbcvModel = nbcv.fit(train)
2
print(nbcvModel)
Copied!
Score the testing dataset using your fitted model for evaluation purposes
1
nbpredictions = nbcvModel.transform(test)
Copied!
Evaluate the model
1
print('Accuracy:', lrevaluator.evaluate(lrpredictions))
2
print('AUC:', BinaryClassificationMetrics(lrpredictions['label','prediction'].rdd).areaUnderROC)
3
print('PR:', BinaryClassificationMetrics(lrpredictions['label','prediction'].rdd).areaUnderPR)
Copied!
Note: When you use the CrossValidatorfunction to set up cross-validation of your models, the resulting model object will have all the runs included, but will only use the best model when you interact with the model object using other functions like evaluate or transform.

Logistic Regression

Decision Tree

Last modified 2yr ago

Export as PDF

Copy link

Contents

Setting Up a Naïve Bayes Classifier

Load in required libraries

Initialize Naïve Bayes object

Create a parameter grid for tuning the model

Define how you want the model to be evaluated

Define the type of cross-validation you want to perform

Fit the model to the data

Score the testing dataset using your fitted model for evaluation purposes

Evaluate the model