Reading and Writing Data

Reading in Data
...from Mounted Storage
dataset = sqlContext.read.format('csv') \
                    .options(header='true', inferSchema='true', delimiter= ',') \
                    .load('/mnt/<FOLDERNAME>/<FILENAME>.csv')
​
## or spark.read.format('csv')...
## Formats: json, parquet, jdbc, orc, libsvm, csv, text, avro
...when Schema Inference Fails
from pyspark.sql.types import *
​
schema = StructType([StructField('ID', IntegerType(), True),
                     StructField('Value', DoubleType(), True),
                     StructField('Category', StringType(), True),
                     StructField('Date', DateType(), True)])
​
dataset = sqlContext.read.format('csv') \
                    .schema(schema) \
                    .options(header='true', delimiter= ',') \
                    .load('/mnt/<FOLDERNAME>/<FILENAME>.csv')
Writing out Data
df.coalesce(1) \
   .write.format("com.databricks.spark.csv") \
   .option("header", "true") \
   .save("file.csv")
Other Resources
Apache Spark Data Sources Documentation: https://spark.apache.org/docs/latest/sql-data-sources.html​

Cloud Service Integration - Previous

Azure Data Factory

Next - Data Preparation

Shaping Data with Pipelines

Last modified 2yr ago

Export as PDF

Copy link

Outline

Reading in Data

...from Mounted Storage

...when Schema Inference Fails

Writing out Data

Other Resources