所以解决方案确实非常简单:
var df = sqlContext.load("com.databricks.spark.csv", Map("path" -> input_file, "header" -> has_header, "delimiter" -> delimiter, "inferSchema" -> infer_schema)) val df_distinct = df.dropDuplicates();