[TBD]Learning Spark

DataFrame useful APIs

show() -> To display data frame table

withColumn () -> To create new column, to modify existing column

withColumnRenamed () -> To rename existing column

select -> To select a column

drop -> Drop an existing column

Dataframe string manipulation

concat_ws -> to concat two columns

alias -> to use alias of name

lower/upper -> To change case

regex_replace -> To replace string

User defined function

https://www.geeksforgeeks.org/how-to-write-spark-udf-user-defined-functions-in-python/

Reference

https://www.geeksforgeeks.org/how-to-write-spark-udf-user-defined-functions-in-python/

Page updated

Google Sites

Report abuse