×

Welcome to TagMyCode

Please login or create account to add a snippet.
0
0
 
0
Language: Python
Posted by: Ivan Shpotenko
Added: Jun 15, 2017 12:32 PM
Modified: Jun 15, 2017 12:34 PM
Views: 13
Apply like a map to columns.
http://spark.apache.org/docs/2.1.0/api/python/pyspark.sql.html#pyspark.sql.functions.udf
Add a new column with the result of applying a function to other twos.
  1. from pyspark.sql.functions import udf
  2.  
  3. def some_funk(s1: str, s2: str):
  4.         return 'some string ' + s1 + s2
  5.  
  6. udf_some_funk = udf(some_funk, returnType=StringType)
  7.  
  8. df.withColumn('result', udf_some_funk(dа.a, df.b)).show()