Writing Generic UDFs in Spark

Apache Spark offers the ability to write Generic UDFs. However, for an idiomatic implementation, there are a couple of things that one needs to keep in mind. You should return a subtype of Option because Spark treats None subtype automatically as null and is able to extract value from Some subtype. Your Generic UDFs should be able to handle Option or regular type as input. To accomplish this, use type matching in case of Option and recursively extract values. This scenario occurs, if...

Read More

Testing Spark Dataframes

Testing Spark Dataframe transforms is essential and can be accomplished in a more reusable manner. The way, I generally accomplish that is to Read the expected and test Dataframe, and Invoke the desired transform, and Calculate the difference between dataframes. The only caveat in calculating the difference is that in built except function is not sufficient for columns with decimal column types and that requires a bit of work. To accomplish generic dataframe comparison: We need to look at the type of the column and when its...

Read More