Python Integration

Sparkflows supports Python in Workflows in a few ways:

• PySpark Processor

The PySpark Processor allows writing PySpark/Python code to processes the incoming DataFrame and create a new DataFrame. It can also be used to build scikit-learn models etc.

• Jython Processor

The Jython Processor allows writing Jython code to processes the incoming DataFrame and create a new DataFrame.

• Pipe Python Processor

Pipe Python Processor allows writing Python script to process the incoming DataFrame.

The incoming DataFrame is piped to the python script.

The Python script takes in each record of the DataFrame as a comma separated string. It parses the string, processes the record and writes out the new record.

• PySpark Processor
  • Interface
  • WorkflowContext
  • Example 1
  • Example 2
• Jython Processor
  • Example Jython Code
• Pipe Python Processor
  • Input DataFrame Schema
  • Simple Example
  • Output Schema of the Python Script
  • Program Execution Output
• Pipe Python2 Processor
  • Input DataFrame Schema
  • Reading in Data in Python into a Pandas DataFrame
  • Transform the Pandas DataFrame
  • Writing the Pandas DataFrame schema back to Spark
  • Writing the Pandas DataFrame back to Spark
  • Output Schema of the Python Script