PySpark Script

Check out the given demonstrations to understand the configuration steps involved in the PySpark Script.

PySpark Script component with Custom Script as an Execution Type

PySpark Script component with DSLab Script as an Execution Type

Configuring Meta Information of Python Script Component

Please Note: Do not provide 'test' as a component name or the component name should not start with 'test' in the component name field in the Meta information of the Python Script component. The word 'test' is used at the backend for some development processes.

• Component Name: Provide a name to the component. Please note that the component name should be without space and special characters. Use the underscore symbol to show space between words.

• Start Function Name: It displays all the function names used in the PySpark script in a drop-down menu. Select one function name with which you want to start.

• In Event Data Type: The user will find two options here:

  • DataFrame

  • List of Dictionary

• External Libraries: The user can provide external PySpark libraries in the script. The user can enter multiple library names separated by commas.

• Execution Type: Select the Type of Execution from the drop-down. There are two execution types supported:

  • Custom Script: The users can write their custom PySpark script in the Script field.

    • Script: The user can write their custom PySpark script in this field. Make sure the start should contain at least one function. The user can also validate the script by clicking the Validate Script option in this field.

    • Start Function: Here, all the function names used in the script will be listed. Select the start function name to execute the PySpark script.

    • Input Data: If any parameter has been given in the function, then the parameter's name is provided as Key, and the value of the parameters has to be provided as a value in this field.

  • DSLab Script: In this execution type, the user can use the script exported from the DSLab notebook. The user needs to provide the following information if this option is selected as an Execution Type:

    • Project Name: Select the same Project using the drop-down menu where the Notebook has been created.

    • Script Name: This field will list the exported Notebook names from the Data Science Lab module to the Data Pipeline.

    • Start Function: All the function names used in the script will be listed here. Select the start function name to execute the PySpark script.

    • Script: The Exported script appears under this space. The user can also validate the script by Clicking on the Validate Script option in this field. For more information about exporting the script from the DSLab module, please refer to the following link: Exporting a Script from DSLab.

    • Input Data: If any parameter has been given in the function, then the parameter's name is provided as Key, and the value of the parameters has to be provided as a value in this field.

• Pull script from VCS: It allows the user to pull the desired committed script from the VCS.

• Push script to VCS: It allows the user to commit different versions of a script to the VCS.

Custom PySpark Script Sample

import pandas as pd
import numpy as np
import json
from datetime import datetime
 
def a():
    data = {
        'Lists': [[1, 2, 3], [4, 5], [6]],
        'JSON': [json.dumps({"name": "Alice", "age": 30}),
                 json.dumps({"name": "Bob", "age": 25}),
                 json.dumps({"name": "Charlie", "age": 35})],
        'Numbers': [10, 20, 30],
        'Timestamps': [datetime.now(), datetime.now(), datetime.now()]
    }
 
    # Create DataFrame
    df = pd.DataFrame(data)
 
    # Display the DataFrame
    return df