> For the complete documentation index, see [llms.txt](https://docs.bdb.ai/data-pipeline/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.bdb.ai/data-pipeline/getting-started/homepage/create-job/pyspark-job.md).

# PySpark Job

{% hint style="info" %}
*Please go through the below given demonstration to create and configure a Pyspark Job.*
{% endhint %}

{% embed url="<https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fuq3RSHHup7fjHYaspk7y%2Fuploads%2Fk2cdix85VueXbeM4ft8c%2F2023-03-28-13-48-06%20(online-video-cutter.com).mp4?alt=media&token=b91b606e-e242-4f42-b93f-af97d8d33225>" %}
&#x20;PySpark Job component in the Pipeline environment
{% endembed %}

## **Creating a PySpark job**

* Open the pipeline homepage and click on the **Create Job** option.

<figure><img src="https://363587200-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fuq3RSHHup7fjHYaspk7y%2Fuploads%2F3H0tMl4vMinC7t6z5pwT%2Fimage.png?alt=media&amp;token=912a8304-4d39-4e90-8186-2a4a9e604683" alt=""><figcaption><p>Accesing the Create Job option from the Pipeline Homepage</p></figcaption></figure>

* The ***New Job*** page opens.&#x20;
* Provide the following information:
  * **Enter name**: Enter the name for the job.&#x20;
  * **Job Descriptio**n: Add description of the Job (It is an optional field).
  * **Job Baseinfo:** Select ***PySpark Job*** option from the drop in Job Base Information.
  * **Trigger By:** The PySpark Job can be triggered by another Job or PySpark Job. The PySpark Job can be triggered in two scenarios from another jobs:
    * **On Success:** Select a job from drop-down. Once the selected job is run successfully, it will trigger the PySpark Job.
    * **On Failure:** Select a job from drop-down. Once the selected job gets failed, it will trigger the PySpark Job.
  * **Is Schedule:** Put a checkmark in the given box to schedule the new Job.&#x20;
  * **Spark config:** Select resource for the job.
  * Click on ***Save*** option to save the Job.&#x20;

<figure><img src="https://363587200-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fuq3RSHHup7fjHYaspk7y%2Fuploads%2FqKkYnJjkDyMyX9opIJHy%2Fimage.png?alt=media&amp;token=d030c03c-8de5-4eea-be77-e54938d22c3f" alt=""><figcaption></figcaption></figure>

* The PySpark Job gets saved and it will redirect the user to the Job workspace.

<figure><img src="https://363587200-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fuq3RSHHup7fjHYaspk7y%2Fuploads%2FDNVVhWY1NarVR0kvxrRQ%2Fimage.png?alt=media&amp;token=3bfb380e-1e0e-45a3-810b-36d4a530535f" alt=""><figcaption><p>PySpark Job workspace</p></figcaption></figure>

## **Configuring a PySpark Job:**

Once the PySpark Job is created, follow the below given steps to configure the ***Meta Information*** tab of the PySpark Job.

* **Project Name:** Select the same Project using the drop-down menu where the concerned Notebook has been created.
* **Script Name:** This field will list the exported Notebook names which are exported from the Data Science Lab module to Data Pipeline.

{% hint style="info" %}
*<mark style="color:green;">Please Note:</mark> The script written in DS Lab module should be inside a function. Refer the* [*Export to Pipeline*](https://docs.bdb.ai/data-science-lab/project/tabs-for-a-data-science-lab-project/tabs-for-pyspark-environment/notebook/notebook-list-page/export/export-to-pipeline) *page for more details on how to export a PySpark script to the Data Pipeline module.*
{% endhint %}

* **External Library:** If any external libraries are used in the script the user can mention it here. The user can mention multiple libraries by giving comma(,) in between the names.
* **Start Function:** Select the function name in which the script has been written.
* **Script:** The Exported script appears under this space.
* **Input Data:** If any parameter has been given in the function, then the name of the parameter is provided as **Key** and value of the parameters has to be provided as **value** in this field.

<figure><img src="https://363587200-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fuq3RSHHup7fjHYaspk7y%2Fuploads%2Fb77tidliJLT2wLYHvbit%2Fimage.png?alt=media&amp;token=40ee3b76-c6ff-4b6a-9ccd-bf40cb6a5221" alt=""><figcaption><p>Configuring Meta information of PySpark Job.</p></figcaption></figure>

{% hint style="info" %}
*<mark style="color:green;">Please note:</mark>* *We are currently supporting JDBC connector like ClickHouse, PostgreSQL and MongoDB Spark connector.*
{% endhint %}