# Jobs

Jobs are used for ingesting and transferring data from separate sources. The user can transform, unify, and cleanse data to make it suitable for analytics and business reporting without using the Kafka topic which makes the entire flow much faster.

{% hint style="success" %}
*Check out the given demonstration to understand how to create and activate a job.*
{% endhint %}

<figure><img src="https://363587200-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fuq3RSHHup7fjHYaspk7y%2Fuploads%2FFSKyFpeWMZP0odK5v11G%2Fjobactivatemode.gif?alt=media&#x26;token=ce6d5b29-7abd-4f55-ae5f-dbf739e8cdb4" alt=""><figcaption><p>Creating and activating a Job</p></figcaption></figure>

## **Listing Jobs**

The ***List Jobs*** option opens the available Jobs List for the logged-in user. All the saved Jobs by a user get listed on this page. By clicking on the Job name the ***Details*** tab on the right side of the page gets displayed with the basic details of the selected job.

* Navigate to the ***Data Pipeline*** homepage.
* Click on the ***List Jobs*** icon.

<figure><img src="https://363587200-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fuq3RSHHup7fjHYaspk7y%2Fuploads%2FRSk4tAyEf6MQgj8gAQCq%2Fimage.png?alt=media&#x26;token=6c599a0b-2216-4082-b096-f37306abe6e4" alt=""><figcaption></figcaption></figure>

* The ***List Jobs*** page opens displaying the created jobs.

<figure><img src="https://363587200-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fuq3RSHHup7fjHYaspk7y%2Fuploads%2FOxGQIUV4T2kcvCL1KFQG%2Fimage.png?alt=media&#x26;token=8cbd907d-93bf-4882-a035-6ad45d4c2bd6" alt=""><figcaption></figcaption></figure>

{% hint style="info" %}
*<mark style="color:green;">Please Note:</mark>*

•       *The user can open the Job Editor for the selected job from the list by clicking the **View** icon.*

*•       The user can search for a specific Job by using the **Search** bar on the Job List. By typing a common name all the existing jobs having that word will list. E.g., By typing '**dem'** all the existing Jobs with the word '**demo'** in it get listed as displayed in the following image:*

![](https://363587200-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fuq3RSHHup7fjHYaspk7y%2Fuploads%2F0h2yPEJYNtpVvebSOGfN%2Fimage.png?alt=media\&token=d3c27b37-d066-455e-b3db-7c713c035ba3)
{% endhint %}

## **Creating a new Job**

* Navigate to the ***Data Pipeline*** homepage.
* Click on the ***Create Job*** icon.

![](https://363587200-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fuq3RSHHup7fjHYaspk7y%2Fuploads%2FGLcuUMUBaHQ49ohSfZki%2F3.png?alt=media)

* The ***New Job*** dialog box appears redirecting the user to create a new Job.
* Enter a name for the new Job.
* Describe the Job(Optional).
* **Job Baseinfo.**: Select spark job from the dropdown menu.
* **Trigger By**: There are 2 options for triggering a job on success or failure of a job:
  * Success Job: On successful execution of the selected job the current job will be triggered.
  * Failure Job: On failure of the selected job the current job will be triggered.
* **Is Scheduled?**
  * A job can be scheduled for a particular timestamp. Every time at the same timestamp the job will be triggered.
  * Job must be scheduled according to UTC.
* **Spark Configuration**
  * Select a resource allocation option using the radio button. The given choices are:
    * Low
    * Medium
    * High
  * This feature is used to deploy the Job with high, medium, or low-end configurations according to the velocity and volume of data that the ***Job*** must handle.

<figure><img src="https://363587200-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fuq3RSHHup7fjHYaspk7y%2Fuploads%2FLiOqU17R4xzAcIttq0mF%2Fimage.png?alt=media&#x26;token=b318027e-0bc7-4bdc-82c5-746ceef2de2d" alt=""><figcaption></figcaption></figure>

* Click the ***Save*** option to create the job.&#x20;

<figure><img src="https://363587200-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fuq3RSHHup7fjHYaspk7y%2Fuploads%2FdizygRevIRJ9dN5C2tFA%2Fimage.png?alt=media&#x26;token=1fd97d0a-8eaf-46dc-ad1b-d069df39dcf9" alt=""><figcaption></figcaption></figure>

{% hint style="info" %}
*<mark style="color:green;">Please Note:</mark> By clicking the **Save** option, the user gets redirected to the job workflow editor.*
{% endhint %}

* A success message appears to confirm the creation of a new job.
* The job editor page opens for the newly created job.

<figure><img src="https://363587200-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fuq3RSHHup7fjHYaspk7y%2Fuploads%2FtbpJrnvJp7pKnL8MIyNq%2Fimage.png?alt=media&#x26;token=48e266a3-068c-4d87-9c7a-495eeb52b374" alt=""><figcaption></figcaption></figure>

{% hint style="info" %}
*<mark style="color:green;">Please Note:</mark>* *The Current Job will not be triggered if the selected job is run in the **Preview** mode.*
{% endhint %}

## **Adding a Task to the Job Workflow**

*Once the Job gets saved in the Job list, the user can add a Task to the canvas. The user can drag the required tasks to the canvas and configure it to create a **Job workflow** or **dataflow**.*

The ***Job Editor*** appears displaying the ***Task Pallet*** containing various components mentioned as **Tasks**.&#x20;

<details>

<summary>Task Pallet</summary>

The ***Task Pallet*** is situated on the left side of the User Interface. It has a ***Task*** tab listing the various tasks.

The tasks are displayed in the below-given image:

![](https://363587200-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fuq3RSHHup7fjHYaspk7y%2Fuploads%2FqNWYGPXrPn9hPLKjTjgX%2Fimage.png?alt=media\&token=595ef235-8cf7-4d22-b98f-1b35c3ce308e)

</details>

<details>

<summary>Task Components</summary>

The components used under the ***Task Pallet*** are broadly classified into:

* [x] **Readers**: Your repository of data can be a reader for you. It could be a database, a file, or a SaaS application.
* [x] **Writers**: The databases or data warehouses to which the data is loaded by the Jobs.
* [x] **Transforms**: The series of transformation tasks that help to cleanse, enrich, and prepare data for smooth analytics.

</details>

<details>

<summary>Searching a Task</summary>

* There is a ***Search Task*** space provided on the ***Task Panel***.

&#x20;     ![](https://363587200-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fuq3RSHHup7fjHYaspk7y%2Fuploads%2Fwu2qA6m5hNbthuTmixKY%2Fimage.png?alt=media\&token=02ff521c-ee8c-41d3-99db-07c262fe3dc9)

* The user can search for a task by typing in the given space, the related Tasks will appear as suggestions.

&#x20;     ![](https://363587200-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fuq3RSHHup7fjHYaspk7y%2Fuploads%2FoA1vZ3SvVXJPUtwORjxP%2Fimage.png?alt=media\&token=850a12fb-660e-4dc0-9660-35c6576480d5)

* Select a suggestion for search and the ***Task Panel*** will display the customised view based on the searched task.

&#x20;     ![](https://363587200-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fuq3RSHHup7fjHYaspk7y%2Fuploads%2FQmMQyuPion2AuZdD8Z5E%2Fimage.png?alt=media\&token=4dfd4e49-7b93-487c-8985-9e3fd0149753)

</details>

Steps to create a Job Workflow:

* Navigate to the ***Job List*** page.
* Select a ***Job*** from the displayed list.
* Click the ***View*** icon for the Job.     &#x20;

<figure><img src="https://363587200-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fuq3RSHHup7fjHYaspk7y%2Fuploads%2FxnTM1fBU0SnTrIbiY7Cq%2Fimage.png?alt=media&#x26;token=defffea1-74d3-420d-a9aa-dbbf77997c90" alt=""><figcaption></figcaption></figure>

{% hint style="info" %}
*<mark style="color:green;">Please Note:</mark> Generally, the user can perform this step-in continuation to the **Job** creation, but in case the user has come out of the **Job Editor** the above steps can help to access it again.*
{% endhint %}

* The ***Job Editor*** opens for the selected Job.
* Drag and drop the new required task or make changes in the existing task’s meta information or change the task configuration as the requirement. (E.g., the RDBMS Reader is dragged to the workspace in the below-given image):

&#x20;        ![](https://363587200-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fuq3RSHHup7fjHYaspk7y%2Fuploads%2FNNsDVy7GxWlIeOwUHyF0%2Fimage.png?alt=media\&token=cef22f55-91f6-4e86-9eac-9f6aab0a858d)

* Click on the dragged task icon.
* The task-specific fields open asking the meta-information about the dragged component.

<figure><img src="https://363587200-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fuq3RSHHup7fjHYaspk7y%2Fuploads%2FSpr8foSaHCr9NE8oktdX%2Fimage.png?alt=media&#x26;token=ad923b4b-7077-4514-8ffc-a285df59a2d1" alt=""><figcaption></figcaption></figure>

* Open the Meta Information tab and configure the required information for the dragged component.
  * Host IP Address
  * Port number
  * Username
  * Password
  * Database Name
  * Driver- Select from the drop-down menu
  * Table Name
  * Query
  * Fetch size
* Click the given icon to validate the connection.&#x20;
* Click the ***Save Task in Storage*** icon.

&#x20;     &#x20;

<figure><img src="https://363587200-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fuq3RSHHup7fjHYaspk7y%2Fuploads%2FP1O3lDUg6V8XNQz5htJl%2Fimage.png?alt=media&#x26;token=47a48115-caac-461b-a24a-b0b604509340" alt=""><figcaption></figcaption></figure>

* A notification message appears.

&#x20;     ![](https://363587200-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fuq3RSHHup7fjHYaspk7y%2Fuploads%2FoDFfG2IVbPGYStYqMMw8%2Fimage.png?alt=media\&token=afe248fe-ae76-45d2-87f4-4a1127002e5b)

* Click the ***Activate Job*** <img src="https://363587200-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fuq3RSHHup7fjHYaspk7y%2Fuploads%2FtiAwoEj20N1nmrNinn9f%2Fimage.png?alt=media&#x26;token=39754cbb-90c8-4e98-bbcd-250a819f911b" alt="" data-size="line"> icon to activate the job(It appears only after the newly created job gets successfully updated).

&#x20;      ![](https://363587200-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fuq3RSHHup7fjHYaspk7y%2Fuploads%2FnKw6Qaz9OkrlAbVLwbGM%2Fimage.png?alt=media\&token=4b0cb829-6033-4cf1-b930-8d03a4cdf8f1)

* A dialog window opens to confirm the action of job activation.
* Click the ***YES*** option to activate the job.

&#x20;       ![](https://363587200-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fuq3RSHHup7fjHYaspk7y%2Fuploads%2F8uhRdZLXOFvGDWBsCSRt%2Fimage.png?alt=media\&token=ae37b083-7c7a-4d96-947e-ae20852ff783)

* A success message appears confirming the activation of the job.

&#x20;     ![](https://363587200-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fuq3RSHHup7fjHYaspk7y%2Fuploads%2FcvWG5lePhPvD5LFJDcxY%2Fimage.png?alt=media\&token=207b560a-5c66-4703-91b2-84d49ff19965)

* The Development mode opens for the selected Job.

{% hint style="info" %}
*<mark style="color:green;">Please Note:</mark> Jobs can be run in the Development mode as well. The user can preview only 10 records if the job is running in the Development mode.*
{% endhint %}

* The ***Status*** for the Job gets changed on the job List page when they are running in the Development mode.

![](https://363587200-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fuq3RSHHup7fjHYaspk7y%2Fuploads%2FJgAcxWkqgb6MzwpcmrbY%2F17.png?alt=media)

{% hint style="info" %} <mark style="color:green;">Please Note:</mark> *Click the **Delete*** ![](https://363587200-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fuq3RSHHup7fjHYaspk7y%2Fuploads%2FNqFbFHQSCV9jp4bp88YZ%2F18.png?alt=media) *icon from the Job Editor page to delete the selected job. The deleted job gets removed from the Job list.*
{% endhint %}

## **Toggle Log Panel**

The Toggle Log Panel displays the Logs and Advanced Logs tabs for the Job Workflows.

* Navigate to the ***Job Editor*** page.
* Click the ***Toggle Log Panel*** icon on the header.

![](https://363587200-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fuq3RSHHup7fjHYaspk7y%2Fuploads%2F6CL4qhAspnJ1dCubxosb%2F19.png?alt=media)

* A panel ***Toggles*** displaying the collective logs of the job under the ***Logs*** tab.

&#x20;      ![](https://363587200-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fuq3RSHHup7fjHYaspk7y%2Fuploads%2FF2jt3TQMcsLJyfX7sSm1%2Fimage.png?alt=media\&token=0d5bc395-040b-426f-9aba-ac974f80176d)

* Select the ***Advanced Logs*** tab to display the pod status of the complete Job.

&#x20;     ![](https://363587200-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fuq3RSHHup7fjHYaspk7y%2Fuploads%2FHiQuZOtFeqfmOudNpXLN%2Fimage.png?alt=media\&token=940e279d-29f6-4621-a720-839a885c94f2)

### **Icons on the Header panel of the Job Editor Page**

<table><thead><tr><th width="126.33333333333331">Icon</th><th>Name</th><th>Action</th></tr></thead><tbody><tr><td><img src="https://363587200-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fuq3RSHHup7fjHYaspk7y%2Fuploads%2FTKfHwRvLUxAg1z9uRv11%2Fimage.png?alt=media&#x26;token=57ad3b5e-ae4c-42a0-a905-4ba921e12ee2" alt=""></td><td>Job Version details</td><td>Displays the latest versions for the Jobs upgrade.</td></tr><tr><td><img src="https://363587200-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fuq3RSHHup7fjHYaspk7y%2Fuploads%2FLlPYrluVIzsUvb5DBKWX%2Fimage.png?alt=media&#x26;token=5e662816-1db5-436b-8da5-65000f6f0973" alt="" data-size="line"></td><td>Log Panel</td><td>Displays Jobs logs.</td></tr><tr><td><img src="https://363587200-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fuq3RSHHup7fjHYaspk7y%2Fuploads%2FpaJx88SrasTjMmDZf0zH%2Fimage.png?alt=media&#x26;token=f370b777-e75e-4ca1-9cd4-710b4426c8af" alt="" data-size="line"></td><td>Preview Mode</td><td>The Job can be previewed, and 10 records will be displayed.</td></tr><tr><td><img src="https://363587200-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fuq3RSHHup7fjHYaspk7y%2Fuploads%2FZI2tvRoEpz3ayvBZaVFf%2Fimage.png?alt=media&#x26;token=fcb373c4-a133-41ba-b7ad-2c10c5da2b4f" alt="" data-size="line"></td><td>Activate Job</td><td>Activates the current  Job.</td></tr><tr><td><img src="https://363587200-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fuq3RSHHup7fjHYaspk7y%2Fuploads%2FlBstv6U3RsoBQZWlqOC7%2Fimage.png?alt=media&#x26;token=dddffe39-cbad-422d-b623-eb17a2787a38" alt="" data-size="line"></td><td>Update Job</td><td>Updates the current Job.</td></tr><tr><td><img src="https://363587200-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fuq3RSHHup7fjHYaspk7y%2Fuploads%2FAeh98orulhfsGh48NkRB%2Fimage.png?alt=media&#x26;token=7a68d9d5-1f97-4442-8cc2-aafe43304c32" alt="" data-size="line"></td><td>Edit Job</td><td>To edit the job name/ configurations.</td></tr><tr><td><img src="https://363587200-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fuq3RSHHup7fjHYaspk7y%2Fuploads%2FiRugvWHrVB9AzKxXQK3J%2Fimage.png?alt=media&#x26;token=6baeb803-4cc4-4a15-a184-c34e5acb8904" alt="" data-size="line"></td><td>Delete Job</td><td>Deletes the current Job.</td></tr><tr><td><img src="https://363587200-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fuq3RSHHup7fjHYaspk7y%2Fuploads%2Fk6CRZ0ag152euaxgfIYg%2Fimage.png?alt=media&#x26;token=efeb2432-3b11-477d-9701-279b61faa290" alt="" data-size="line"></td><td>List Job</td><td>Redirects to the <em><strong>List Job</strong></em> page.</td></tr><tr><td><img src="https://363587200-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fuq3RSHHup7fjHYaspk7y%2Fuploads%2FKd4wM6LTXNtaXKr7Uola%2Fimage.png?alt=media&#x26;token=794ad142-e8e9-495e-a5f8-20cba49681b3" alt="" data-size="line"></td><td>Settings</td><td>Redirects to the <em><strong>Settings</strong></em> page.</td></tr></tbody></table>

###

###