Transformations

This page aims to explain the various transformation options provided on the Jobs Editor page.

The following transformations are provided under the Transformations section.

1. Alter Columns

2. Select Columns

3. Date Formatter

4. Query

5. Filter

6. Formula

7. Join

8. Aggregation

9. Sort Task

Alter Columns

The Alter Columns command is used to change the data type of a column in a table.

Meta information of alter column task

Meta Information

Add the Column name from the Alter Columns tab where the datatype needs to be changed.

• Name (*): Column name.

• Alias Name(*) : New column name.

• Column Type(*): Specify datatype from dropdown.

• Add New Column: Multiple columns can be added for desired modification in the datatype.

Select Columns

Helps to select particular columns from a table definition.

Meta information of Select Columns task

Meta Information

• Name (*): Column name.

• Alias Name(*) : New column name.

• Column Type(*) : Specify datatype from dropdown.

• Add New Column: Multiple columns can be added for the desired result

Date Formatter

It helps in converting Date and Datetime columns to a desired format.

Meta information of Date Formatter task

Meta Information

• Name (*): Column name.

• Input Format(*) : The function has total 61 different formats.

• Output Format(*): The format in which the output will be given.

• Output Column Name(*): Name of the output column.

• Add New Row: To insert a new row.

Query

The Query transformation allows you to write SQL (DML) queries such as Select queries and data view queries.

Meta information of Query task

Meta Information

• Query(*): Provide a valid query to transform data.

• Table Name(*): Provide the table name.

• Schema File Name: Upload Spark Schema file in JSON format.

• Choose File: Upload a file from the system.

Please Note: Alter query will not work.

Filter

The Filter columns allow the user to filter table data based on different defined conditions.

Meta information of Filter task

Meta Information

• Field Name(*): Provide field name.

• Condition (*): 8 condition operations are available within this function.

• Logical Condition(*)(AND/OR):

• Add New Column: Adds a new Column.

Formula

It gives computation results based on the selected formula type.

Meta information of Formula task

Meta Information

• Field Name(*): Provide field name.

• Formula Type(*): Select a Formula Type from the drop-down option.

  • Math (22 Math Operations)

  • String (16 String Operations )

  • Bitwise (3 Bitwise Operations)

• Output Field Name(*): Provide the output field name.

• Add New Column: Adds a new column.

Join

It joins 2 tables based on the specified column conditions.

Meta information of Join task

Meta Information

• Join Type (*): Provides drop-down menu to choose a Join type.

  • The supported Join Types are:

    • Inner

    • Outer

    • Full

    • Full outer

    • Left outer

    • Left

    • Right outer

    • Right

    • Left semi

    • Left anti

• Left Column(*): Conditional column from the left table.

• Right Column(*) : Conditional column from right table.

• Add New Column: Adds a new column.

Aggregation

An aggregate task performs a calculation on a set of values and returns a single value by using the Group By column.

Meta information of Aggregation task

Meta Information

• Group By Columns(*): Provide a name for the group by column.

• Field Name(*): Provide the field name.

• Operation (*): 30 operations are available within this function.

• Alias(*): Provide an alias name.

• Add New Column: Adds a new column.

Sort Task

This transformation sorts all the data from a table based on the selected column and order.

Meta information of Sort task

Meta Information

• Sort Key(*): Provide a sort key.

• Order(*): Select an option out of Ascending or Descending.

• Add New Column: Adds a new column.