Transformations
The Transformations section of the Jobs Editor provides a set of configurable tasks to modify, filter, compute, and join data before it is written to a target. Each transformation exposes a Meta Information tab where users define the parameters of the operation.
Available Transformations
The following transformations are supported:
Alter Columns
Select Columns
Date Formatter
Query
Filter
Formula
Join
Aggregation
Sort Task
Alter Columns
Change the data type or rename existing columns.
Meta Information Fields
Name
Name of the column to alter.
Alias Name
New name of the column.
Column Type
Select the new data type from the drop-down.
Add New Column
Allows multiple column definitions for modification.
Select Columns
Choose specific columns from the dataset and optionally rename or re-type them.
Meta Information Fields
Name
Name of the column to select.
Alias Name
New name of the column.
Column Type
Select the data type from the drop-down.
Add New Column
Allows multiple columns to be selected.
Date Formatter
Convert date or datetime columns into a specified format.
Meta Information Fields
Name
Column name to format.
Input Format
Select from 61 supported input formats.
Output Format
Format for the transformed output.
Output Column Name
Name of the resulting column.
Add New Row
Add multiple formatter rules.
Query
Apply SQL queries to transform data (e.g., SELECT queries, views).
Meta Information Fields
Query
Provide a valid SQL query. (Note: ALTER queries are not supported.)
Table Name
Target table name.
Schema File Name
Upload a Spark schema file (JSON format).
Choose File
Upload schema file from the system.
Filter
Filter records based on conditions and logical operators.
Meta Information Fields
Field Name
Column to apply the filter on.
Condition
Choose from 8 available condition operators.
Logical Condition
Apply AND or OR to combine filters.
Add New Column
Add multiple filter conditions.
Formula
Apply mathematical, string, or bitwise formulas to generate new columns.
Meta Information Fields
Field Name
Input field name.
Formula Type
Choose from: <ul><li>Math (22 operations)</li><li>String (16 operations)</li><li>Bitwise (3 operations)</li></ul>
Output Field Name
Name of the result column.
Add New Column
Add multiple formulas.
Join
Join two datasets based on column conditions.
Meta Information Fields
Join Type
Choose from: Inner, Outer, Full, Full Outer, Left Outer, Left, Right Outer, Right, Left Semi, Left Anti.
Left Column
Join key column from the left dataset.
Right Column
Join key column from the right dataset.
Add New Column
Add multiple join conditions.
Aggregation
Perform aggregation functions with optional grouping.
Meta Information Fields
Group By Columns
Name of the grouping column(s).
Field Name
Column on which the operation is applied.
Operation
Select from 30 aggregation operations.
Alias
Alias name for the aggregation result.
Add New Column
Add multiple aggregation rules.
Sort Task
Sort data based on column(s) and order.
Meta Information Fields
Sort Key
Column used for sorting.
Order
Sorting order: Ascending or Descending.
Add New Column
Add multiple sort keys.