RDBMS Writer

Configuration

All component configurations are classified broadly into the following sections:

• Basic

• Metadata

• Resource Configuration

• Connection Validation

Please follow the demonstration to configure the component.

RDBMS Writer

Drivers Available:

• MySQL

• Oracle

• PostgreSQL

• MS-SQL

• ClickHouse

Please Note:

• The ClickHouse driver in the Spark components will use HTTP Port and not the TCP port.

• It is always recommended to create the table before activating the pipeline to avoid errors as RDBMS has a strict schema and can result in errors.

Save Modes

RDBMS writer supports 3 save modes:

Append

As the name suggests it adds all the records without any validations.

Overwrite

This mode truncates the table and adds fresh records. after every run you will get records that are part of the batch process.

Upsert

This operation allows the users to insert a new record or update existing data into a table. For configuring this we need to provide the Composite Key.

We support composite key based upsert, in case of composite key, we can specify the second key by using comma separator eg:- key1, key2

It has now an option to upload the spark schema; This can greatly improve the speed of the write operation as the component will ignore inferring schema and go with the provided schema.

Spark Schema upload

Note: For ClickHouse Component Upsert is comparatively slow. We prefer to create a table where the engine is ReplacingMergeTree and a view where we load the view with the Final clause. In the component keep the write mode to Append.