Data Pipeline
1. Spark/Scala/ Kafka upgrade [All the components will run on the below versions of Scala and spark with the support of the below Kafka version]:
a. Upgraded the Spark version to 3.3.0
b. Upgraded the Scala version to 2.12.12
c. Upgraded the Kafka version to 3.1.0
2. Writers: HA (High Availability) support for:
a. MSSQL
b. Oracle
c. ClickHouse
d. PostgreSQL
e. MySQL
f. MongoDB
3. Readers: HA (High Availability) support for:
a. MSSQL
b. Oracle
c. ClickHouse
d. PostgreSQL
e. MySQL
f. MongoDB
4. Provided HA (High Availability) support for the Enrichment component.
6. ClickHouse Support:
a. Introduced a component Kafka Producer to connect to external Kafka topics for different applications.
b. Kafka Consumer supports SSL Security Type with Host Aliasing.
14. Implemented required changes to align with the new version of the DS Lab module that supports Data Preparation and Auto ML options.
Last updated
Was this helpful?