MongoDB Writer Lite (PyMongo Writer)

The PyMongo Writer component writes data into a MongoDB collection using the PyMongo client. It supports flexible connection options, SSL-based secure communication, and two save modes: Append and Upsert. Users can also filter or transform columns before writing.

Configuration Sections

The PyMongo Writer configurations are organized into the following sections:

  • Basic Information

  • Meta Information

  • Resource Configuration

  • Connection Validation

Meta Information Tab

Connection Settings

Parameter
Description
Example

Connection Type

Select the type of MongoDB connection. Options: Standard, SRV, Connection String.

Standard

Port Number (*)

Port of the MongoDB server. Required only for Standard.

27017

Host IP Address (*)

Host IP or hostname of the MongoDB server.

192.168.1.100

Username (*)

Username for MongoDB authentication.

mongo_user

Password (*)

Password for authentication.

********

Database Name (*)

Name of the database where data will be written.

analytics_db

Collection Name (*)

Name of the collection where data will be written.

customer_orders

Save Mode

Write behavior: Append or Upsert.

Upsert

Enable SSL

Enable SSL for secure MongoDB connections. Requires different credentials.

true

Composite Keys (*)

Required only in Upsert mode. Comma-separated key fields used to identify records for update.

customer_id, order_id

Additional Parameters

Optional MongoDB connection parameters.

replicaSet=rs0&authSource=admin

Connection String (*)

Required only when Connection String option is chosen.

mongodb+srv://cluster0.mongodb.net

Note: Fields marked with (*) are mandatory. Meta Information fields vary based on the Connection Type selected.

Connection Type Variations

  • Standard: Requires Host IP, Port, Username, Password, Database, and Collection.

  • SRV: Simplifies replica set connections. Requires Username, Password, Database, and Collection.

  • Connection String: Uses a MongoDB URI string directly for authentication and connectivity.

Save Modes

Save Mode
Description

Append

Inserts all new records into the collection.

Upsert

Updates a record if it exists (based on Composite Keys) or inserts it if it does not exist.

Column Handling (Selected Columns)

The Selected Columns section allows renaming or changing the data type of columns before writing.

Field
Description
Example

Name

Name of the source column.

customer_id

Alias

Alias name for the column (optional).

cust_id

Column Type

Data type to cast the column into.

STRING

Additional Options:

  • Upload File: Upload CSV/JSON (≤2 MB) to auto-populate schema.

  • Download Data (Schema): Export schema mapping in JSON.

  • Delete Data: Clear column filter entries.

Workflow

  1. Configure Component

    • Fill in all required fields in the Meta Information tab.

    • Choose connection type (Standard, SRV, or Connection String).

    • Select Append or Upsert as the Save Mode.

  2. Save Configuration

    • Click the Save Component in Storage icon.

    • A confirmation message will appear on successful save.

  3. Activate Pipeline

    • Click the Activate Pipeline icon.

    • The PyMongo Writer will begin writing in-event data into the configured MongoDB collection.

Notes

  • Composite Keys are mandatory when using Upsert.

  • For SSL, ensure that proper certificates are uploaded in Admin Settings → Certificate Upload.

  • Schema mismatches between Spark and MongoDB may result in errors.

  • Use Append for bulk inserts and Upsert for incremental updates.

PreviousMongoDB WriterNextMachine Learning