Decide on the Module to be Used for Data Load

Data Acquisition is the extraction of data from various sources, using APIs, ETL/ELT processes, or manual data collection. BDB Platform offers a data pipeline and jobs for data extraction

Pipelines for Real-Time Data

Use a Pipeline when you need to handle stream data or real-time reads as data arrives. This method facilitates near real-time read and write operations. (Further details on pipeline data ingestion are provided below.)

Jobs for Batch Processing

Utilize a Job for batch processing, whether the process is scheduled or triggered by the success or failure of another job. The specific type of job should be selected based on the requirements:

• Spark Jobs: Select for scheduled read/write operations on databases (RDBMS), HDFS, BLOB, or S3 that involve basic transformations and aggregations.

• Python Jobs: Ideal for processing smaller datasets when code flexibility is a priority.

• PySpark Jobs: Employ this when dealing with large-scale datasets (big data) that exceed the memory capacity of a single machine, while still maintaining coding flexibility.

• On-Demand Python Jobs: Use this to initiate a Python job immediately via an API call with a payload. This trigger can also be initiated directly from a Pipeline.

• Script Executor Jobs: Use this to execute code snippets or scripts written in various languages (like Go, Julia, or Python). This job type allows for seamless execution of code fetched directly from a Git repository.

Note: Currently, Python is supported, and other languages will be supported as per market demand.