> For the complete documentation index, see [llms.txt](https://docs.bdb.ai/pre-sales/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.bdb.ai/pre-sales/data-platform-evaluation-criteria/ingestion-process.md).

# Ingestion Process

<table><thead><tr><th width="174">Requirement</th><th width="161">Evaluation</th><th>Remarks</th></tr></thead><tbody><tr><td>Change data capture</td><td>medium</td><td>Integration with thirdparty tools like Debezium can be provided</td></tr><tr><td>Scheduled ingestion</td><td>very high</td><td>Yes, it is avaliable off-the-shelf.<img src="https://4072512490-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FlLFg37sm5677zgZ4m8et%2Fuploads%2FwNemgeKUE3HntLwWijav%2Fimage.png?alt=media&amp;token=85f102ba-22bc-4b95-8c0b-88a1f3d3893c" alt=""></td></tr><tr><td>Minimise fields</td><td>very high</td><td>Field minimization can be defined through the Data Preparation tool, and published to be used in the live Data Pipelines.</td></tr><tr><td>Filter by lookup</td><td>very high</td><td>Yes, it is a standard component.</td></tr><tr><td>Filter by consent</td><td>very high</td><td>It can be achieved via API integration with consent system, or through consent database lookup.</td></tr><tr><td>Anonymise fields</td><td>very high</td><td>Standard anonymization available via Data Preparation option or the Spark SQL component.</td></tr><tr><td>Compose Ingestion Processors</td><td>very high</td><td>Drag and drop based low-code platform</td></tr><tr><td>Ingestion Fault tolerant</td><td>very high</td><td>Ability to track faults and initiate sub process</td></tr><tr><td>Bootstrap + updates</td><td>very high</td><td>Can define pipeline to load historic data and subsequent updates as per the data load strategy</td></tr><tr><td>Reports + Metrics</td><td>high</td><td>Data Pipeline generates metric report about every process, like Memory used, CPU used, no. of records processed, etc.<img src="https://4072512490-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FlLFg37sm5677zgZ4m8et%2Fuploads%2FcctoedrLLwjM9MI6l0wd%2Fimage.png?alt=media&amp;token=9baf901c-dfa5-419f-b014-5618d240bd88" alt=""></td></tr><tr><td>performance impact threshold</td><td>high</td><td>Configurable compute resource allocation and instances to scale up</td></tr><tr><td>Secrety Management integration</td><td>very high</td><td>All secrets are stored in the Kubernestes secrets, platform provides direct integration with this.</td></tr><tr><td>Data Catalogue integration</td><td>very high</td><td>Platform automatically generate data catalog from the underlying meta data.</td></tr><tr><td>Visual interface</td><td>very high</td><td>Pipeline studio has drag and drop based visual interface, based on No-code/low-code approach.<img src="https://4072512490-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FlLFg37sm5677zgZ4m8et%2Fuploads%2FmKwA15sWKaJ41Fkz3kdh%2Fimage.png?alt=media&amp;token=d190bb41-d866-43aa-9a89-7166c75dcdf9" alt=""></td></tr><tr><td>Ingestion Manifest file</td><td>very high</td><td>It is achievable via internal metadata.</td></tr><tr><td>CI/CD Pipelines Integration</td><td>high</td><td>Yes, it provides facility to check-in and check-out Pipeline definitions and metadata to GIT Lab.</td></tr><tr><td>Ingestion Access Management</td><td>very high</td><td>Data Pipeline supports RBAC.</td></tr><tr><td>Ingestion Audit Logs</td><td>very high</td><td>Logs can be pushed to thirdparty log monitoring systems like Datadog, Promethues, etc.</td></tr></tbody></table>