Technical Requirements

1. Data Ingestion

• Data Platform should have Out of the Box capability (connectors) of integrating/interfacing with Cloud hosted and On-Premises Enterprise systems, manufacturing tools/application and external 3rd party systems.

• Data Ingestion should support connecting to data source using methods not limited to DB Connection (JDBC/ODBC), API, Data Replication, Event Brokers, Managed File Transfer & ETL/ELT process.

• Data Platform should support ingestion of data (Structured, unstructured, semi-structured) of any size and format without prior schema definition, data transformation and modelling.

• Ingestion process shall support data coming at different speed, from real time event streaming, to scheduled batch/Micro Batch imports.

• Support incremental (Change Data Capture) and ad-hoc data ingestion capabilities with a provision to handle Late Arrival of Data.

• Data platform shall offer out-of-the-box connectors for easier integration for other external platforms and legacy systems.

• Provide supported capabilities for creation and execution of batch ingestion and batch export flows (e.g., command line interface, Graphical User Interface).

2. Data Processing

2.1 Batch Data Processing

• Data platform should have out-of-the-box capability of processing the data (structured, semi-structured, unstructured) in batch mode.

• These includes not limited to Data cleansing, validations, transformations, aggregation, enrichment.

• Support incremental (Change Data Capture) and ad-hoc data processing capabilities.

• Provide scheduling capabilities with respect to batch processing jobs. Also support scheduling capabilities with 3rd party scheduling tools if required.

• Support integration with Open-Source tools and technologies.

• Support processing capabilities for large datasets On-Premise in Hybrid Mode

2.2 Realtime Data Processing

• Data platform should have the capability of processing the data (structured, semi-structured, unstructured) in real/near real time mode. These includes not limited to Data cleansing, validations, transformations, aggregation, enrichment.

• Processing capabilities shall support real time filtering, masking & processing (transformations, aggregations, enrichment etc.) of streaming data.

• Capabilities for in-memory processing for real time data.

• Integration capabilities with external sources providing streams of data.

• Processing capabilities of incoming events into more-meaningful complex events to be stored in the platform.

• Support integration with Open-Source tools and technologies.

3. Data Preparation

• Option to view sample data (10 K) in a paginated grid.

• Option to get a statistical profile of the data

• A quality bar that indicates the percent of valid, invalid and blank rows.

• 60+ transforms that can be performed on the data.

• Ability to view the changes in data, after each transform.

• Ability to undo/redo the transforms if changes are not acceptable.

• Option to write nested transforms using SQL transform.

• Option to view the list of transforms performed on the current data.

• Option to filter data by clicking on the profiling charts.

• Option to perform transforms on the filtered data.

• Option to export the steps to ETL /Pipeline so that it can be performed on full data.

4. Data Store

4.1 Data Lake

• Data platform should have the provision for a Data Lake with a storage tier able to store all (structured, semi-structured and unstructured) data in a cost-effective way while the platform should be horizontally scalable.

• Data Lake should have data zones categorized into Staging Layer (Raw Data), Curated Layer (Cleansed Data) and Processed Layer (transformed data). There should be provision for creating a sandbox layer within the data lake for running analytical workloads.

• Data Lake architecture should have Storage and Compute nodes decoupled which can be scaled up/down as needed.

• Data Lake should have a tiered storage architecture to store data in a cost-effective manner. Infrequently accessed data should be archived and stored in an archive-based storage.

• Provision of Querying the data directly from data lake without moving the data to Enterprise Datawarehouse.

• For this section, a strong Data Virtualization (Data Fabric) feature is required In the platform.

4.2 Enterprise Datawarehouse

• Data Lake should have integration with an Enterprise Datawarehouse Platform which can provide low latency random read/write/update to small amounts of data.

• Enterprise Datawarehouse should be effectively able to store aggregated data for reporting which is easily accessible for analysis through SQL based queries and seamlessly integrated with Business Intelligence tools provided by the Platform

• Storage and Compute clusters should be decoupled with dedicated nodes for processing and storage of data.

• Enterprise Datawarehouse should automatically scale with zero down time while provisioning new nodes.

• Enterprise Datawarehouse should have the capability of store the data compressed format to save storage cost and disk space.

• Enterprise Datawarehouse should have the capability of store the data compressed format to save storage cost and disk space.

5. Query Engines

• Different tools/options should be offered by the Data Platform to query and manipulate data in Data Lake and Enterprise Datawarehouse.

• Ability to query data in Data Lake through ad-hoc queries without loading into Enterprise Datawarehouse should be possible.

• Provision of SQL-based languages supported to query the data.

  Support different type of data can be queried (structured, semi-structured, unstructured).

• Provision for Query optimization functionality.

• Provision of user interface (e.g. web, CLI) for end users to query the data.

• Support data platform scalability in terms of concurrent users issuing queries.

6. Data Visualization

Provision for out of the box tools to build and deploy custom reports and dashboards.

• Provision for out of the box tools to build and deploy custom reports and dashboards.

• Provides an easy, flexible way to combine and analyse data across multiple tables and at different levels of detail without having to learn advanced database concepts or write custom SQL code(low code).

• Augmented Analytics Capability - Get explanations of a data point with a single click.

• Supports integration with Data Science Outputs and other Data Types.

• Able to provide analytic content to end users on an on-demand and scheduled basis

• Must support most commonly used mobile devices such as iPad, iPhone and Android devices through device browser and native mobile app.

• Flexibility to seamlessly integrate with Data Lake, Enterprise Datawarehouse and other application databases using Live/Direct connection with low latency to deliver sub-second performance on reports & dashboards.

• Out of the Box integration capabilities and data source connectors available to connect and extract data from the client’s applications and 3rd party applications/tools.

• Provision to extract into the Data Visualization tool (in-memory platform) from the data platform and applications to run queries and dashboards as required.

• Ability to seamlessly access and query data and create reports using self service capability both from data lake and Enterprise Datawarehouse.

• Ability to analyse and generate insights based on different data structures and data types.

• Support seamless transition/migration path of queries, reports and dashboards from a different environment to Production environment.

• Support hosting the data visualization tool in any public cloud of choice as well as on-Premises with the same features and capabilities supported.

• Seamlessly scale compute and storage capabilities based on the workloads and concurrency.

• Support publishing of reports and dashboards to a portal which can be governed and accessed by business users through authorized view.

• Ease of self-service data Preparation capability with no scripting or programming required with Licensing covered in Data Visualization Licenses only (no Separate Licensing)

7. Search

• Data Platform shall support full text search and faceted text search capabilities

• Search capability shall provide a rich user experience with auto-complete, smart suggestions, filtering and faceting features.

• Provide state of the art user interface and user-friendly options like filters which can improve search results.

• Data stored in Data Lake and Enterprise Datawarehouse can be searched (both structured & unstructured data) as well as future data sources that will be integrated.

• Provision of a Graphical user interface as Out of the Box capability to perform search activities

• Search functionality should be scalable and fault tolerant with the increasing number of users and doesn’t crash or lag with more data.

• Analytical capabilities offered out of the box that leverage the search functionality.

• Supports role-based security and document level security

8. Advanced Analytics & Data Science

• Provision of Open-Source analytical frameworks or tools that are provided out of the box as part of the Data platform (e.g., machine learning). [Providing Integrated Data Science Notebook Feature]

• Ability to create analytical models, perform data mining and apply statistical methods on data to generate predictive insights.

• Provisioning of a Sand Box environment to run analytical models which is scalable based on analytical workloads.

• Support for advanced analysis and modelling, leveraging optimized resources management to limit the tool needs while enabling computational heavy consumption for advanced tasks.

• Ability to support data preparation activities which is built in feature.

• Monitoring and providing a layer of governance to guarantee the continued quality of the analytical models.

• Capabilities in Deep Learning, with less dependencies on external libraries and hardware (e.g., GPU).

• Capabilities to support NLP & Computer Vision.

• Enables connection with other external systems and components.

• Combine technical capabilities and intuitive representations for the user to easily read and interpret the different diagnostics, explaining how models are performing.

• Support of advanced modelling capabilities, enabling the creation of the Deep Learning architecture and provision for high level of customization.

• Has a significant community using, supporting and documenting the framework

9. Data -As-A-Service

• Data Platform shall support data services for both inbound and outbound data requirements.

• Ability to extract all types of data from source systems processed and stored in storage volumes.

• Share/transfer the data to downstream systems, applications, or 3rd party on-premises or cloud platforms based on a time bound schedule.

  Share/transfer the data to downstream systems, applications, or 3rd party on-premises or cloud platforms in real time based on event driven approach.

• Ensure that the data is not removed from the Source/origin and can be accessed by multiple applications/end points.

• Exposing data including analytics model outcomes in real-time to the downstream systems.

• Support API (inbound and outbound) integration through an API gateway.

• Data and Digital Services for future technologies for IOT and Block Chain in future

10. Security Requirements

• Data platform provide data integrity ensuring no loss or corruption of data through its lifecycle

• Data platform should have encryption capabilities both for data-in-motion (data moved in the network) as well as data-at-rest (data stored in the Data platform repositories)

• Data platform supports granular levels of data encryption, such as file-level encryption, Format Preserving Encryption (FPE) or tokenization.

• Data platform should support data masking capabilities.

• Data platform should support Identity and Access Management through user authentication (Role Bases Access Control) and credential management capabilities.

• Permissions should be driven by policies based on least privilege principle.

• Data Platform should have the capability of setting up Multifactor Authentication (MFA).

• Users should be able to access the Data Platform using Single Sign On capability.

• Protect from common attacks not limited to DDoS attacks, SQL Injection, Cross Site Scripting

• Support End Point security to prevent detect and respond to threats.

• Support out-of-the-box capabilities to manage compliance with respect to GDPR (“General Data Protection Regulation”) , SOX and other regulatory requirements.

11. Networking Requirements

• Provide Network Topology Diagram of the proposed cloud-based data platform with details around each component.

• Provide bandwidth requirements to accommodate all platform users covering the typical response times based on the maximum number of users.

• Share options to support hybrid connectivity across on-premises and in-cloud Data Platform with an encrypted network connection.

• Share communication protocols used by the data platform.

• Data platform shall provide data integrity ensuring no loss or corruption of data through its lifecycle through its network.

• Support high throughput and lowest latency cloud network for data platform workloads. with faster response time.

• Provide details across how VPC networks can be optimized for cost, security and performance.

• Provision to provide lower latency access to data and decrease network egress costs.

• Support for API Gateway as a fully managed service to create, publish, maintain, monitor, and secure APIs at scale and types of API supported both inbound & outbound.

• Details around Network security features for the data platform to protect against possible threats and attacks not limited to DDoS attacks, SQL Injection, Cross Site Scripting.

• Provide details around network monitoring tools and alerts that can be configured.

• Ensure high availability of the data platform and fault tolerance.

12. Operational Requirements

• Supported Out-of-the-Box capabilities to manage, monitor and run the data platform.

• Provide Performance monitoring metrics (Business Metrics) not limited to CPU Usage, network utilization using a dashboard.

• Mechanism to detect and Trigger Events and Alerts as part of monitoring capabilities.

• Ability to collect and aggregate system logs for analysis and insights.

• Monitor inbound and outbound API calls.

• Create audit trails for resource usage.

• Manage and record configuration changes to the data platform

13. Hardware Sizing Requirements for Data Platform

• Describe hardware sizing requirements necessary to implement the proposed solution. Please include the details around of the hardware components and the recommended minimum configuration (processors, speed, memory, storage, operating system etc.) for each data platform component (ingestion, processing, storage (data lake, Datawarehouse, search, query engine, data visualization).

• Describe the typical hardware architecture & configuration (servers’ configuration, storage, bandwidth etc.)