Data pipelines and streams

Trend 3. AI-driven data engineering to increase the velocity of innovation

Agility is critical to reduce time to market and remain competitive. While most organizations have adopted agility from a process perspective, data engineering techniques largely follow traditional ETL frameworks. They still rely on a requirement-driven approach, increasing the cycle time involved in provisioning new data for analytics needs. AI-driven engineering is the future of data engineering, where AI helps simplify the entire data engineering lifecycle to accelerate data availability for analytics. Leveraging ML in entity resolution, data cleansing, outlier detection, source-totarget mapping, and relationship discovery combined with industry semantics takes us closer to autonomous data engineering. The goal here is to enable engineered systems to ingest multiple input streams from disparate sources, learn from experience, and work collaboratively with both humans and machines in a symbiotic relationship.

A leading U.S. bank was migrating payments data from legacy to new systems, across regions. This required manually mapping several sources and target attributes. Infosys implemented a cognitive data mapper solution that works on ML-based data mapping techniques to automatically identify the source-to-target attribute mapping. The solution delivered 82% automation in mapping using value-based techniques and 60% automation in name-based mapping. This enabled the client to accelerate its migration journey across multiple regions.