Tips for successful data integration modernization
By Cindy Turner, Insights editor
A barrage of changes in the world of data has sent organizations scrambling to manage day-to-day requirements while preparing for the future of big data, analytics and real-time operations. To do it, most will need to rethink and modernize their data management infrastructures, teams and skills.
Because data integration (DI) plays such a broad role in capturing, processing and moving data – old and new alike – data integration modernization is a vital part of the solution. Here’s a summary of some tips from Philip Russom at TDWI to guide your data integration modernization efforts.
Complement traditional practices with a broader range of data ingestion techniques
The traditional extract, transform, load (ETL) approach to data integration usually runs overnight, with data refreshing on a 24-hour cycle. While this type of data integration is still valuable, much of today’s data needs to be handled differently. Consider machine data from sensors on vehicles, factory machines and handheld devices. This data may stream continuously, or at specified intervals.
As you work toward data integration modernization, you’ll need to be able to capture and process new types of data at different speeds and with different tools. This requires either new DI solutions or adjustments to older methods. By capturing fresh data in its original state, you can repurpose it to make it ready for reporting, analytics and operations. Luckily, today’s fast, scalable hardware and software make it practical to process data at whatever speed is needed.
Embrace new data prep practices and tools
Data preparation (data prep) goes by many names. Some call it data wrangling, data munging, data blending or “DI light.” Whatever it’s called, there are many tools to help you prepare data for other uses – tools for integration, profiling, quality, exploration, analytics and visualization.
Data preparation for analytics is a common technique that data scientists and analysts use when working with source data. As a result, it’s an important part of your arsenal for data integration modernization. Data scientists and analysts are on a mission to discover valuable nuggets of insight hidden in the data, such as outliers that might indicate a new customer segment. Modern data prep tools and techniques can help, because they’re fast, flexible and easy to use, and they encourage data exploration on the fly.
Use DI modernization to enable self-service access to new and big data
From data scientists to power users, many people rely on self-service functions for data access, data prep, report creation, visualization and analytics. Self-service capabilities allow freedom and spontaneity, and are advantageous for both business and IT users. Data integration modernization approaches should enhance self-service data access and prep by:
- Integrating data specifically for self-service, such as by feeding big data directly into data lakes, vaults and enterprise data hubs that are housed on Hadoop, relational databases or file systems.
- Relying on self-service tools to present business-friendly views of the data and to enable self-service access and preparation.
Add more right-time functions to your data integration modernization solutions
Effective approaches to data integration modernization let you work in real time, as well as at other speeds and frequencies required for specific business processes and databases. Modernizing DI to happen at the right time requires techniques like high performance, microbatch and data federation. Fortunately, modern data integration platforms are multitool environments that can handle data at the appropriate speed and frequency.
Want to know more? Get all seven tips from Philip Russom by downloading the TDWI Checklist Report, Modernizing Data Integration to Accommodate New Big Data and New Business Requirements.
