When it comes to being data-driven, organizations run the gamut with maturity levels. Most believe that data and analytics provide insights. But only one-third of respondents to a TDWI survey1 said they were truly data-driven, meaning they analyze data to drive decisions and actions.
Successful data-driven businesses foster a collaborative, goal-oriented culture. Leaders believe in data and are governance-oriented. The technology side of the business ensures sound data quality and puts analytics into operation. The data management strategy spans the full analytics life cycle. Data is accessible and usable by multiple people – data engineers and data scientists, business analysts and less-technical business users.
Based on a survey of business intelligence and data professionals across industries, TDWI identified five best practices for becoming a data-driven organization.
Five Data Management and Analytics Best Practices for Becoming Data-Driven
In a survey, TDWI found that one-third of organizations don’t govern their data – and fewer than 20 percent do any type of analytics governance. Governance is just one discipline that’s essential for becoming data-driven. Learn more in this checklist report from TDWI.
Build relationships to support collaboration
If IT and business teams don’t collaborate, the organization can’t operate in a data-driven way – so eliminating barriers between groups is crucial. Achieving this can improve market performance and innovation; but collaboration is challenging. Business decision makers don’t think IT understands the importance of fast results. IT doesn’t think the business understands data management priorities. Politics come into play.
Having clearly defined roles and responsibilities encourages teamwork. These roles should include: IT/architecture, business and “others” that manage various tasks on the business and IT sides (from business sponsors to DevOPs).
Make data accessible and trustworthy
Making data accessible – and ensuring its quality – are key to breaking down barriers, treating data as a corporate asset and becoming data-driven. Whether it’s a data engineer assembling and transforming data for analysis or a data scientist building a model, everyone benefits from trustworthy data that’s unified and built around a common vocabulary.
As organizations analyze new forms of data – text, sensor, image and streaming – they’ll need to do so across multiple platforms like data warehouses, Hadoop, streaming platforms and data lakes. Such systems may reside on-site or in the cloud. TDWI recommends several best practices to help:
- Establish a data integration and pipeline environment with tools that provide federated access and join data across sources. It helps to have point-and-click interfaces for building workflows, and tools that support ETL, ELT and advanced specifications like conditional logic or parallel jobs.
- Manage, reuse and govern metadata – that is, the data about your data. This includes size, author, database column structure, security and more.
- Provide reusable data quality tools with built-in analytics capabilities that can profile data for accuracy, completeness and ambiguity.
Provide tools to help the business work with data
From marketing and finance to operations and HR, business teams need self-service tools to speed and simplify data preparation and analytics tasks. Such tools may include built-in, advanced techniques like machine learning, and many work across the analytics life cycle – from data collection and profiling to monitoring analytical models in production. These “smart” tools feature three capabilities:
- Automation helps during model building and model management processes. Data preparation tools often use machine learning and natural language processing to understand semantics and accelerate data matching.
- Reusability pulls from what has already been created for data management and analytics. For example, a source-to-target data pipeline workflow can be saved and embedded into an analytics workflow to create a predictive model.
- Explainability helps business users understand the output when, for example, they’ve built a predictive model using an automated tool. Tools that explain what they’ve done are ideal for a data-driven company.
As organizations mature analytically, it’s important for the platform to support multiple roles in a common interface with a unified data infrastructure. This strengthens collaboration and makes it easier for people to do their jobs.
Consider a cohesive platform that supports collaboration and analytics
As organizations mature analytically, it’s important for their platform to support multiple roles in a common interface with a unified data infrastructure. This strengthens collaboration and makes it easier for people to do their jobs. For example, a business analyst can use a discussion space to collaborate with a data scientist while building a predictive model, and during testing. The data scientist can use a notebook environment to test and validate the model as it’s versioned and metadata is captured. The data scientist can notify the DevOps team when the model is ready for production – and they can use the platform’s tools to continually monitor the model.
Use modern governance technologies and practices
Governance – that is, rules and policies that prescribe how organizations protect and manage their data and analytics – is critical in learning to trust data and become data-driven. But a TDWI survey indicates that one-third of organizations don’t govern their data at all. Instead, many focus on security and privacy rules. Their research also indicates that fewer than 20 percent of organizations do any type of analytics governance, which includes vetting and monitoring models in production.
Decisions based on poor data – or models that have degraded – can have a negative effect on the business. As more people across an organization access data and build models, and as new types of data and technologies emerge (big data, cloud, stream mining), data governance practices need to evolve. TDWI found three features of governance software that can strengthen your data and analytics governance:
- Data catalogs, glossaries and dictionaries. These tools often include sophisticated tagging and automated procedures for building and keeping catalogs up to date – as well as discovering metadata from existing data sets.
- Data lineage. Data lineage combined with metadata helps organizations understand where data originated and track how it was changed and transformed.
- Model management. Ongoing model tracking is crucial for analytics governance. Many tools automate model monitoring, schedule updates to keep models current and send alerts when a model is degrading.
In the future, organizations may move beyond traditional governance council models to new approaches like agile governance, embedded governance or crowdsourced governance. But involving both IT and business stakeholders in the decision-making process – including data owners, data stewards and others – will always be key to robust governance at data-driven organizations.
1. Five Data Management and Analytics Best Practices for Becoming Data-Driven. A TDWI Checklist Report by Fern Halper.
- Next generation anti-money laundering: robotics, semantic analysis and AIAnti-money laundering taken to it's next level is sometimes referred to as AML 2.0 or AML 3.0. What does this next wave of AML technology look like? What can it do that you can’t do with traditional AML? See the results innovative financial institutions around the globe are already getting.
- Shut the front door on insurance application fraud!Fraudsters love the ease of plying their trade over digital channels. Smart insurance companies are using data from those channels (device fingerprint, IP address, geolocation, etc.) coupled with analytics and machine learning to detect insurance application fraud perpetrated by agents, customers and fraud rings.
- 3 steps for AI ethicsWill artificial intelligence benefit humanity or usher in a series of unintended consequences? AI ethics may be one way to ensure artificial intelligence is used for good.