Big Data
What it is and why it matters
Big data is a term that describes large, hard-to-manage volumes of data – both structured and unstructured – that inundate businesses on a day-to-day basis. But it’s not just the type or amount of data that’s important, it’s what organizations do with the data that matters. Big data can be analyzed for insights that improve decisions and give confidence for making strategic business moves.
History of Big Data
Big data refers to data that is so large, fast or complex that it’s difficult or impossible to process using traditional methods. The act of accessing and storing large amounts of information for analytics has been around for a long time. But the concept of big data gained momentum in the early 2000s when industry analyst Doug Laney articulated the now-mainstream definition of big data as the three V’s:
Volume. Organizations collect data from a variety of sources, including transactions, smart (IoT) devices, industrial equipment, videos, images, audio, social media and more. In the past, storing all that data would have been too costly – but cheaper storage using data lakes, Hadoop and the cloud have eased the burden.
Velocity. With the growth in the Internet of Things, data streams into businesses at an unprecedented speed and must be handled in a timely manner. RFID tags, sensors and smart meters are driving the need to deal with these torrents of data in near-real time.
Variety. Data comes in all types of formats – from structured, numeric data in traditional databases to unstructured text documents, emails, videos, audios, stock ticker data and financial transactions.
At SAS, we consider two additional dimensions when it comes to big data:
Variability
In addition to the increasing velocities and varieties of data, data flows are unpredictable – changing often and varying greatly. It’s challenging, but businesses need to know when something is trending in social media, and how to manage daily, seasonal and event-triggered peak data loads.
Veracity
Veracity refers to the quality of data. Because data comes from so many different sources, it’s difficult to link, match, cleanse and transform data across systems. Businesses need to connect and correlate relationships, hierarchies and multiple data linkages. Otherwise, their data can quickly spiral out of control.
Big data and IoT analytics foster economic growth, sustainability and well-being
Jakarta had vast amounts of unstructured data scattered across silos and multiple applications. To enhance public service, the city partnered with SAS to develop a one-stop super app for residents, businesses and government departments. Now, everyone can use the app to access public services, get emergency updates, report accidents and crimes, and much more. The city even uses SAS to build models that predict and mitigate flooding – reducing damage to lives, properties and businesses.
Why is big data important?
The importance of big data doesn’t simply revolve around how much data you have. The value lies in how you use it. By taking data from any source and analyzing it, you can find answers that 1) streamline resource management, 2) improve operational efficiencies, 3) optimize product development, 4) drive new revenue and growth opportunities and 5) enable smart decision making. When you combine big data with high-performance analytics, you can accomplish business-related tasks such as:
- Determining root causes of failures, issues and defects in near-real time.
- Spotting anomalies faster and more accurately than the human eye.
- Improving patient outcomes by rapidly converting medical image data into insights.
- Recalculating entire risk portfolios in minutes.
- Sharpening deep learning models' ability to accurately classify and react to changing variables.
- Detecting fraudulent behavior before it affects your organization.
Big Data in Today’s World
To derive insight from all their big data, organizations today need to combine data and AI. Together, big data and artificial intelligence technologies can change the way the world uses business information. Learn more about big data’s impact.
What's a data hero to do?
Who are data heroes? A data scientist analyzes and looks for insights in data. Data engineers build pipelines focused on DataOps. Data officers ensure data is reliable and managed responsibly. Synergy among roles drives analytics success.
What is a data lake and why does it matter?
Unlike its older cousin – the data warehouse – a data lake is ideal for storing unstructured big data like tweets, images, voice and streaming data. But it can store all types of data – any source, size, speed or structure.
Big data and cloud
Big data projects demand intense resources for data processing and storage. Working together, big data technologies and cloud computing provide a cost-effective way to handle all types of data – for a winning combination of agility and elasticity.
Deep learning craves big data because big data is necessary to isolate hidden patterns and to find answers without overfitting the data. With deep learning, the more good quality data you have, the better the results.
Who's Focusing on Big Data?
Big data is a big deal for industries. The onslaught of IoT and other connected devices has created a massive uptick in the amount of information organizations collect, manage and analyze. Along with big data comes the potential to unlock big insights – for every industry, large to small.
- Select an industry
- Retail
- Manufacturing
- Banking
- Health Care
- Education
- Small and Midsize Businesses
- Government
- Insurance
Retail
Customer relationship building is critical to the retail industry – and the best way to manage that is to manage big data. Retailers need to know the best way to market to customers, the most effective way to handle transactions, and the most strategic way to bring back lapsed business. Big data remains at the heart of all those things.
Manufacturing
Armed with insight that big data can provide, manufacturers can boost quality and output while minimizing waste – processes that are key in today’s highly competitive market. More and more manufacturers are working in an analytics-based culture, which means they can solve problems faster and make more agile business decisions.
Banking
With large amounts of information streaming in from countless sources, banks are faced with finding new and innovative ways to manage big data. While it’s important to understand customers and boost their satisfaction, it’s equally important to minimize risk and fraud while maintaining regulatory compliance. Big data brings big insights, but it also requires financial institutions to stay one step ahead of the game with advanced analytics.
Health Care
Patient records. Treatment plans. Prescription information. When it comes to health care, everything needs to be done quickly, accurately – and, in some cases, with enough transparency to satisfy stringent industry regulations. When big data is managed effectively, health care providers can uncover hidden insights that improve patient care.
Education
Educators armed with data-driven insight can make a significant impact on school systems, students and curriculums. By analyzing big data, they can identify at-risk students, make sure students are making adequate progress, and can implement a better system for evaluation and support of teachers and principals.
Small and Midsize Businesses
Between the ease of collecting big data and the increasingly affordable options for managing, storing and analyzing data, SMBs have a better chance than ever of competing with their bigger counterparts. SMBs can use big data with analytics to lower costs, boost productivity, build stronger customer relationships, and minimize risk and fraud.
Government
When government agencies are able to harness and apply analytics to their big data, they gain significant ground when it comes to managing utilities, running agencies, dealing with traffic congestion or preventing crime. But while there are many advantages to big data, governments must also address issues of transparency and privacy.
Insurance
Telematics, sensor data, weather data, drone and aerial image data – insurers are swamped with an influx of big data. Combining big data with analytics provides new insights that can drive digital transformation. For example, big data helps insurers better assess risk, create new pricing policies, make highly personalized offers and be more proactive about loss prevention.
Data-driven innovation
Today’s exabytes of big data open countless opportunities to capture insights that drive innovation. From more accurate forecasting to increased operational efficiency and better customer experiences, sophisticated uses of big data and analytics, along with AI and GenAI, propel advances that can change our world – improving lives, healing sickness, protecting the vulnerable and conserving resources. But we can't overlook the importance of always starting with good quality data and relying on trustworthy analytics and AI ethics to ensure the best outcomes for all of humanity.
How Big Data Works
Before businesses can put big data to work for them, they should consider how it flows among a multitude of locations, sources, systems, owners and users. There are five key steps to taking charge of this "big data fabric" that includes traditional, structured data along with unstructured and semistructured data:
- Set a big data strategy.
- Identify big data sources.
- Access, manage and store the data.
- Analyze the data.
- Make intelligent, data-driven decisions.
1) Set a big data strategy
At a high level, a big data strategy is a plan designed to help you oversee and improve the way you acquire, store, manage, share and use data within and outside of your organization. A big data strategy sets the stage for business success amid an abundance of data. When developing a strategy, it’s important to consider existing – and future – business and technology goals and initiatives. This calls for treating big data like any other valuable business asset rather than just a byproduct of applications.
2) Identify big data sources
- Streaming data comes from the Internet of Things (IoT) and other connected devices that flow into IT systems from wearables, smart cars, medical devices, industrial equipment and more. You can analyze this big data as it arrives, deciding which data to keep or not keep, and which needs further analysis.
- Social media data stems from interactions on Facebook, YouTube, Instagram, etc. This includes vast amounts of big data in the form of images, videos, voice, text and sound – useful for marketing, sales and support functions. This data is often in unstructured or semistructured forms, so it poses a unique challenge for consumption and analysis.
- Publicly available data comes from massive amounts of open data sources like the US government’s data.gov, the CIA World Factbook or the European Union Open Data Portal.
- Other big data may come from data lakes, cloud data sources, suppliers and customers.
3) Access, manage and store big data
Modern computing systems provide the speed, power and flexibility needed to quickly access massive amounts and types of big data. Along with reliable access, companies also need methods for integrating the data, building data pipelines, ensuring data quality, providing data governance and storage, and preparing the data for analysis. Some big data may be stored on-site in a traditional data warehouse – but there are also flexible, low-cost options for storing and handling big data via cloud solutions, data lakes, data pipelines and Hadoop.
4) Analyze the data
With high-performance technologies like grid computing or in-memory analytics, organizations can choose to use all their big data for analyses. Another approach is to determine upfront which data is relevant before analyzing it. Either way, big data analytics is how companies gain value and insights from data. Increasingly, big data feeds today’s advanced analytics endeavors such as artificial intelligence (AI) and machine learning.
5) Make intelligent, data-driven decisions
Well-managed, trusted data leads to trusted analytics and trusted decisions. To stay competitive, businesses need to seize the full value of big data and operate in a data-driven way – making decisions based on the evidence presented by big data rather than gut instinct. The benefits of being data driven are clear. Data-driven organizations perform better, are operationally more predictable and are more profitable.
Next Steps
Big data demands sophisticated data management technology to transform your analytics and AI programs into big opportunities. SAS has you covered.
SAS® Information Governance
Regardless of source, where the data is stored, or how large and complex it is, SAS Information Governance makes it faster and easier for data users to find, catalog and protect the big data that is most valuable for analysis. Metadata-oriented search results show detailed information about each data asset. In turn, business users can evaluate the data’s fitness for purpose with less reliance on IT while avoiding rework and making more informed choices.
Recommended Reading
- Series Meet the data scientist: Manuel David GarciaWhile he did study statistics, operations research and relational marketing, Data Scientist Manuel-David Garcia's "real school" came from on-the-job-training.
-
- Article Modernize your approach to big dataModernization is not just about systems and technology. In fact, technology changes can seem easy when compared with cultural change. Learn how to evolve your mind sets and leverage your data sets to accelerate knowledge throughout the organization.