Text Mining Reveals Hidden Value and Opportunities

Manufacturers, hospitals and public sector organizations discover strategic and tactical benefits by mining text-based documents

By Mary Crissey, SAS

I remember my fourth-grade math teacher telling me that it's impossible to add apples and oranges. She was wrong.

Now there's a way to throw apples and oranges into an equation without making a mess. The newest text mining technology enables decision makers to crunch words and numbers at the same time, yielding practical knowledge that can be leveraged for business growth.

Today, business information is gathered through multiple channels. And it arrives in a wide variety of forms. Transforming this torrent into manageable streams of business intelligence requires working with a mix of organized labeled data (structured data), as well as working with new records that typically require manual processing before being grouped by content topic (unstructured data).

The difficulties involved with processing disparate forms of data aren't likely to go away anytime soon. In fact, as more organizations become aware of the benefits that continuous innovation and agility offer, the desire for creating growth strategies will certainly intensify. The ever-growing size of data collections can be thought of as the fuel for fact-based decisions. When a company goes the extra mile to conduct high-quality analyses, it's like pouring in a fuel additive to deliver turbocharged valuable insights for developing business strategies.

Crunching text and traditional data together enables organizations to leap beyond standard goal-oriented search methodologies. With integrated text mining, nimble organizations leverage the power of electronic information to reveal gaps in the market and take action long before their competitors are even aware of the opportunities.

Text mining in action
American Honda, Dreyfus, Sub-Zero Freezer Co./Wolf Appliance Co., the City of Turin Tourism Office, the Australian Taxation Office and HP are just a few of the farsighted organizations that already have begun to discover the value of deploying advanced text mining capabilities.

Thanks to highly efficient systems for data storage and the reduced cost of memory, most organizations have amassed huge repositories of data. Despite advances in processing speeds and analytic techniques, these repositories sit largely untouched and unexplored, like vast undiscovered continents. Some of the databases contain feedback from customers reflecting their desires, opinions and interests. Unlike natural resources such as oil or gold, however, buried information is relatively easy to extract.

This buried treasure is often captured as words or text in a variety of languages and sentence structures. Before the advent of text mining technologies, deciphering patterns or trends hidden in this sort of nuanced information would have required a team of trained linguists. Converting such nuanced information into any sort of usable business intelligence was considered a Herculean task.

Better understand customer input
Today, global organizations such as large manufacturing companies with far-flung customer relationship management teams are faced with a steady deluge of text from e-mail, customer surveys, warranty claims forms, call reports, technician reports and dealer feedback. The database for HP's call center alone contains 300,000 records and grows daily.

"In the past, we lacked the ability to make much sense of the influx of data," says Randy Collica, Senior Business Analyst at HP. "With the volume of notes we have, we could not physically assign someone to read each data record and manually transform the freeform text comment string to structured fields so we could proceed in our data mining and statistical analysis project. There's just no way any one person can do that."

Collica and his team now use text mining to uncover underlying themes or concepts contained in large document collections. They also use text mining to automatically group documents into topical clusters and classify documents into predefined categories. By integrating text data with structured data, Collica and his colleagues are able to enrich their predictive modeling efforts and formulate business decisions on the basis of insightful customer information, instead of relying on "gut instinct" from their more experienced employees.

Before turning to text mining, the job of product classification was tedious, difficult and time-consuming. Collica and his colleagues had to pore over vast amounts of information. Their work was often hampered by data they didn't understand.

"We're not product experts," says Collica. "Before, it would take me several hours to research new products coming in just to know what bucket to categorize them in; but now it's all automated, so now I can easily take the new SKU numbers and analyze them. Text mining classifies them for me, so I don't spend hours doing unnecessary research, and now I do it all on a monthly basis in about 20 minutes."

By adding text to existing data mining investigations, companies can fold in behavior indicators to answer the difficult "why" questions buried in miscellaneous data to complement the "what" answers traditionally found by comparing current values with historical files contained in large document collections.

Text mining offers tactical benefits as well. HP, for example, provides technology solutions to consumers, businesses and institutions globally. HP's offerings span IT infrastructure, global services, business and home computing, imaging and printing. Information describing details of customer accounts is stored in various forms in various locations across the globe.

"Until recently, we hadn't been able to combine revenue and product data from our customer data warehouse with the textual information received through our call center," says Collica. The addition of text mining capabilities removed the apples/oranges dilemma and helped drive HP's customer relationship management efforts to an entirely new level.

Life-saving benefits
Text mining is also proving its value in the academic and not-for-profit sectors. At the University of Louisville in Kentucky, for example, researchers use text mining to highlight potential areas for cost savings and to discover new ways to improve patient care. Dr. Patricia Cerrito and her team of researchers use text mining to quantify and analyze text from hospital billings, medication orders and chart notes to support their research into cancer, genetics, heart disease and environmental health.

Cerrito can now examine notes made by physicians on thousands of patient charts to track outcome records, revealing, for example, that the prescription of certain medications can result in prolonged or shortened patient hospital stays.

The text analysis process generates a quantitative representation of text, and then distills this information and performs traditional data mining techniques, essentially pulling relevant variables from patient charts and making it practical for Cerrito to look at patterns in patient treatments and patient outcomes.

The benefits of Cerrito's text mining hit close to home when her husband, John Cerrito, underwent open-heart surgery at Jewish Hospital in Louisville, Kentucky. His surgical team members made medication decisions based, in part, on Cerrito's own research. Specifically, they relied on a recent study about the post-surgical outcomes of diabetic patients who underwent open-heart surgery and were prescribed antibiotics.

Her research detected a relationship between high glucose levels and the risk of infection. Text mining revealed a connection between medication and surgical recovery. When Cerrito witnessed her husband's positive outcome, she saw firsthand the immediate benefit of a decision based on the results of text mining.

Cerrito and her colleagues also use text mining to understand how consistent reporting practices can improve patient care and hospital accreditation rankings. By examining textual documents alongside large hospital databases, researchers such as Cerrito discover hidden – and sometimes critical – relationships between various physician practices and patient outcomes.

Even when text mining does not involve life-or-death situations, it can offer immediate benefits. In addition to providing validation for existing programs, it can shed light on challenges that are lurking beneath the surface. For instance, if you're a car maker with text mining capabilities and you suddenly notice a spike in complaints about rear door locks or balky transmissions, you can predict the prevalence of the problem across your customer base with reasonable accuracy – and quickly devise an appropriate response – before it becomes a worldwide embarrassment.

As the benefits of text mining continue to emerge, more organizations will pursue efforts to analyze both structured and unstructured data on a routine basis. Organizations with business intelligence technologies can share their text mining insights as Web reports or eye-catching graphical reports. Managers at various locations across global enterprises can then visualize the text findings immediately, without using survey analysts, behavioral experts, psychologists or linguists to decipher meaning buried in the text.

It's already clear that data mining offers companies a competitive advantage by delivering powerful predictive modeling capabilities. As new examples of successful text mining applications crop up daily, companies will eagerly welcome this emerging technology to help them discover unseen opportunities and create better forward-thinking policies. It's only a matter of time before the ability to analyze structured and unstructured data together is regarded as an essential competency for healthy organizations everywhere.

*This article originally appeared in the December 2006 edition of DM Review.

Bio: Mary Grace Crissey is the Analytics Marketing Manager at SAS, where she follows her passion for applying mathematics and advanced analytics to real-world challenges.