Text Mining with SAS^® Text Miner

Features

Universal data access

• Access to numerous forms of textual data, including PDF, extended ASCII text, HTML and Microsoft Word.
• Web crawling capabilities.
• Ability to extract, transform and load textual data into a SAS data set for mining.

Support for multiple languages

• Total language list:  English, French, German, Italian, Portuguese, Spanish, Traditional Chinese and Simplified Chinese.
• Support for Latin-1, Double Byte Character and UTF-8 encodings.
• European languages (Latin-1 encoding): English, French, German, Italian, Portuguese and Spanish.
• Far-Eastern languages (Double Byte Character Support): Simplified Chinese and Traditional Chinese.
• Encoding support for Unicode UTF-8.

Self-documenting interface

• User-friendly interface eliminates manual coding with visual diagrams.
• Process flow diagrams can be modified, saved and shared with others.
• Flexible reporting allows results to be published in a concise HTML format.

Comprehensive text preprocessing capabilities

• Capture and distill the most important underlying information within a document collection.
• Default or customized stop lists for each language to remove terms with little or no informational value.
• Automated spelling correction.
• Stemming to identify root words.
• Part-of-speech tagging based on sentence context.
• Noun group extraction for identifying phrase-level concepts such as "competitive intelligence."
• User-defined multiword tokens, such as "point and click."
• User-customized and default synonym lists.
• Compound word splitting into distinct subterms.

Extensive feature extraction

• Broad customizable data dictionaries can extract particular pieces of information such as names of people, products, organizations, URLs and addresses.
• Extracted entities are then normalized and included in a matrix table.
• Entity extraction is available for English, French, German and Spanish.

Dimension reduction techniques

• Textual data is preprocessed into an information-rich matrix for application of powerful dimension reduction techniques.
• Rollup terms automatically identify the n highest-weighted terms in a document.
• Singular value decomposition (SVD) transforms each document into an n-dimensional subspace.

Text clustering algorithms

• Group documents based on their content.
• Expectation-maximization clustering groups documents using spatial clustering techniques.
• Hierarchical clustering using Ward’s agglomerative method facilitates automatic grouping of documents into taxonomies. Documents grouped into hierarchical clusters belong to one leaf cluster as well as its parent clusters.
• Cluster documents downstream in the Process Flow Diagram using K-means or SOM/Kohonen clustering.
• Profile clusters using additional structured data from original documents (age, purchase propensity, etc.).

Interactive training

• Provides a concise summary of results that includes document, term and cluster tables.
• Sorts term table by terms, term frequency, number of documents, weight and term role.
• Expands parent terms to identify child terms and their related statistics.
• Toggle between full and partial text view of the documents.
• Find the n most similar items for the selected document, term or cluster.
• Filter term(s) to show documents that contain them and clusters that contain those documents.
• Filter document(s) to show all terms in the documents, as well as revised cluster counts.
• Filter cluster(s) to show all documents in the filtered clusters, as well as the terms in those documents.
• Modify the keep and drop term lists.
• Treat selected terms as equivalent.
• Reweight terms using a different algorithm.
• Select the number of SVD dimensions.
• Browse concept links.
• Browse taxonomies.
• View the top n most representative terms for each cluster.
• Recluster anytime using a subset of documents or terms.
  

Document categorization

• Advanced techniques such as neural networks, memory-based reasoning, regression models and decision trees will assign documents to predefined categories.
• Seamlessly combine quantitative and qualitative data with text analysis to improve predictions.
• Graphically display performance assessments of multiple models to compare and select the best one to deploy as score code for categorizing new documents.