The ‘Big Data Era’ has arrived — multi-petabyte data warehouses, social media interactions, real-time sensory data feeds, geospatial information and other new data sources are presenting organisations with a range of challenges, but also significant opportunities. IDC believes that as CIOs start to adopt the new class of technologies required to process, discover and analyse these massive data sets that cannot be dealt with using traditional databases and architectures, it will become clear that the real value will be derived from the high-end analytics that can be performed on the increasing volumes, velocity and variety of data that organisations are generating – or Big Data analytics.
Based on IDC’s research in this space, here are three suggestions for CIOs in dealing with these issues:
Cloud Bursting. The private cloud journey will line up well with the enterprisewide analytical requirements highlighted in this research, but CIOs need to ensure that workload assessments are conducted rigorously and that risk is mitigated where possible. Critical to this approach will be the evaluation of cloud bursting capabilities from external vendors (i.e. Infrastructure as a service), particularly as organisations start to leverage more real-time analytics environments, to ensure that the use of infrastructure resources maps closely to demand – and that there are no issues in terms of performance and availability.
Analytical Appliance. In terms of delivery models, IDC has seen significant performance benefits from analytical appliances for customers that are dealing with the impact of Big Data. In addition, since the software is optimised and pre-integrated with appliances, the deployment timeframes are typically shorter. As part of a recent global survey of CIOs, 10% of the respondents indicated that they will be looking at analytical appliances as a delivery model in 2011. IDC also believes that the demand for reference architectures will rise as CIOs look to integrate these appliances within existing data warehousing environments. In line with this increased adoption of the analytical appliance as a delivery model, IDC believes that IT departments will allocate less budget towards technical skills (i.e. installation, configuration and management), and more on the high-end analytical skills needed to help drive the necessary business impact across multiple functions.
Enterprise Architecture. Enterprise analytics needs an enterprise architecture that scales effectively with growth – and the rise of Big Data analytics means that this issue needs to be addressed more urgently. Organisations need to look at creating a ‘high performance analytical environment’ that leverages in-database analytics, parallel processing as well as in-memory storage to deal with the increased volume, velocity and variety of data. Particularly, in terms of dealing with unstructured data, more attention needs to be paid to Hadoop – an open source software framework set up by Apache that allows for the distributed processing of large data sets across clusters of computers. However, there will be an ongoing tension between global standards and local requirements – and the use of Hadoop would be a good example of this. Another would be the ability to process mixed workloads (e.g. analytical and operational) in the same infrastructure environment such as the appliance that was mentioned earlier. CIOs need to consider ways in which they can deliver value in terms of solving specific business problems, while at the same time, being cognizant of global architecture standards and specifications. While certain global governance models will not allow for the usage of some of these technologies in a production environment, business expectations will force IT departments to re-assess the way the enterprise architecture agenda is utilised at a local level.
And what types of business problems will big data analytics solve, exactly? IDC believes that these use cases can be best mapped out across two of the Big Data dimensions – namely velocity and variety as outlined below.
This is an excerpt from the white paper, Big Data Analytics: Future Architectures, Skills and Roadmaps for the CIO – read the full report for more detail.