How old is your data warehouse? It’s a simple question and probably one you don’t think about much. The majority of production data warehouses are now 15-20 years old and probably very transactional centric. Over the years, you’ve probably remodeled “the house” more than a few times—adding some “rooms” and “upgrades” here and there. It’s starting to feel its age as more Business Intelligence requirements have been added, including Mobile applications and specialized analytics. And more and more ideas seem to show up in your inbox every day, especially Big Data questions.
Just because things are feeling a little stretched, you don’t just throw your existing and mature investment away. The good news is that with today’s technology, often related to the surge in Big Data technologies and ideas, there are new architectures to modernize these “grey-haired data veterans” and make them more responsive to business needs and in super-charged time-frames.
Some of the technology/platform shifts relate to the way companies are now analyzing information and enabling insights you didn’t think of a decade or two ago. This article gives you several ideas to think about as you look at the aging of your data warehouse and the new technologies that help you modernize those investments.
Some of you are IBM shops with your data warehouse primarily built around DB2. If you’re not, skip to idea #2. This first idea is all about the IBM environment.
The IBM DB2 Analytics Accelerator (IDAA) is a workload-optimized appliance that enables companies to integrate strategic business data into operational processes. By linking z/OS and the IDAA hardware-accelerated analytics, speeds for complex queries are delivered in unprecedented response times—and in a highly secure and available environment.
This kind of superior performance and scalability, with rapid appliance deployment, provides an ideal solution for complex analysis.
A key feature is its transparent integration into DB2, which allows running unchanged workload and queries that automatically benefit from faster response times and reduced load on the mainframe central processors.
IDAA integrates behind the application layer of a DB2 for z/OS environment, which results in important performance benefits:
With the arrival of new data sources and organizations who have a need to analyze everything, from text to streaming data, it became clear that the data warehouse had some limitations. Data warehouse architects can respond by tuning existing systems, but if you really need to incorporate a variety of new data types for Big Data, you may need to look at some of the new technology out there. As a result, many companies are turning to Hadoop to modernize their data warehouse and data integration architectures.
As most of you are aware, Hadoop is an open source framework for processing, storing and analyzing massive amounts of data. It was designed to handle petabytes and exabytes of data, both structured and unstructured, distributed over multiple nodes in parallel. You can leverage Hadoop to capture interesting data in different ways, store it as additional data types and enable exploration of data prior to normalization. The data can prepared for loading into your data warehouse by using MapReduce or one the many commercially available Hadoop-based tools.
So what does all this mean? Philip Russom, research director for data management at The Data Warehousing Institute, said in a report, “Hadoop is a wonderful complement to the data warehouse, but no one that has worked their way through it would see it as a replacement for the data warehouse.”
Hadoop is worth looking at in your modernization effort. With careful evaluation and modeling, a Hadoop platform alongside your data warehouse, can be used as a data management landing zone, a pre-processing area for ETL processes and as a queryable, low-cost archive platform.
More and more experts are saying that it is getting less likely that everything can be put into a data warehouse. The key is to make all the pieces work together.
Unlike your data warehouse(s), MPP analytic databases can quickly take in large amounts of primarily structured data with minimal data modeling required and can scale to handle many terabytes and even petabytes of data. These highly capable systems support near real-time results to complex SQL queries. This is something missing in Hadoop and can sometimes mean Hadoop isn’t the answer to support near real-time Big Data applications.
The fundamental characteristics of these MPP analytic databases include:
MPP analytic databases are not perfect for every Big Data application, but hybrid systems and all types of new vendor products are in development.
Over the past few years, the high adoption of cloud computing technology by enterprises has become one of the emerging trends in the data warehousing market. Cloud-based data warehousing is becoming viewed by many as an infrastructure design with a higher potential than traditional data warehouse deployment methods. The huge early success of Amazon Web Services Redshift, a cloud-based data warehouse, has caught many skeptics by surprise.
Some organizations are starting to view a private cloud for supporting their data warehouse and analytics projects—especially some of the newer or even departmental or vertical data mart efforts. Budget constraints for IT, insufficient time to build in-house systems, and cost advantages attached to hosted or on-demand software-as-a-service licensing are important factors responsible for the increased adoption of cloud-based data warehousing solutions by enterprises.
In addition, cloud services do not have overhead costs of space, power, and cooling that are generally incurred with the deployment of traditional in-house data warehouse solutions. Security issues concern some, but most of these issues are being removed by the various aggressive and powerful vendors coming into the marketplace.
This is one of the biggest developments in Big Data. In-database analytics addresses one of the biggest hassles of advanced analytics. Doing the analysis in the database, where the data resides, eliminates the costs, time and security issues associated with the old approach by doing the processing in the data warehouse itself.
In-database analytics changes this by moving the analytics algorithms and processing to where the data is stored. Elimination of data movement obviously removes the hours needed to move terabytes of information to near zero. The huge increases in data volumes in today’s data warehouses has made this capability very interesting to many IT organizations. As more organizations use data to answer important questions, the questions they ask are becoming more complex, demanding more sophisticated tools, more precise results and better performance.
All of these factors in combination have created the need for in-database processing. The introduction of the column-oriented databases, specifically designed for analytics, data warehousing and reporting, has helped make the technology possible. Several key vendors provide tools and platforms in this area.
Idea #6: Extend Your Data Warehouse with External Data Sources
There is a growing market of data virtualization vendors that allow you to easily extend the power of your data warehouse by accessing non-data warehouse and external data sources on an “as needed basis. This type of “extended virtual data warehouse” allows you to get quick access to infrequently referenced data sources without having to go through long delays of processing to get and load interesting, external data into your warehouse.
For example, many companies have current/historic sales data, along with pricing information. But think about the analytical possibilities if you were to take that data and gain additional insight by loading information about things that could drive sales such as weather, income tax distribution timeframes, regional or local population growth/decline, household demographics, and other information. You can virtually access this information and then decide to load permanently if it is needed on a more frequent basis. While this kind of information has been available for some time, there are newer types of information available and through vendors that build management tools to make the whole process more agile.
Spend some time this year and do some deep thinking about what you could do to “modernize” some of your data warehouse investments. With all the focus on Big Data, start thinking “hybrid” as you consider how to use some of the new breakthroughs in combination with your data warehouse. It will take some time and effort. There is a shortage of skilled technologists in some of these areas. But now is the time to get started.