The origins of today’s OLAP products

OLAP products and applications have been around for much longer than most people think

You can contact Nigel Pendse, the author of this section, by e-mail on NigelP@olapreport.com if you have any comments or observations. Last updated on August 23, 2007.

The OLAP term dates back to 1993, but the ideas, technology and even some of the products have origins long before then.

APL

Multidimensional analysis, the basis for OLAP, is not new. In fact, it goes back to 1962, with the publication of Ken Iverson’s book, A Programming Language. The first computer implementation of the APL language was in the late 1960s, by IBM. APL is a mathematically defined language with multidimensional variables and elegant, if rather abstract, processing operators. It was originally intended more as a way of defining multidimensional transformations than as a practical programming language, so it did not pay attention to mundane concepts like files and printers. In the interests of a succinct notation, the operators were Greek symbols. In fact, the resulting programs were so succinct that few could predict what an APL program would do. It became known as a ‘Write Only Language’ (WOL), because it was easier to rewrite a program that needed maintenance than to fix it.

Unfortunately, this was all long before the days of high-resolution GUI screens and laser printers, so APL’s Greek symbols needed special screens, keyboards and printers. Later, English words were sometimes used as substitutes for the Greek operators, but purists took a dim view of this attempted popularization of their elite language. APL also devoured machine resources, partly because early APL systems were interpreted rather than being compiled. This was in the days of very costly, under-powered mainframes, so applications that did APL justice were slow to process and very expensive to run. APL also had a, perhaps undeserved, reputation for being particularly hungry for memory, as arrays were processed in RAM.

However, in spite of these inauspicious beginnings, APL did not go away. It was used in many 1970s and 1980s business applications that had similar functions to today’s OLAP systems. Indeed, IBM developed an entire mainframe operating system for APL, called VSPC, and some people regarded it as the personal productivity environment of choice long before the spreadsheet made an appearance. One of these APL-based mainframe products was originally called Frango, and later Fregi. It was developed by IBM in the UK, and was used for interactive, top-down planning. A PC-based descendant of Frango surfaced in the early 1990s as KPS, and the product remains on sale today as the Analyst module in Cognos Planning. This is one of several APL-based products that Cognos has built or acquired since 1999.

However, APL was simply too elitist to catch on with a larger audience, even if the hardware problems were eventually to be solved or become irrelevant. It did make an appearance on PCs in the 1980s (and is still used, sometimes in a revamped form called “J”) but it ceased to have any market significance after about 1980. Although it was possible to program multidimensional applications using arrays in other languages, it was too hard for any but professional programmers to do so, and even technical end-users had to wait for a new generation of multidimensional products.

Express

By 1970, a more application-oriented multidimensional product, with academic origins, had made its first appearance: Express. This, in a completely rewritten form and with a modern code-base, became a widely used contemporary OLAP offering, but the original 1970’s concepts still lie just below the surface. Even after 30 years, Express remains one of the major OLAP technologies, although Oracle struggled and failed to keep it up-to-date with the many newer client/server products. Oracle announced in late 2000 that it would build OLAP server capabilities into Oracle9i starting in mid 2001. The second release of the Oracle9i OLAP Option included both a version of the Express engine, called the Analytic Workspace, and a new ROLAP engine. In fact, delays with the new OLAP technology and applications means that Oracle is still selling Express and applications based on it in 2005, some 35 years after Express was first released. This means that the Express engine has lived on well into its fourth decade, and if the OLAP Option eventually becomes successful, the renamed Express engine could be on sale into its fifth decade, making it possibly one of the longest-lived software products ever.

More multidimensional products appeared in the 1980s. Early in the decade, Stratagem appeared, and in its eventual guise of Acumate (now owned by Lucent), this too was still marketed to a limited extent till the mid 1990s. However, although it was a distant cousin of Express, it never had Express’ market share, and has been discontinued. Along the way, Stratagem was owned by CA, which was later to acquire two ROLAPs, the former Prodea Beacon and the former Information Advantage DecisionSuite, both of which soon died.

System W

Comshare’s System W was a different style of multidimensional product. Introduced in 1981, it was the first to have a hypercube approach and was much more oriented to end-user development of financial applications. It brought in many concepts that are still not widely adopted, like full non-procedural rules, full screen multidimensional viewing and data editing, automatic recalculation and (batch) integration with relational data. However, it too was heavy on hardware and was less programmable than the other products of its day, and so was less popular with IT professionals. It is also still used, but is no longer sold and no enhancements are likely. Although it was released on Unix, it was not a true client/server product and was never promoted by the vendor as an OLAP offering.

In the late 1980s, Comshare’s DOS One-Up and later, Windows-based Commander Prism (later called Comshare Planning) products used similar concepts to the host-based System W. Hyperion Solution’s Essbase product, though not a direct descendant of System W, was also clearly influenced by its financially oriented, fully pre-calculated hypercube approach, which causes database explosion (so a design decision made by Comshare in 1980 lingered on in Essbase until 2004). Ironically, Comshare subsequently licensed Essbase (rather than using any of its own engines) for the engine in some of its modern OLAP products, though this relationship was not to last. Comshare (now Geac) later switched to the Microsoft Analysis Services OLAP engine instead.

Metaphor

Another creative product of the early 1980s was Metaphor. This was aimed at marketing professionals in consumer goods companies. This too introduced many new concepts that only became popular in the 1990s, like client/server computing, multidimensional processing on relational data, workgroup processing and object-oriented development. Unfortunately, the standard PC hardware of the day was not capable of delivering the response and human factors that Metaphor required, so the vendor was forced to create totally proprietary PCs and network technology. Subsequently, Metaphor struggled to get the product to work successfully on non-proprietary hardware and right to the end it never used a standard GUI.

Eventually, Metaphor formed a marketing alliance with IBM, which went to acquire the company. By mid 1994, IBM had decided to integrate Metaphor’s unique technology (renamed DIS) with future IBM technology and to disband the subsidiary, although customer protests led to the continuing support for the product. The product continues to be supported for its remaining loyal customers, and IBM relaunched it under the IDS name but hardly promoted it. However, Metaphor’s creative concepts have not gone and the former Information Advantage, Brio, Sagent, MicroStrategy and Gentia are examples of vendors covered in The OLAP Report that have obviously been influenced by it.

One other surviving Metaphor tradition is the unprofitability of independent ROLAP vendors: no ROLAP vendor has ever made a cumulative profit, as demonstrated by Metaphor, MicroStrategy, MineShare, WhiteLight, STG, IA and Prodea. The natural market for ROLAPs seems to be just too small, and the deployments too labor intensive, for there to be a sustainable business model for more than one or two ROLAP vendors. MicroStrategy may be the only sizeable long-term ROLAP survivor.

EIS

By the mid 1980s, the term EIS (Executive Information System) had been born. The idea was to provide relevant and timely management information using a new, much simpler user interface than had previously been available. This used what was then a revolutionary concept, a graphical user interface running on DOS PCs, and using touch screens or mice. For executive use, the PCs were often hidden away in custom cabinets and desks, as few senior executives of the day wanted to be seen as nerdy PC users.

The first explicit EIS product was Pilot’s Command Center though there had been EIS applications implemented by IRI and Comshare earlier in the decade. This was a cooperative processing product, an architecture that would now be called client/server. Because of the limited power of mid 1980s PCs, it was very server-centric, but that approach came back into fashion again with products like EUREKA:Strategy and Holos and the Web. Command Center is no longer on sale, but it introduced many concepts that are recognizable in today’s OLAP products, including automatic time series handling, multidimensional client/server processing and simplified human factors (suitable for touch screen or mouse use). Some of these concepts were re-implemented in Pilot’s Analysis Server product, which is now also in the autumn of its life, as is Pilot, which changed hands again in August 2000, when it was bought by Accrue Software. However, rather unexpectedly, it reappeared as an independent company in mid 2002, though it kept a low profile before being bought by SAP in 2006.

Spreadsheets and OLAP

By the late 1980s, the spreadsheet was already becoming dominant in end-user analysis, so the first multidimensional spreadsheet appeared in the form of Compete. This was originally marketed as a very expensive specialist tool, but the vendor could not generate the volumes to stay in business, and Computer Associates acquired it, along with a number of other spreadsheet products including SuperCalc and 20/20. The main effect of CA’s acquisition of Compete was that the price was slashed, the copy protection removed and the product was heavily promoted. However, it was still not a success, a trend that was to be repeated with CA’s other OLAP acquisitions. For a few years, the old Compete was still occasionally found, bundled into a heavily discounted bargain pack. Later, Compete formed the basis for CA’s version 5 of SuperCalc, but the multidimensionality aspect of it was not promoted.

Lotus was the next to attempt to enter the multidimensional spreadsheet market with Improv. Bravely, this was launched on the NeXT machine. This at least guaranteed that it could not take sales away from 1-2-3, but when it was eventually ported to Windows, Excel was already too big a threat to 1-2-3 for Improv’s sales to make any difference. Lotus, like CA with Compete, moved Improv down market, but this was still not enough for market success, and new development was soon discontinued. It seems that personal computer users liked their spreadsheets to be supersets of the original 1-2-3, and were not interested in new multidimensional replacements if these were not also fully compatible with their old, macro driven worksheets. Also, the concept of a small multidimensional spreadsheet, sold as a personal productivity application, clearly does not fit in with the real business world. Microsoft went this way, by adding PivotTables to Excel. Although only a small minority of Excel users take advantage of the feature, this is probably the single most widely used multidimensional analysis capability in the world, simply because there are so many users of Excel. Excel 2000 included a more sophisticated version of PivotTables, capable of acting as both a desktop OLAP, and as a client to Microsoft Analysis Services. However, the OLAP features even in Excel 2003 are inferior to those in OLAP add-ins, so there is still a good opportunity for third-party options as well.

By the late 1980s, Sinper had entered the multidimensional spreadsheet world, originally with a proprietary DOS spreadsheet, and then by linking to DOS 1-2-3. It entered the Windows era by turning its (then named) TM/1 product into a multidimensional back-end server for standard Excel and 1-2-3. Slightly later, Arbor did the same thing, although its new Essbase product could then only work in client/server mode, whereas Sinper’s could also work on a stand-alone PC. This approach to bringing multidimensionality to spreadsheet users has been far more popular with users. So much so, in fact, that traditional vendors of proprietary front-ends have been forced to follow suit, and products like Express, Holos, Gentia, MineShare, PowerPlay, MetaCube and WhiteLight all proudly offered highly integrated spreadsheet access to their OLAP servers. Ironically, for its first six months, Microsoft OLAP Services was one of the few OLAP servers not to have a vendor-developed spreadsheet client, as Microsoft’s (very basic) offering only appeared in June 1999 in Excel 2000. However, the (then) OLAP@Work Excel add-in filled the gap, and still (under its new snappy name, BusinessQuery MD for Excel) provided much better exploitation of the server than did Microsoft’s own Excel interface. Since then there have been at least ten other third party Excel add-ins developed for Microsoft Analysis Services, all offering capabilities not available even in Excel 2003. However, Business Objects’ acquisition of Crystal Decisions has led to the phasing out of BusinessQuery MD for Excel, to be replaced by technology from Crystal.

There was a rush of new OLAP Excel add-ins in 2004 from Business Objects, Cognos, Microsoft, MicroStrategy and Oracle. Perhaps with users disillusioned by disappointing Web capabilities, the vendors rediscovered that many numerate users would rather have their BI data displayed via a flexible Excel-based interface rather than in a dumb Web page or PDF. Microsoft is taking this further with PerformancePoint, whose main user interface for data entry and reporting is via Excel.

ROLAP and DOLAP

A few users demanded multidimensional applications that were much too large to be handled in multidimensional databases, and the relational OLAP tools evolved to meet this need. These presented the usual multidimensional view to users, sometimes even including a spreadsheet front-end, even though all the data was stored in an RDBMS. These have a much higher cost per user, and lower performance than specialized multidimensional tools, but they are a way of providing this popular form of analysis even to data not stored in a multidimensional structure. Most have not survived.

Other vendors expanded into what is sometimes called desktop OLAP: small cubes, generated from large databases, but downloaded to PCs for processing (even though, in Web implementations, the cubes usually reside on the server). These have proved very successful indeed, and the one vendor that sells both a relational query tool and a multidimensional analysis tool (Cognos, with Impromptu and PowerPlay) reports that the latter is much more popular with end-users than is the former.

Now, even the relational database vendors have embraced multidimensional analysis, with Oracle, IBM, Microsoft, the former Informix, CA and Sybase all developing or marketing products in this area. Ironically, having largely ignored multidimensionality for so many years, it seemed for a while that Oracle, Microsoft and IBM might be the new ‘OLAP triad’, with large OLAP market shares, based on selling multidimensional products they did not invent. In the event, Oracle and IBM stumbled badly, but Microsoft is now by far the largest OLAP vendor.

However, with Oracle’s purchase of Siebel (with its Analytics product) and Hyperion, it is now back in the OLAP game. It has also finally embedded its OLAP Option into the RDBMS with the 11g version. This makes the relational query engine aware of OLAP cubes, which it can use as a form of multidimensional materialized views. SQL queries can then be redirected to the cube which can replace a large number of conventional materialized views, potentially delivering faster queries and faster updates, without the client tool needing to be aware of the cube or doing anything special.

Lessons

So, what lessons can we draw from this 45-year history?

	Multidimensionality is here to stay. Even hard to use, expensive, slow and elitist multidimensional products survive in limited niches; when these restrictions are removed, it booms. We are about to see the biggest-ever growth of multidimensional applications.
	End-users will not give up their general-purpose spreadsheets. Even when accessing multidimensional databases, spreadsheets are the most popular client platform, and there are numerous third-party Excel add-ins for Microsoft Analysis Services to fill the gaps in Microsoft’s own offering. Most other BI vendors now also offer Excel add-ins as alternative front-ends. Stand-alone multidimensional spreadsheets are not successful unless they can provide full upwards compatibility with traditional spreadsheets, something that Improv and Compete failed to do.
	Most people find it easy to use multidimensional applications, but building and maintaining them takes a particular aptitude — which has stopped them from becoming mass-market products. But, using a combination of simplicity, pricing and bundling, Microsoft now seems determined to prove that it can make OLAP servers almost as widely used as relational databases.
	Multidimensional applications are often quite large and are usually suitable for workgroups, rather than individuals. Although there is a role for pure single-user multidimensional products, the most successful installations are multi-user, client/server applications, with the bulk of the data downloaded from feeder systems once rather than many times. There usually needs to be some IT support for this, even if the application is driven by end-users.
	Simple, cheap OLAP products are much more successful than powerful, complex, expensive products. Buyers generally opt for the lowest cost, simplest product that will meet most of their needs; if necessary, they often compromise their requirements. Projects using complex products also have a higher failure rate, probably because there is more opportunity for things to go wrong.

OLAP milestones

This page is part of the free content of The OLAP Report, but ten times more information is available to subscribers. You can register to access a free preview of a small sample of the large volume of subscriber-only information.

All information copyright ©2007, Business Application Research Center, all rights reserved.