OLAP Vendors Capitulate

I did some research on where the Business Intelligence market was heading and came to the conclusion that there has been complete capitulation in the marketplace. The table to the right shows the market share of the top 10 OLAP vendors in 2006 (courtesy of The OLAP Report).

In 2007 the following events took place. Oracle bought Hyperion. Cognos bought Applix. Business Objects bought Cartesis. SAP announced that it is buying Business Objects. IBM announced that it is buying Cognos.

In essence, the top 10 OLAP vendors which included several important Independent Software Vendors (ISV) is being replaced by the top 4 enterprise software vendors: IBM, Microsoft, Oracle and SAP.

There are two independent vendors left in the list and neither is going to break out. Microstrategy is in the ROLAP niche, suitable for big gnarly problems that need a lot of money. Microstrategy could make a good fit with the newly independent Teradata another company in a similar niche. Infor is a private company that has been rolling up second tier software vendors in the same way that Computer Associates did in the 80's and 90's. It is the new "home for old software".

Fake Steve Jobs, Tech Journalist

Someone in the audience put it best when he said that he uses "The Secret Diary of Steve Jobs" as his primary source of tech news. This was at the Computer History Museum last Tuesday evening when Dan Lyons AKA Fake Steve Jobs (FSJ) appeared on stage with former Apple Evangelist Guy Kawasaki, and Brad Stone the journalist who outed him. In the meeting, Dan Lyons seemed to be a little confused that anyone would think that his column is a useful commentary on tech events, but I know what that audience member means.

I too use FSJ as a primary source of my tech news and commentary. While the blog is intended as a satire, Dan Lyons is a knowledgeable tech writer and he gets the details right. Moreover, the format means that he can be perfectly frank and even rudely negative as he often is. I get more information out of 200 word rant from FSJ on a subject such as Facebook valuation that I get out of a perfectly balanced 1500 page article in the NYT or WSJ. What is more, a spoonful of humor sugar helps the medicine go down.

There is the question of bias. Truth be told, I find it much easier to deal with information when the bias is clear and on the surface rather than cleverly buried as it is in the 'balanced' articles written by Journalists. Moreover, Journalists need to make nice with their subjects so that they will continue to get access in the future, so they tend to backpedal on the negative, or at least bury it in the n'teenth paragraph. On the other hand, FSJ in character says what he thinks, and the truth comes straight out.

The event itself was well worth attending. Dan is a very funny guy and he had us laughing for nearly two hours, mocking everyone from the Appletards in the front row to Megan McCarthy of Vallywag at the back. There were at least 200 in the audience, including several notables such as well known tech writer John Markoff and iJustine.

A Developer Notebook

For some time I have wanted to write technical posts on specific programming and programming languages topics. While I could post them in this blog, they would not really fit with the lighter hearted and wider ranging posts here. So I started a new blog called "A Developer Notebook" for the technical posts. When you look at the content, there should be no mistaking the goals and purposes of these two blogs. Enjoy!

The Facebook Platform Internals

Ari Steinberg and Charlie Cheever of Facebook gave a fascinating talk about the design of the Facebook platform when they spoke on Tuesday to the SDForum Web Services SIG. As might be expected, the presentation drew a large audience including renowned Facebook fanboy Dave Maclure.

The Facebook platform allows outside developers to build applications and offer them to Facebook users. These applications can reach into Facebook data to do their work so security and privacy concerns are paramount. Also with Facebook approaching 50 million users and reportedly 85,000 applications, performance and scaling are big concerns.

Ari spoke first on the Facebook API. After looking at other Web 2.0 APIs and finding them wanting, they decided to use a query language approach which would give applications much greater flexibility and at the same time have the possibility of being more efficient by only fetching data that is needed. The language, FBQL is basically a simplified and restricted version of SQL.

The linguistic approach is also used in Facebook Markup Language (FBML) the language that applications and users use to define and customize their pages. As Charlie told us, it is much easier to validate and sanitize markup and style sheets from a parse tree than with regular expressions or any other technique. Also the result is much less likely to have the security loopholes that seem to plague so many Web 2.0 sites.

OLAP with Mondrian

I got three things out of Julian Hyde's talk the the SDForum SIG last week. Julian is the lead developer on the Open Source Mondrian project and his talk was entitled "Building scalable OLAP applications with Mondrian and Pentaho".

The first thing is the nature of Mondrian. While most OLAP servers are database like servers that persistently store the OLAP cubes, Mondrian is a cache. At the back end, Mondrian uses JDBC to fetch persistent data from a database. At the front end Mondrian accepts queries in the MDX language and replies to these queries with data cubes. In the middle, Mondrian caches data cubes in memory so that it can respond to queries faster. Mondrian is a component. It is packaged as a single Java .jar file, and its schema is defined by an XML file. Mondrian needs both a back end database and a front end, either something like JPivot that can display OLAP cubes or an XML/A driver that can communicate MDX queries with a remote front end.

This brings us to the second thing. While there have been several attempts to create a standard interface to OLAP servers, none have really succeeded. Julian has spearheaded a new Open Source initiative called olap4j. The concept is that olap4j will be the JDBC of OLAP. It is based on JDBC, and uses the MDX language to express OLAP queries. MDX is the universal language of OLAP, just as SQL is the universal language of relational databases. The only curiosity is MDX is owned by Microsoft and that other implementations of MDX like Mondrian have to scramble to match each Microsoft release. I think that it is about time that MDX is managed by an independent standards committee.

Finally Julian talked about a "real-time" feature that he has recently added to Mondrian. This is a cache control API that allows you to invalidate parts of the Mondrian cache when new data is loaded into the underlying database. Having invalidated part of the cache, new queries fetch the new data and display up to date results. Initially the example looked complicated, and the audience reacted with some skepticism. On looking back at the example in the presentation, it seems less forbidding, however the long term goal should be to automatically manage the OLAP cache from an ETL tool. When pressed, Julian told us that the API is being used, for example, in applications for securities analysis.

In all, it was an satisfyingly informative evening.

The Future of Music Revisited

Michael Arrington has a great post in TechCrunch on "The Inevitable March of Recorded Music Towards Free". Most interesting is both the large response and the highly negative nature of some of the responses. I wrote about this subject under the title of "The Future of Music" a couple of years ago, partly inspired by an article in Wired about MySpace.

I completely agree with Michael Arrington, and I am surprised at the reaction of some performers. I expect that a performer would want to escape the clutches of their record company, who seems to feel that it is their right to keep 90% of their artists earnings. The fact that groups like Radiohead are escaping from their contract and planned to go free shows the way forward.

It is worth remembering that the Recording business is relatively recent. Big recording companies have only existed for 60 to 70 years. They came into their position of power by controlling the means of production, first records and then CDs. Now that they are no longer needed to reproduce music, they are irrelevant and should just fade away.

Software Architecture: Profession and Skills

Paul Preiss, president of the International Association of Software Architects (IASA) spoke on "Software Architecture: Profession and Skills" at the SDForum SAM SIG September meeting. Paul founded the IASA as a professional association for Information Technology (IT) Architects 4 years ago and since then it has grown to almost 7000 members in 50 countries.

Paul started off by talking about his career and how through a series of jobs at different companies he came to understand that the role of IT Architect is often unacknowledged and unappreciated. Also came the realization that IT Architect is an emerging profession and that the profession needs a professional body. One pressing reason for having a professional body is that he believes that the profession will be regulated within the next 5 years and that practitioners should have a say in writing the legislation.

The kernel of talk was about defining what an IT Architect is and does. We all understand the profession of being a Lawyer or Doctor or Accountant, thus we need a similar simple understanding of IT Architect. Paul attacked the problem from several different directions, eventually coming up with two words: Technology Strategy. Just as the Chief Financial Officer is in charge of directing the financial strategy of a business, the Chief Architect is in charge of directing the Technology Strategy. In particular, the role of the architect it to deploy just enough software to enable a competitive advantage for the business. I have been rolling this around in my mind for a couple of days and it seems to make sense.

The IT Architect has to at least understand a business problem and know the technologies that are available to solve it. The choice of the right technologies is not always easy. Technologies are changing fast, resources are limited and implementations do not always succeed. Moreover, other players may have their own agendas. For example, developers may want to use some new technology so that they can put it on their resume. Paul gave us a specific example. He came into a situation where the business was already using several object relational mapping tools and the developers wanted to adopt yet another one for a new project.

Paul ended his talk by plugging the IT Regional Conference in San Diego in October. Everyone who is involved in IT Architecture should check out the conference and the IASA

The Evolution of Web Analytics

We have not had a talk on Web Analytics for many years at the SDForum Business Intelligence SIG, so it was great to hear Stephen Oachs, founder and CTO of VisiStat, speak on "The Evolution of Web Analytics" at our September meeting. VisiStat is a two year old start-up that provides a web site performance measurement and analytics service (Software as a Service model) in the Small and Medium Business market.

As Stephen told us, first generation web analytics was about collecting data from web logs, integrating that with data from other sources and presenting historical results to IT specialists. The current generation, which he called web site performance management, collects data by page tagging, which entails adding a small snippet of JavaScript to each page. In practice the code snippet is added to a common page header or footer so it only needs to be added once to cover all pages in a site.

Page tagging collects more information than can be extracted from web server logs and it does not require difficult integration to make sense of the data. With better analysis software, the results of page tagging are ready to show directly to end users like the marketing and sales people who are responsible for the contents of the web site. Also, with page tagging we can see the data in real time, which allows the following of a user as they browse around the web site.

Real time access to information opens new doors. Stephen told us about a specific case where a bank became aware that it was subject to a phishing attack on its customers when the bank noticed an unusual change in the patterns of access to their web site. Similarly, click fraud can be detected by unusually high bounce rates from specific a key word. If detected in time the click fraud may be subverted by changing the price for the specific keyword. Finally, real time data provides web site availability monitoring, an additional service that for example, VisiStat offers for free.

The evening ended with a demo of the VisiStat product by Tina Bean, VisiStat Director of Sales and Marketing. The demo involved logging into real live customer web sites. You can see much of the same thing by visiting the VisiStat web site and looking at their live demo. VisiStat is a powerful tool for understanding how a web site is being used. At the same time we could see that it is designed from the end user perspective, so that typical small business user can use it effectively without needing support from an IT department or consultant. Most impressive is the fact that all the power of VisiStat is available to a small web sites for as little as $20 a month.

iPhones in the Bargain Bin

The big technology news of last week was the iPhone price reduction from $600 to $400 (plus tax). Pundits were crawling out of the woodwork to voice their opinion. Negative sentiment drove Apple's stock price down a bit. Here is my view of what it means.

The iPhone is worth $600. We know that because almost a million of America's brightest and best were willing to pay that price for for it. Now you can own it for $400. What a bargain!

Google Flight Simulator Earth

Last Year I wrote about Microsoft re-purposing their Flight Simulator program for their Live search engine. Now comes word that Google is fighting back by adding a flight simulator to Google Earth. While playing with a flight simulator can be fun, I still do not see what it has to do with search.