OLAP with Mondrian

I got three things out of Julian Hyde's talk the the SDForum SIG last week. Julian is the lead developer on the Open Source Mondrian project and his talk was entitled "Building scalable OLAP applications with Mondrian and Pentaho".

The first thing is the nature of Mondrian. While most OLAP servers are database like servers that persistently store the OLAP cubes, Mondrian is a cache. At the back end, Mondrian uses JDBC to fetch persistent data from a database. At the front end Mondrian accepts queries in the MDX language and replies to these queries with data cubes. In the middle, Mondrian caches data cubes in memory so that it can respond to queries faster. Mondrian is a component. It is packaged as a single Java .jar file, and its schema is defined by an XML file. Mondrian needs both a back end database and a front end, either something like JPivot that can display OLAP cubes or an XML/A driver that can communicate MDX queries with a remote front end.

This brings us to the second thing. While there have been several attempts to create a standard interface to OLAP servers, none have really succeeded. Julian has spearheaded a new Open Source initiative called olap4j. The concept is that olap4j will be the JDBC of OLAP. It is based on JDBC, and uses the MDX language to express OLAP queries. MDX is the universal language of OLAP, just as SQL is the universal language of relational databases. The only curiosity is MDX is owned by Microsoft and that other implementations of MDX like Mondrian have to scramble to match each Microsoft release. I think that it is about time that MDX is managed by an independent standards committee.

Finally Julian talked about a "real-time" feature that he has recently added to Mondrian. This is a cache control API that allows you to invalidate parts of the Mondrian cache when new data is loaded into the underlying database. Having invalidated part of the cache, new queries fetch the new data and display up to date results. Initially the example looked complicated, and the audience reacted with some skepticism. On looking back at the example in the presentation, it seems less forbidding, however the long term goal should be to automatically manage the OLAP cache from an ETL tool. When pressed, Julian told us that the API is being used, for example, in applications for securities analysis.

In all, it was an satisfyingly informative evening.

The Future of Music Revisited

Michael Arrington has a great post in TechCrunch on "The Inevitable March of Recorded Music Towards Free". Most interesting is both the large response and the highly negative nature of some of the responses. I wrote about this subject under the title of "The Future of Music" a couple of years ago, partly inspired by an article in Wired about MySpace.

I completely agree with Michael Arrington, and I am surprised at the reaction of some performers. I expect that a performer would want to escape the clutches of their record company, who seems to feel that it is their right to keep 90% of their artists earnings. The fact that groups like Radiohead are escaping from their contract and planned to go free shows the way forward.

It is worth remembering that the Recording business is relatively recent. Big recording companies have only existed for 60 to 70 years. They came into their position of power by controlling the means of production, first records and then CDs. Now that they are no longer needed to reproduce music, they are irrelevant and should just fade away.

Software Architecture: Profession and Skills

Paul Preiss, president of the International Association of Software Architects (IASA) spoke on "Software Architecture: Profession and Skills" at the SDForum SAM SIG September meeting. Paul founded the IASA as a professional association for Information Technology (IT) Architects 4 years ago and since then it has grown to almost 7000 members in 50 countries.

Paul started off by talking about his career and how through a series of jobs at different companies he came to understand that the role of IT Architect is often unacknowledged and unappreciated. Also came the realization that IT Architect is an emerging profession and that the profession needs a professional body. One pressing reason for having a professional body is that he believes that the profession will be regulated within the next 5 years and that practitioners should have a say in writing the legislation.

The kernel of talk was about defining what an IT Architect is and does. We all understand the profession of being a Lawyer or Doctor or Accountant, thus we need a similar simple understanding of IT Architect. Paul attacked the problem from several different directions, eventually coming up with two words: Technology Strategy. Just as the Chief Financial Officer is in charge of directing the financial strategy of a business, the Chief Architect is in charge of directing the Technology Strategy. In particular, the role of the architect it to deploy just enough software to enable a competitive advantage for the business. I have been rolling this around in my mind for a couple of days and it seems to make sense.

The IT Architect has to at least understand a business problem and know the technologies that are available to solve it. The choice of the right technologies is not always easy. Technologies are changing fast, resources are limited and implementations do not always succeed. Moreover, other players may have their own agendas. For example, developers may want to use some new technology so that they can put it on their resume. Paul gave us a specific example. He came into a situation where the business was already using several object relational mapping tools and the developers wanted to adopt yet another one for a new project.

Paul ended his talk by plugging the IT Regional Conference in San Diego in October. Everyone who is involved in IT Architecture should check out the conference and the IASA

The Evolution of Web Analytics

We have not had a talk on Web Analytics for many years at the SDForum Business Intelligence SIG, so it was great to hear Stephen Oachs, founder and CTO of VisiStat, speak on "The Evolution of Web Analytics" at our September meeting. VisiStat is a two year old start-up that provides a web site performance measurement and analytics service (Software as a Service model) in the Small and Medium Business market.

As Stephen told us, first generation web analytics was about collecting data from web logs, integrating that with data from other sources and presenting historical results to IT specialists. The current generation, which he called web site performance management, collects data by page tagging, which entails adding a small snippet of JavaScript to each page. In practice the code snippet is added to a common page header or footer so it only needs to be added once to cover all pages in a site.

Page tagging collects more information than can be extracted from web server logs and it does not require difficult integration to make sense of the data. With better analysis software, the results of page tagging are ready to show directly to end users like the marketing and sales people who are responsible for the contents of the web site. Also, with page tagging we can see the data in real time, which allows the following of a user as they browse around the web site.

Real time access to information opens new doors. Stephen told us about a specific case where a bank became aware that it was subject to a phishing attack on its customers when the bank noticed an unusual change in the patterns of access to their web site. Similarly, click fraud can be detected by unusually high bounce rates from specific a key word. If detected in time the click fraud may be subverted by changing the price for the specific keyword. Finally, real time data provides web site availability monitoring, an additional service that for example, VisiStat offers for free.

The evening ended with a demo of the VisiStat product by Tina Bean, VisiStat Director of Sales and Marketing. The demo involved logging into real live customer web sites. You can see much of the same thing by visiting the VisiStat web site and looking at their live demo. VisiStat is a powerful tool for understanding how a web site is being used. At the same time we could see that it is designed from the end user perspective, so that typical small business user can use it effectively without needing support from an IT department or consultant. Most impressive is the fact that all the power of VisiStat is available to a small web sites for as little as $20 a month.

iPhones in the Bargain Bin

The big technology news of last week was the iPhone price reduction from $600 to $400 (plus tax). Pundits were crawling out of the woodwork to voice their opinion. Negative sentiment drove Apple's stock price down a bit. Here is my view of what it means.

The iPhone is worth $600. We know that because almost a million of America's brightest and best were willing to pay that price for for it. Now you can own it for $400. What a bargain!

Google Flight Simulator Earth

Last Year I wrote about Microsoft re-purposing their Flight Simulator program for their Live search engine. Now comes word that Google is fighting back by adding a flight simulator to Google Earth. While playing with a flight simulator can be fun, I still do not see what it has to do with search.

Vista DRM and Performance

Six months ago there was a fuss when Peter Guttman published his article on the Microsoft's new operating system Vista and how its DRM implementation could disrupt computer system performance when DRMed media is being played. Now the crows are coming home to roost.

In the last week there have been reports that Vista networking performance drops to 5% of maximum when the music player is active and 10% of maximum when any visual media player is active. This appears to happen even although the media player is paused, and even on multi-core systems where you might expect other cores to pick up the load. For example:

I have a Q6600, and while experiencing this crappy network performance, all 4 cores are practically idle.

So what will be the conclusion of all this? Microsoft has had many products that have knocked it out of the ball park, but every company produces a duff product now and again. It looks like Windows Vista could become the Windows ME of the 21st century.

Search and Analytics

"Search and Analytics" was the topic at the SDForum Business Intelligence SIG yesterday evening where Sajjad Jaffer, co-founder of Mercatrix spoke. The presentation covered the search space describing the latest trends in search with the intent of convincing us that Search is not over. Although one company dominates internet search now this was not always the case and it need not be the case in the future.

There are hundreds of startups with available products that are working on improving search. Sajjad talked us through the usual suspects like media search, semantic search and the use of crowdsourcing to improve relevancy. He also pointed out some interesting analytic resources on the web like Google Trends and Yahoo! Buzz.

The meat of the talk came a the end where Sajjad argued that traditional Business Intelligence spends too much time looking inwards to operational data from the business. For example, if the analytics looked out to what others are saying about their products, the business may be able to react more quickly than wait until product returns and slow sales told them they had a problem with the product.

While I agree that the web can be used for analytics, I am skeptical because there are also some huge hurdles. One problem is that large search companies can show off aggregated results of their most popular queries as in Yahoo! Buzz but any attempt to provide more narrow results that would be of use to a particular business will run into privacy and other problems.

I may be proved wrong. From what I can can make out of Mercatrix, its business is to provide internet search based analytics to companies.

Disruptive Technology?

The SDForum Emerging Tech SIG meeting last night on "eBay on Disruptive Technologies" did not turn our to be quite what what I expected, however it did contain a couple of interesting insights. Firstly Adam Trachtenberg, senior manager of Platform Evangelism, gave an overview of some of the things that eBay is doing to explore and protect itself from disruptive innovation. He described what they were doing with the wonderful word: "explorimenting".

An enthusiastic audience asked Adam about various of the companies like Skype, StubHub and Rent.com that eBay had recently bought and what their plans for them were. Unfortunately these groups are not in Adams bag and he could not say anything about them. However he did talk about a couple of initiatives that were in his bailiwick.

One of them is a social commerce effort that will allow people to share their eBay watch lists with their friends. The most interesting thing about this project is that it is built on Facebook. Last year Facebook turned their social networking site into a platform that allowed third party applications. As Adam explained, the Facebook platform makes it possible, even easy, for eBay to create social networking experiments like this. Recently there has been a lot of noise about the Facebook platform, and this experiment is another signpost as to how important it will become. The eBay Facebook application will launch in a couple of weeks.

Alan Lewis, a Technical Evangelist with the eBay Developers Program showed us the other eBay initiative a rich internet client application for eBay buyers called San Dimas. The idea is to explore giving eBay buyers a better user experience and also showcase the eBay API. The most interesting thing about this application is that it is written using the new Adobe Apollo/AIR/Flex platform. Adobe AIR is the next generation of Flash player, acquired when Adobe bought Macromedia in 2005. While AIR is still in beta, we have been hearing a lot about it recently, and I expect it to have a big impact when it launches. San Dimas has been in a private beta. We were the first people to see a public demonstration of it.

So Busted

In May I wrote about Fake Steve Jobs and the Vallywag campaign to unmask him. Then I remarked that given the Vallywag reputation for Bulldog journalism they would not get very far. Now I stand corrected.

On Sunday August 5th at 5:56 PST, Vallywag broke the news about how Forbes Editor Daniel Lyons is the Fake Steve Jobs, having read about it in the New York Times, like all the rest of us. Since then they have published 10 further posts on the subject in 24 hours as if quantity could make up for quality. In the mean time FSJ's put down of Vwag as Dr. Evil and Bigglesworth is much funnier than anything they have published in the last 6 months.