Analytics at Work

Analytics has become a major driving force for competitive advantage in business. The new book "Analytics at Work: Smarter Decisions, Better Results" by Thomas H. Davenport, Jeanne G. Harris and Robert Morison discusses what analytics can do for a business, how to manage analytics and how to make a business more analytical.

Analytics at Work has a useful introductory chapter and then divides into two parts. The first part discusses five major aspects of analytics in a business environment. The second part looks at the lifecycle of managing analytics in a business. The organization is good and there is no overlap between the topics in each part, however the order in which the information is presented seems designed to put the reader off.

The first part starts with a plodding chapter on what needs to be done to get the data organized and related topics, followed by a diffuse chapter called Enterprise. The interesting chapters in this part are the last two chapters. The Targets chapter discusses the important topic of picking targets for analytics. The Analysts chapter discusses how to effectively employ and organize analysts in a large enterprise. Similarly the second part of the book starts with a plodding chapter on how to Embed Analytics in Business Processes, followed by much more inspiring chapters on building an analytical culture, and the need to continually review a business comprehensively as part of an analytics push. If you find yourself stuck reading the book, try skipping to one of the interesting chapters that I have indicated.

Scattered throughout the book are many useful tools. In the introductory chapter there are the six key questions that an analyst asks. We come back to these questions from several places in the book. Running throughout the book is a five step capability maturity model for judging how analytical an organizations is and showing the path to making the organization more analytical. Each chapter in the first part ends with a discussion on how to take that aspect of the organization through the five steps.

It is important to understand the target audience. The book is aimed at senior management and executives, particularly in large enterprises. While the book contains many brief case studies as inspiration and it touches on all the important management issues that need to be considered, it does not go into great depth about what analytics is or the specific analytical techniques and how they can be used. This is not a book for analysts, unless they have ambitions to grow their career beyond analytics. I recommend this book to anyone in the target audience who wants to grow their organizations analytics capabilities.

Data Management in the Cloud

Over the last couple of years, I have seen several presentations on the computing Cloud and how it is the next big thing. I realized that I still have a lot to learn from Daniel Graham's presentation "Data Management in the Cloud" at the July meeting of the Business Intelligence SIG. Dan leads Active Data Warehouse marketing programs for Teradata. If you have been living under a rock and do not know what cloud computing is, Wikipedia has a reasonable explanation. Dan distinguished between the public cloud as a rentable computing resource like Amazon's Elastic Computing Service and a private cloud which is your businesses computing resources in a datacenter behind the company firewall which uses virtualization software like VMWare to allow many applications to share hardware.

The big picture that Dan painted is that cloud computing is coming and that you need to get ready for it. By 2015, 20% of computing resources worldwide will be in the cloud. Start now by getting experience with the cloud to find out what works, what needs to be changed to make it work and what does not work. Teradata has been experimenting with cloud computing and is working with hardware and software vendors like VMWare and Amazon to ensure that Teradata database systems work well in the cloud. Informatica is another example of a software vendor that is working to ensure that their data integration software works well in the cloud and between clouds. NetFlix is an example of a company that has adopted cloud computing and recently announced that they were moving all their movie hosting into the Amazon computing cloud. The US Government is the leading user of cloud services having moved much of their computing needs into the cloud.

Cloud computing uses commodity hardware, which combined with the overhead of virtual machine software will not give you the best performance, however it is "good enough" for most applications. Dan took the well known quote from the movie Forrest Gump and bent it to his needs. “Clouds are like a box of chocolates. You never know what you're gonna get.” There is some high end software that is not suitable for cloud computing, the main problem coming from high IO requirements. The size and capabilities of a cloud computing host is often optimized to run a single instance Oracle database doing OLTP. In practice most applications are less demanding than this.

There were many other interesting tidbits in the presentation. Here are some examples. It is more expensive to get data out of a cloud than to bring it in. Why is unknown, but something to take into consideration when using a cloud. An interesting application for cloud computing is what Dan called "Workload Isolation". The idea is that when you have partners or consultants who need access to your data it is often preferable to put the data they need in the cloud rather than let them inside your firewall. In all the examples that Dan showed of Business Intelligence applications in the cloud, he talked about a Data Mart with the implication that a full Enterprise Data Warehouse was too large and demanding an application for the cloud for now.

The slides from the presentation are available at the SDForum Business Intelligence SIG web site.

Another Angle on the iPhone Woes

In all the discussion over the antenna problems with the new Apple iPhone 4, there is one thing that I have not heard, and that is how few product lines that Apple has. The iPhone 4 is a prime example. It comes in two memory sizes and we are promised a second color real soon now. On the Apple site you can still buy the previous generation iPhone 3, but that still makes it 3 models available with another one or two to come. Compare this with BlackBerry which has six ranges and thirteen models. Blackberry is restrained in the number of models it offers compared to the likes of Motorola, Samsung or Nokia.

The same its true in other areas. With the Mac, Apple has three ranges of laptops, three ranges of desktops and one rackable server. Compared to HP, Dell or Acer this is a ridiculously small number of product lines. Again with the iPod there are 4 product ranges each with a couple of different memory size and and more variation on color.

There are a number of advantages in having a small number of product lines. Economy of scale will make the product cheaper to manufacture, however by the time you get to millions of devices, the additional advantage is not that great. More important are a brutally strong brand image and a lack of consumer confusion. There is no question about which version of the iPhone to get, the only question is whether you are willing to pay more for the extra memory.

However, there is one big disadvantage to having a single product line like the iPhone, and that is that you have all your eggs in one basket. If the product should prove to have a flaw, there is no other product line to take up the slack. If a consumer want to buy an iPhone now, they either have to go ahead and take the risk that it might be a lemon or wait until the problem is fixed. They cannot go out and buy the other Apple phone because it does not exist. For a big consumer goods company, Apple has had remarkably few dud products, but their life depends on getting each one right.

The HP Tablet and the Elephant

Recently HP bought Palm and in the acquisition press release announced that they are developing "... webOS based hardware products, from a robust smartphone roadmap to future slate PCs and netbooks". In all the discussion of this event, nobody seems to be discussing the elephant in the room, or more correctly, the elephant who is no longer in the room.

Ten years ago, HP would not have dared announce that it was going produce its own operating system (OS) in competition with the dominant Microsoft Windows OS. Then, most hardware developers had been cowed by Microsoft's aggressive and successful response to any attempt to develop a rival operating system. To give a couple of examples, in the early 90's the Go Corporation had developed its Penpoint OS for handheld computing. Then in 1992, Microsoft announced its own Windows for Pen Computing. Go Corporation faltered, was taken over by AT&T and then the project was shuttered. Another example is the fate of Be Inc. who had developed BeOS, initially to power their own hardware. In 2002, Be Inc. sued Microsoft claiming that Hitachi had been dissuaded from selling PCs loaded with BeOS, and that Compaq had been pressured to not market an Internet appliance in partnership with Be. The case was eventually settled out of court with no admission of liability on Microsoft's part. However by this time Be Inc had admitted defeat and sold its intellectual property to Palm Inc.

In the late 90's Microsoft was so dominant that no Silicon valley Venture Capital firm would fund a start up that would have the remotest chance of challenging Microsoft in any way. Since then Microsoft seems to have been transformed from a lithe competitor into a stumbling giant. The Vista version of the Windows OS is widely regarded as a failure, and was quickly replaced by Windows 7. While the Windows Mobile OS for smartphones has been around for a long time and gone through several versions, it has been losing market share for some time. Recently Microsoft introduced a new smartphone, the Kin with much ballyhoo, only to give up on it six week later. There are plenty other examples of Microsoft's left hand not knowing what the right hand was doing, like the PlayForSure debacle.

We have come to the point where Microsoft is so crippled by its own self inflicted wounds that one of its most important OEM customers is going to use its own operating system on future slate PCs and netbooks. The elephant is no longer in the room.

Winning With Big Data

Michael Driscoll gave us Secrets of a Successful Data Scientist at the June meeting of the SDForum Business Intelligence SIG in his talk "Winning With Big Data". Michael is founder of a data consultancy Dataspora, where he has done work on projects ranging to analyzing baseball pitchers through helping cell phone companies understand their customer churn. You can see slides for the talk here, and follow Micheal's thoughts in his excellent blog on the Dataspora site.

After Michael revved up the crowd by giving the Hal Varian quote that "... the sexy job in the next ten years will be statisticians", he went through 9 ways to win as a Data Scientist. His first suggestion is to use the right tools. Michael uses a variety of tools including database systems, Hadoop and the R language. Large data takes a long time to process and often we can gain insights by just working with a sample of the data, however you have to be careful when taking a sample to ensure that it makes sense and that the results will scale. Which leads us to the another way to win, which is to know, understand and use statistics.

Statistics is a field of mathematics that is still developing and it is not easy, however statistics is a core competence of a Data Scientist. It is not enough to do the analysis, the Data Scientist has to be able to present the results and turn them into a compelling story. Both analysis and presentation requires good visualization tools and the knowledge of how to use them.

To illustrate his ways to win, Michael led us through a specific example of a successful data analysis that he had done. He had been asked by a cell phone company to investigate customer churn. Although he looked at the data in several different ways, his successful analysis went as follows. The starting point was Call Data Record (CDR) which records each call that a customer makes. Cell phone traffic generates billions of CDRs, so Michael first cut the data set down to a more manageable size by just looking at the CDRs for a single city. He then created social graphs between customers that call one another frequently, and was able to show that if one customer dropped service it was a predictor that other customers in that social graph would also leave the service. The study ended with a clever visualization of connected customers leaving the cell phone provider.

Which Cloud Standards Matter?

The SDForum Cloud Services SIG June meeting was a panel session with multiple speakers devoted to the question "Which Cloud Standards Matter?". The answer came through loud an clear as speaker after speaker discussed Open Virtualization Format (OVF). No other standard got more than a mention or so.

OVF is a container that defines the contents of a virtual machine. It is simply a set of file in a directory and an XML descriptor file. The standard is managed by the Distributed Management Task Force (DMTF). Panel speaker Priya Ketkar of Abiquo showed OVF being used to move a virtual machine from one cloud service provider to another. Winston Bumpus, the final panel speaker, is President of the of the DMTF and Director of Director of Standards Architecture for VMWare. He made a convincing case for DMTF and its management of the OVF standard.

Another panel member James Urquhart of Cisco mentioned several standards including OVF, however he spent considerable time on XMPP, surely the most unlikely standard for cloud computing. I discussed XMPP some time ago. It is a standard for exchanging instant messages and Twitter feeds between large service providers. While it is a useful standard I do not see its place in cloud computing. If you can explain how XMPP helps cloud computing, please enlighten me.

Reporting from the Production Database

Salesforce.com does their analytics directly out of their production database. For me, this was the interesting story that emerged from the talk on "Real Time Analytics at Salesforce.com" at the May meeting of the SDForum Business Intelligence SIG. Note that this post is not a report on the meeting, rather it is a reflection on a topic that came up during the meeting. Both my co-chair Paul O'Rorke and SIG member James Downey have written great summaries of the meeting.

Directly reporting from a production database is an issue that comes up from time to time. Deciding on whether to do it is a two step process. The first question is to ask whether it is possible. A database can be oriented to report the current state of affairs or alternatively to contain a record of how we got to the current state of affairs. In practice we need both views, and it is common to have a production database that is oriented to the maintaining the current status and a data warehouse that maintains the historical record. Typically an enterprise has several databases with production information and the historical record is combined in a single reporting data warehouse.

The tension between the requirements for production and reporting databases shows up in a number of ways. Production needs a fast transaction execution. One way to achieve this is to make the database small, cutting out anything that is not really needed. On the other hand, we want to keep as much information as possible for reporting, so that we can compare this time period with a year ago or maybe even two years ago. Reporting wants a simple database structure like a star schema that makes it straightforward to write ad-hoc queries that that generate good answers. Production databases tend to have more interlinked structures.

Salesforce.com is in the business of Customer Relationship Management (CRM), where it is useful to keep the historical record of interactions with each customer. As Salesforce.com has the historical record in their production database, reporting from that database makes perfect sense. In fact much of the impetus for real time data warehousing has come from CRM like applications. One common example is where a business wants to drive call center applications from data in their data warehouse.

The next question is whether it is a good idea to combine reporting and production queries in the same database. Production queries are short, usually reading a few records and then updating and inserting a few records. Reporting queries are read only, but they are longer running and may touch many records to produce aggregate results. A potential issue is that a longer running reporting query may interfere with production queries and prevent them from doing their job. This is the other major reason for doing reporting from a separate database than the production database.

The Oracle database used by Salesforce.com has optimistic read locking so that read only queries do not lock out queries that update the database. Also, as came out in the presentation, Salesforce.com has a multi-tenant database where each customer customizes their use of data fields in a different ways. Because of this, they sometimes copy the data out of the big table into a smaller temporary table to transform the data into the form that the customers query expects. Making a copy of the relevant data for further massaging is a common tactic in data reporting tools so this is not unusual. It also gets the reporting data out of the way of production data so they two do not interfere with one another.

Finally, Salesforce.com is large enough that they can afford a luxury of having a performance team whose sole purpose is to look at queries that take the longest to run or use up the most resources. Any database application requires some performance tuning, however it is especially important when doing reporting from a production database.

Google's Got Background

Go away for a few days and when I come back, Google looks like Bing. Instead of a restful blank page they had a background picture. Arrgh! Fortunately, it lasted for less than a day, and then we went back to the blank page we knew and loved.

Actually it is very clever. Firstly it tells the people who might be attracted to Bing because they can customize how the page looks that they can do the same thing with Google. Secondly, and more importantly, it encourages people to create and log in to their Google account so that they can customize their Google home page. Google can give you a better search experience when it knows who you are, and it can make more money from the advertisements that are pitched at you when it knows who you are.

I thought of trying to customize the page to something less distracting when I realized that I would have to give up something of my identity in exchange. On weighing this transaction I decided that what I would give up outweighed the benefit, particularly when the backlash would probably cause the background image to be a short lived experiment.

Dancing About Architecture

"Writing about music is like dancing about architecture." is one of these quotes that never seem to die. Last week I heard it again while listening to a podcast. The quote is attributed to many people including Elvis Costello in a 1983 interview, although the origin seems to be older, perhaps much older than that. More interesting, from reading the linked piece is that someone tested dancing about architecture to see if it "was really so strange".

Whoever said it, they certainly caught the truth that written words cannot adequately capture an aural sensation. A great illustration of this is the 1998 interview* of Ray Manzarek of The Doors by Terry Gross on the Fresh Aire radio program. Manzarek describes with the help of a keyboard how the Doors worked as a group and how they wrote the song "Light My Fire". A written transcript of this interview would be unintelligible, whereas the audio interview is a revelation. Terry Gross has recorded many interviews with musicians where they play their music and they are all worth hearing.

Although I spend plenty of time listening, I have never found it useful to read about music. That is not to say that there cannot be good writing about music. In my experience the best has been in fiction, particularly the novels of Ian McEwan. In "Saturday", there are a few pages with a magical description of a rock band performing one song, followed by a meditation on how certain passages of music, and certain performances can affect us to the core.

A good part of "Amsterdam" is about the process of composing a classical symphony. While I have never composed music, I do design and write software and there are similarities to the process. I will write more about this another time. Unfortunately the novel and the symphony are cut short by the books annoying ending.

* Unfortunately, to listen to this piece, you have to have the Real Player. I have it because I have installed the BBC iPlayer to listen to old comedy shows including the Goon Show. If you do not want the Real Player, here is a another piece from NPR Music about "Light My Fire" with more palatable download requirements.

The App Economy

The evolution of the App Economy is a marvelous thing to watch. In March I questioned whether apps for the iPad would develop with the same strength as apps for the iPhone, because more content is accessible through the browser. Jacob Weisberg at Slate discussed the same thing recently in more depth. He exhorts publishers to beware of getting tangled up with Apple for both monetary and censorship reasons.

On the other hand, web content is not fully available on the iPad. Steve Jobs has denigrated Flash for being slow, buggy and inefficient, and has sworn that it will never be seen on the iPad. In its place Jobs suggests HTML5. The problem is that HTML5 does not do everything that Flash does. This recent piece on on Apple Insider explains the shortcomings of HTML5 and why Hulu will not be using it any time soon for their video distribution.

If Hulu cannot use Flash, then its only alternative is to develop an App, which it is reportedly doing. If Hulu has an App, it may charge a subscription as is being discussed. If Hulu charges a subscription, some of that revenue flows to Apple. By banning a rival development platform, Apple is encouraging the App Economy to its own advantage. Thus it is a pity that so many of the early publishing apps have received such bad reviews.