Which Cloud Standards Matter?

The SDForum Cloud Services SIG June meeting was a panel session with multiple speakers devoted to the question "Which Cloud Standards Matter?". The answer came through loud an clear as speaker after speaker discussed Open Virtualization Format (OVF). No other standard got more than a mention or so.

OVF is a container that defines the contents of a virtual machine. It is simply a set of file in a directory and an XML descriptor file. The standard is managed by the Distributed Management Task Force (DMTF). Panel speaker Priya Ketkar of Abiquo showed OVF being used to move a virtual machine from one cloud service provider to another. Winston Bumpus, the final panel speaker, is President of the of the DMTF and Director of Director of Standards Architecture for VMWare. He made a convincing case for DMTF and its management of the OVF standard.

Another panel member James Urquhart of Cisco mentioned several standards including OVF, however he spent considerable time on XMPP, surely the most unlikely standard for cloud computing. I discussed XMPP some time ago. It is a standard for exchanging instant messages and Twitter feeds between large service providers. While it is a useful standard I do not see its place in cloud computing. If you can explain how XMPP helps cloud computing, please enlighten me.

Reporting from the Production Database

Salesforce.com does their analytics directly out of their production database. For me, this was the interesting story that emerged from the talk on "Real Time Analytics at Salesforce.com" at the May meeting of the SDForum Business Intelligence SIG. Note that this post is not a report on the meeting, rather it is a reflection on a topic that came up during the meeting. Both my co-chair Paul O'Rorke and SIG member James Downey have written great summaries of the meeting.

Directly reporting from a production database is an issue that comes up from time to time. Deciding on whether to do it is a two step process. The first question is to ask whether it is possible. A database can be oriented to report the current state of affairs or alternatively to contain a record of how we got to the current state of affairs. In practice we need both views, and it is common to have a production database that is oriented to the maintaining the current status and a data warehouse that maintains the historical record. Typically an enterprise has several databases with production information and the historical record is combined in a single reporting data warehouse.

The tension between the requirements for production and reporting databases shows up in a number of ways. Production needs a fast transaction execution. One way to achieve this is to make the database small, cutting out anything that is not really needed. On the other hand, we want to keep as much information as possible for reporting, so that we can compare this time period with a year ago or maybe even two years ago. Reporting wants a simple database structure like a star schema that makes it straightforward to write ad-hoc queries that that generate good answers. Production databases tend to have more interlinked structures.

Salesforce.com is in the business of Customer Relationship Management (CRM), where it is useful to keep the historical record of interactions with each customer. As Salesforce.com has the historical record in their production database, reporting from that database makes perfect sense. In fact much of the impetus for real time data warehousing has come from CRM like applications. One common example is where a business wants to drive call center applications from data in their data warehouse.

The next question is whether it is a good idea to combine reporting and production queries in the same database. Production queries are short, usually reading a few records and then updating and inserting a few records. Reporting queries are read only, but they are longer running and may touch many records to produce aggregate results. A potential issue is that a longer running reporting query may interfere with production queries and prevent them from doing their job. This is the other major reason for doing reporting from a separate database than the production database.

The Oracle database used by Salesforce.com has optimistic read locking so that read only queries do not lock out queries that update the database. Also, as came out in the presentation, Salesforce.com has a multi-tenant database where each customer customizes their use of data fields in a different ways. Because of this, they sometimes copy the data out of the big table into a smaller temporary table to transform the data into the form that the customers query expects. Making a copy of the relevant data for further massaging is a common tactic in data reporting tools so this is not unusual. It also gets the reporting data out of the way of production data so they two do not interfere with one another.

Finally, Salesforce.com is large enough that they can afford a luxury of having a performance team whose sole purpose is to look at queries that take the longest to run or use up the most resources. Any database application requires some performance tuning, however it is especially important when doing reporting from a production database.

Google's Got Background

Go away for a few days and when I come back, Google looks like Bing. Instead of a restful blank page they had a background picture. Arrgh! Fortunately, it lasted for less than a day, and then we went back to the blank page we knew and loved.

Actually it is very clever. Firstly it tells the people who might be attracted to Bing because they can customize how the page looks that they can do the same thing with Google. Secondly, and more importantly, it encourages people to create and log in to their Google account so that they can customize their Google home page. Google can give you a better search experience when it knows who you are, and it can make more money from the advertisements that are pitched at you when it knows who you are.

I thought of trying to customize the page to something less distracting when I realized that I would have to give up something of my identity in exchange. On weighing this transaction I decided that what I would give up outweighed the benefit, particularly when the backlash would probably cause the background image to be a short lived experiment.

Dancing About Architecture

"Writing about music is like dancing about architecture." is one of these quotes that never seem to die. Last week I heard it again while listening to a podcast. The quote is attributed to many people including Elvis Costello in a 1983 interview, although the origin seems to be older, perhaps much older than that. More interesting, from reading the linked piece is that someone tested dancing about architecture to see if it "was really so strange".

Whoever said it, they certainly caught the truth that written words cannot adequately capture an aural sensation. A great illustration of this is the 1998 interview* of Ray Manzarek of The Doors by Terry Gross on the Fresh Aire radio program. Manzarek describes with the help of a keyboard how the Doors worked as a group and how they wrote the song "Light My Fire". A written transcript of this interview would be unintelligible, whereas the audio interview is a revelation. Terry Gross has recorded many interviews with musicians where they play their music and they are all worth hearing.

Although I spend plenty of time listening, I have never found it useful to read about music. That is not to say that there cannot be good writing about music. In my experience the best has been in fiction, particularly the novels of Ian McEwan. In "Saturday", there are a few pages with a magical description of a rock band performing one song, followed by a meditation on how certain passages of music, and certain performances can affect us to the core.

A good part of "Amsterdam" is about the process of composing a classical symphony. While I have never composed music, I do design and write software and there are similarities to the process. I will write more about this another time. Unfortunately the novel and the symphony are cut short by the books annoying ending.

* Unfortunately, to listen to this piece, you have to have the Real Player. I have it because I have installed the BBC iPlayer to listen to old comedy shows including the Goon Show. If you do not want the Real Player, here is a another piece from NPR Music about "Light My Fire" with more palatable download requirements.

The App Economy

The evolution of the App Economy is a marvelous thing to watch. In March I questioned whether apps for the iPad would develop with the same strength as apps for the iPhone, because more content is accessible through the browser. Jacob Weisberg at Slate discussed the same thing recently in more depth. He exhorts publishers to beware of getting tangled up with Apple for both monetary and censorship reasons.

On the other hand, web content is not fully available on the iPad. Steve Jobs has denigrated Flash for being slow, buggy and inefficient, and has sworn that it will never be seen on the iPad. In its place Jobs suggests HTML5. The problem is that HTML5 does not do everything that Flash does. This recent piece on on Apple Insider explains the shortcomings of HTML5 and why Hulu will not be using it any time soon for their video distribution.

If Hulu cannot use Flash, then its only alternative is to develop an App, which it is reportedly doing. If Hulu has an App, it may charge a subscription as is being discussed. If Hulu charges a subscription, some of that revenue flows to Apple. By banning a rival development platform, Apple is encouraging the App Economy to its own advantage. Thus it is a pity that so many of the early publishing apps have received such bad reviews.

Google Books Rocks

Awesome is too small a word to express what Google Books has achieved. Last year Google settled the class action law suit that allows them to index out of print books that they had digitized. As part of the settlement they also have to sell the books, which means that Google is now a bookseller. The most important part of the settlement is the Books Right Registry:

"The agreement will also create an independent, not-for-profit Book Rights Registry to represent authors, publishers and other rightsholders. In essence, the Registry will help locate rightsholders and ensure that they receive the money their works earn under this agreement. You can visit the settlement administration site, the Authors Guild or the AAP to learn more about this important initiative."

One of the biggest practical issue with Intellectual Property is that it is impossible to use most Intellectual Property because you do not know who owns it, and therefore you do not know who to ask for permission to use it. Laurence Lessig has been talking about this for a long time as a part of his campaign to fix copyright laws. The establishment of a Book Rights Registry goes some way to address the problem with one type of Intellectual Property. Perhaps this will be the beginning of a trend.

I will write more about this issue another day. For now, here is how I stumbled upon the awesomeness of Google Books. My father would often quote "but tomorrow by the living god, we'll try the game again" after some setback. I knew it was from a poem, but not much more. So the other day, I typed "but tomorrow by the living god" into Google and was astonished by the progress that has been made in search over the last few years. The first entry in the search results linked to a poetry anthology in Google Books that has the full poem by John Masefield.

Masefield is best known for his poems "Sea Fever", "I must go down to the seas again, to the lonely seas and the sky, ..." and "Cargoes", "Quinquireme of Nineveh from distant Ophir, ..." For poem collectors, here is the rarely seen poem TOMORROW by John Masefield:

Oh yesterday the cutting edge drank thirstily and deep,
The upland outlaws ringed us in and herded us as sheep,
They drove us from the stricken field and bayed us into keep;
But tomorrow
By the living God, we'll try the game again!

Oh yesterday our little troop was ridden through and through,
Our swaying, tattered pennons fled a broken, beaten few,
And all a summer afternoon, they hunted us and slew;
But tomorrow
By the living God, we'll try the game again!

And here upon the turret-top the bale-fires glower red,
The wake-lights burn and drip about our hacked, disfigured dead,
And many a broken heart is here and many a broken head;
But tomorrow
By the living God, we'll try the game again!

In my original search results, there was a link to Google newspapers where a Virgin Islands Daily News edition from 1950 quotes part of the poem. This time when I did the search, that link did not come up. Instead there was a link to a 1991 zine for Vietnam War vets that quotes a verse of the poem. Who knows what you may find when you do the search.

More on The Big Short

When I wrote that Michael Lewis had written an almost uplifting account of the financial crisis in "The Big Short" by concentrating on some of the winners, I did not consider that he was keeping something back. If you want to find out what he really thinks, read this interview on Bloomberg.com. He explains many of the choices that he made in the book, like for instance not including John Paulson who has been celebrated in other places for "The Greatest Trade Ever". He also expresses his outrage over what happened and suggests that part of the reason he left Wall Street in 1989 was because his job was basically "exploiting the idiocy of my customers". It is a long interview and well worth reading in its entirety.

One issue that Lewis touches on is the fact that shorting the market is supposed to dampen the market and perhaps bring sanity into it, but in this case the structured investment vehicles like synthetic CDOs had the opposite effect of amplifying the market and making the subsequent downfall much worse. The "This American Life" radio show and podcast has a recent segment where they discuss the role of the Magnetar Hedge Fund in creating many several subprime bonds and then making huge sums of money by shorting parts of them. Again well worth hearing.

Finally, Lewis discusses the poisonous interface between the big Wall Street firms and their customers. If Goldman Sachs is responsible for defrauding its customers as the recent lawsuit suggests, there is the question of why anyone would want to do business with them. The Big Money blog posits that Goldman Sachs is losing its "Social License" to operate in an interesting post. Given their behavior, this may be a good thing.

Business Rules OK!

Performance Management Systems collect the data to make decisions but they do not make decisions, they do not ensure that decisions get made or even track the results of the decision so made. James Taylor (no relation) called this the "over-instrumented" enterprise when he spoke to the the April meeting of the SDForum Business Intelligence SIG on "Performance Management and Agility". James is CEO of Decision Management Solutions where he consults on using technology to better effect decision making.

James divides the decisions that an organization makes into three levels: strategic, tactical and operational. He is interested in the operation decisions, the little decisions that are taken all the time. An example of an operational decision is what offer to make to a customer that has called a call center. Every enterprise has their own set of operational decisions, however they have the characteristic that is a large number of them that in aggregate they represent a lot of value, so they are well worth managing.

Many operational decisions are or should be automated, and there are a set of principles that need to be recognized when decision making is automated. The first principle is that no decision is going to be forever, so the logic for making the decision should not be locked up into something inflexible such as program code. Much better to use a rules based decision engine which allows everybody to see the rules in a language that they can understand. Another principle is that making a decision is a business process and as such should be managed. A good business rules engine allows rules to be tested, measured and perhaps even simulated in action to understand what they are doing and how they can be optimized.

According to James, the purpose of the information gathered for a Performance Management Systems is to make decisions, so it should be used to make decisions. Too many enterprises are over-instrumented. They have spent all their effort to get and present the data, however they have no measurable ability to turn that data into actions. You can read more about these ideas in the book Smart Enough Systems: How to Deliver Competitive Advantage by Automating Hidden Decisions by James Taylor and Neil Raden. You can also read my co-chair Paul O'Rorke's take on the meeting in his blog.

And the Future of Television is ...

Wait for it, wait for it ... Sports! The path to this conclusion requires a couple of steps, so bear with me. The Convergence Consulting Group just published their annual report on "The Battle for the North American Couch Potato", and several news sources and commentators immediately picked up on one element of their report. According to Convergence, by the end of 2009, 800,000 US households had cut the cable and that they expected this to double to 1.6 million households by 2011. Cutting cable means cutting subscription TV service like cable or satellite and getting all media content from the Internet, Netflix and over the air. I recently wrote about Television being in trouble because of increasing subscription fees and less content. There has been a trickle of cable cutters for some time and now Convergence Consulting tells us that the numbers are starting to swell.

So why should we not cut the cable? It turns out that sports is the only type of content where subscription television offers a compelling product that you cannot easily get if you cut the cable. I came to this conclusion after skimming through the comments on TechCrunch post on the cable cutting story. The majority of comments are either from people who have cut the cable and the only thing they miss is sports, or from people who say that they cannot cut the cable because they would not be able to get the sports that they want to see. The fact that sports is the only type of content mentioned is quite startling.

The Big Short

What is the best way to write an uplifting book about the recent financial crisis? In his new book, The Big Short, Michael Lewis has taken the approach of following the winners, the people who saw that the bubble would burst and made a huge sums of money by betting on it bursting. Along the way we also meet some of the people who took the other side of the bet and lost big. When you compare these two groups, the losers come across as your average every day kind of person while the winners area strange group of outsiders.

The book follows three groups of people. There is the Frontpoint Partners hedge fund led by Steve Eisman, who as a stock analyst had been best known for correctly trashing companies that he followed. Then there is Mike Burry, a one eyed doctor living in San Jose who took up investing because it allowed him to get away from having to interact with other people. Finally there are the guys at the garage band hedge fund who turn $100,000 in to more than $100 million and whose main problem is being taken seriously by the big Wall Street Firms.

Along the way we see scenes of madness from the financial machine that created the bubble. There is the explanation for why a Mexican strawberry picker with no English and an income of $14,000 per year could be loaned every penny he needed to buy a house for $724,000 in Bakersfield California. As it turns out, because he had no debt and no credit history, he has a relatively high credit rating, and that credit rating was needed to balance out the low credit rating of some deadbeat American when their home loans were packaged together with many others into a mortgage bond.

Another scene is the American Securitization Forum, the annual conference of the of the subprime mortgage industry. In early 2007 the conference takes place in the Venetian Hotel in Las Vegas. The Frontpoint Partners and the garage band hedge fund are both there trying to get more information to substantiate their huge bets against the subprime mortgage market. By this time the cracks were beginning to show. For example, the CEO of the Option One mortgage corporation gave a reassuring speech even although Option One was in trouble because they had made loans to people who could not afford to make even the first payment on the loan. When his partner asks "Who takes out a home loan and doesn't make the the first payment?" Steve Eisman responds "Who the #$%^ lends money to people who can't make the first payment?"

Michael Lewis knows what he is talking about in writing about Wall Street because he started his career as a bond salesman for Salomon Brothers as hilariously told in his first book, Liar's Poker. In The Big Short he successfully continues that tradition. The Big Short is full of interesting characters, amusing insights, clear explanations and some genuine tension as towards the end you wonder whether the hero's will get their big payoff or whether Wall Street will totally collapse taking down everybody with them. Like of his other books, The Big Short is highly recommended.