Blog Ennui

It is Sunday morning and I notice that TechCrunch has a couple of new posts. One is a stream of consciousness piece from Steve Gillmor called "Bill's Gold Watch". This one was better than the stream of consciousness piece Steve wrote last week called "The Blood Brain Barrier", mainly because it was shorter. Steve can write conventional blog posts, for example on Saturday morning he had an excellent piece called "Facebook's Glass Jaw" which comments on the Facebook - Friends Connect fracas.

So what is Steve trying to do with "Bill's Gold Watch"? Is he trying to create a new Journalism? To me, it reads more like poetry. Even if it does not make complete sense, it sparks off thoughts and associations, and that appears to be the intention. Another commenter suggested it reads like rap. If it were printed as blank verse we would see what was going on and Steve could concentrate in getting the rhythms right as well as avoiding some of the more tortuous and interconnected thoughts.

In other blog thoughts, I have completely given up on Vallywag. Shortly after I wrote about Vallywag a year and a half ago, the then editor Nick Douglas departed and it has been downhill ever since. Now it is just a load of social claptrap of the sort that fills the gossip column of a tabloid newspaper.

Hypertable - A Massively Parallel Database System

Now everyone can have their own database system that scales to thousands of processors, as we heard at the April meeting of the SDForum Software Architecture and Modeling SIG. Doug Judd from zEvents, and the Hypertable Lead Developer spoke on "Architecting Hypertable-a massively parallel high performance database".

Hypertable is an Open Source database system designed to deal with the massive scale of data that is found in web applications such as processing the data returned by web crawlers as they crawl the entire internet. It is also designed to run on the massive commodity computer farms, which can consist of thousands of systems, that are employed to process such data. In particular Hypertable is designed so that its performance will scale with the number of computers used and to handle the unreliability problems that inevitably ensue from using large computer arrays.

From a user perspective, the data model has a database that contains tables. Each table consists of a set of rows. Each row has a primary key value and a set of columns. Each column contains a set of key value pairs commonly known as a map. A timestamp is associated with each key value pair. The number of columns in a table is limited to 256, otherwise there are no tight constraints on the size of keys or values. The only query method is a table scan. Tables are stored in primary key order, so a query easily accesses a row or group of rows by constraining on the row key. The query can specify which columns are returned, and the time range for key value pairs in each column.

The basic unit for inserting data is the key value pair, along with its row key and column. An insert will create a new row if none exist with that row key. More likely, an insert will add a new key value pair to an existing column map or have the existing value superseded if the new column key already exists in the column map.

As Doug explained, Hypertable is neither relational or transactional. Its purpose is to store vast amounts of structured data and make that data easily available. For example, while Hypertable does have logging to ensure that information does not get lost, it does not support transactions whose purpose is to make sure that multiple related changes either all happen together or none of them happen. Interestingly, many database systems switch off transactional behavior for large bulk loads. There is no mechanism for combining data from different tables as tables are expected to be so large that there is little point in trying to combine them.

The current status is that Hypertable is in alpha release. The code is there and works as Doug showed us in a demonstration, however it uses a distributed file system like Hadoop to store its data and while they are still developing they are also waiting for Hadoop to implement a consistency feature before they declare beta. Even then there are a number of places where they have a single point of failure, so there is plenty of work to make it a complete and resilient system.

Hypertable is closely modeled on Google Bigtable. At several times in the presentation when asked about a feature, Doug explained it as something that Bigtable does. At one point he even went so far as to say "if it is good enough for Google, then it is good enough for us".

SaaS, Cloud, Web 2.0... it’s time for Business Intelligence to evolve!

The most surprising phrase in Roman Bukary's presentation to the April meeting of the SDForum Business Intelligence SIG was "right time, not real time", and it was said more than once. Roman is Vice President of Marketing and Business Development at Truviso and his presentation entitled "SaaS, Cloud, Web 2.0... it’s time for Business Intelligence to Evolve!" brought a large audience to our new location at the SAP Labs on Hillview Avenue in Palo Alto.

Truviso provides software to continuously analyze huge volumes of data, enabling instant visibility, immediate action and more profitable decision making. In other words, their product is a streaming database system.

Over the years, the Business Intelligence SIG has heard about several streaming database systems. Truviso distinguishes themselves in a number of ways. Firstly it leverages the open source Postgres database system, so it is a real database system and real SQL. Other desirable characteristics are handling large volumes of data, large numbers of queries and the ability to change queries on the fly. They also have a graphics front end that can draw good looking charts. Roman showed us several Truviso applications including stock and currency trading applications that are both high volume and a rapidly changing environment.

Then we come to the "right time, not real time" phrase. In the past I have associated this phrase with business intelligence systems that could not present the data in a timely manner. Obviously, that is not a problem with streaming database systems that process and aggregate data on the fly and always have the most up to date information.

I think that Roman was trying to go in the other direction. He was suggesting that Truviso is not only useful for high pressure real time applications like stock trading, it also has a place in other applications where time is less pressing but the volume of data is high and there is still a need for a real time view of the current state. Such applications could include RFID, logistics and inventory management.

Open Source 10 Years Later

April 7 2008 marks 10 years since the landmark Freeware Summit that signaled the opening of the Open Source movement. By coincidence I recently read the manifesto of the Open Source movement, "The Cathedral and The Bazaar" by Eric S Raymond. The book, published in 1999 and revised in 2001, contains the namesake essay several others including "Revenge of the Hackers" which describes the events leading up to and following the Freeware Summit from an insiders point of view. The essay is valuable as a history of Open Source however its veracity is slightly marred because it dates summit meeting as happening on March 7.

One thing that the Revenge of the Hackers does not shy away from is explaining why Richard Stallman and the Free Software Foundation was not present at the Freeware Summit. In the past I have written on the distinction between Open Source and Free Software. Raymond is tactful but firm in explaining why creating a separation between these two ideas was essential to getting Open Source accepted by the mainstream.

On the other hand, the end of the essay that looks into the future of Open Source does suffer in hindsight. Open Source has advanced by leaps and bounds in the last 10 years. However it is still not in the position of ruling the world as the Revenge of the Hackers suggests it might. Lets give it at least another 10 years.

An Evening with The Difference Engine

One day I write about Doron Swade and "The Cogwheel Brain". Three days later I get an invitation from the Computer History Museum. Doron Swade is coming to Silicon Valley with a Difference Engine!

The occasion is that another Difference Engine has been commissioned by Nathan Myhrvold, ex CTO of Microsoft. It is being exhibited at the Computer History Museum in Mountain View, and to celebrate its arrival, there is an "Evening with Nathan Myhrvold and Doron Swade" at the museum. We have signed up for the event, have you?

Building Better Products Through Experimentation

Experimentation is the theme of the SDForum Business Intelligence SIG so far this year. The March meeting featured Deepak Nadig, a Principal Architect at eBay, talking about "Building Better Products Through Experimentation". Experimentation is an important technique for Business Intelligence, although its first uses were with medicine. In 1747, James Lind, a British naval surgeon performed a controlled experiment to find a cure for scurvy. In his book "Supercrunchers", Ian Ayres describes how the Food and Drug Administration has used experimentation since the 1940s to determine whether a medical treatment is efficacious.

While eBay has always used experimentation test and fine tune its web pages, in recent years the process has been formalized. While anyone can propose an experiment, product managers are the group of people who are most likely to do so. Deepak took us through the eBay process and discussed issues with using experimentation. Because they have the infrastructure, simple experiments can be set up within a matter of days. eBay usually runs an experiment for at least a week so that it is exposed to a full cycle of user behavior. Simple experiments to test a small feature typically run for a week or so, larger experiments may run for a month or two and some critical tests run continuously.

For example, eBay is interested in whether it is a good idea to place advertising on their pages. On the one hand it brings in extra revenue in the short term, on the other hand, it might cannibalize revenue in the long term. Experimentation has shown that advertising is a good thing in some situations, however its use is being monitored by some long term experiments to ensure that it remains beneficial.

Deepak took us through some of the issues that with experimentation. One issue is concurrency, how many experiments can be carried out at the same time. As eBay has a high traffic web site, they can get good results with experiments on a small proportion of the users, at most a few percent. As each experiment uses a small percentage of the users, several experiments can be run in parallel. Another issue is establishing a signal to noise ratio for experiments to ensure that experiments are working and giving valid results. eBay has done some AB experiments where A and B are exactly the same to establish whether their experimental technique has any biases.

The Cogwheel Brain

The Cogwheel Brain by Doron Swade is the story of Charles Babbage and his quest to build the first computer. The book also details how Doron Swade built a Babbage Difference Engine for the 200th anniversary of Babbage's birth in 1991.

Charles Babbage designed 3 machines. His started with the Difference Engine that would use the method of finite differences to generate tables such as logarithms and navigation tables. The computing section of his first design was built although it did not have a printer. Next he conceived and designed an Analytic Engine, which was a fully functioning computer that was programmed by the same kind of punched cards that were used to run a Jacquard weaving loom. In the course of designing the Analytic Engine he realized that he could improve the design of the Difference Engine to make it faster and use less parts. This resulted in the design of Difference Engine 2. Only small demonstration parts of the Analytic Engine were built and the Difference Engine 2 existed only as a set of plans.

I expected the story to be similar to several other computing projects that I have seen and worked with. You know the projects, the ones where the architect keeps jumping to a new idea while the overall project goals get lost and the project overruns for years before it is abandoned. Building the Difference Engine was a lot more disciplined. The core of the first difference machine was built and worked even although it used orders of magnitude more machined parts than any other machine built up to that time. While it did take a long time, given the engineering practices of the day, all the parts had to be made by a single craftsman in a single workshop.

One thing from the book that surprised me is that during the 19th century other difference engines were built by other engineers. Although these machines were completed, they were never successfully used for any purpose. I think this goes to show that the 19th century was not ready for mechanical computing. The book is easy to read and highly recommended.

Customer Relationship Intelligence

There is a curious thing about the organization of a typical company. While there is one Vice President in charge of Finance and one Vice President in charge of Operations there can be up to three Vice Presidents facing the customer: a Marketing Vice President, a Sales Vice President, and a Service Vice President. On the one hand, the multiplicity of Vice Presidents and their attendant organizations is a testament to the importance of the customer. On the other hand, multiple organizations mean that no one is in charge of the customer relationship and thus no one takes responsibility for it.

We see this in the metrics that are normally used to measure and reward customer-facing employees. Marketing measure themselves on how well they find leads regardless of whether sales uses the leads. Sales measure themselves on the efficiency of the sales people in making sales regardless of whether the customer is satisfied. Service, left to pick up the pieces of an overpromised sale, measure themselves on how quickly they answer the phone. Every one is measuring their own actions and no one is measuring the customer.

Linda Sharp addresses this conundrum head on in her new book Customer Relationship Intelligence. As Linda explains, a customer relationship is built upon a series of interactions between a business and its customer. For example, the interactions starts with acquiring a lead, perhaps through an email or mass mailing response or a clickthrough on a web site. Next, more interactions qualify the lead as a potential customer. Making the sale requires further interactions leading up to the closing. After the sale there are yet more interactions to deliver and install the product and service to keep it working. Linda's thesis is that each interaction builds the relationship and that by recording all the interactions and giving them both a value and a cost, the business builds a quantified measure of the value of its customer relationships and how much it has spent to build them.

Having a value for a customer relationship completely changes the perspective of that relationship. It gives marketing, sales and service an incentive to work together to build the value in the relationship rather than working at cross purposes to build their own empires. Moreover, knowing the cost of having built the relationship suggests the value in continuing the relationship after the sale is made. In the book, Linda takes the whole of the second chapter to discuss customer retention and why that is where the real profit is.

The rest of the book is logically laid out. Chapter Three “A Comprehensive, Consistent Framework” creates a unified model of a customer relationship throughout its entire lifecycle from the first contact by marketing through sales and service to partnership. This lays a firm bedrock for Chapter Four, “The Missing Metric: Relationship Value” which explains the customer relationship metric, the idea that by measuring the interactions that make the relationship we can give a value to the relationship.

The next two chapters discuss how the metric can be used to drive customer relationship strategy and tactics. The discussion of tactics lays the foundation for Chapter Seven, which shows how the metric is used in the execution of customer relationships. Chapters Six and Seven contain enough concrete examples of how the data can be collected and used to give to give us a feeling of the metric’s practicality. Chapter Eight compares the customer relationship metric with other metrics and explores the many ways in which it can be used. Finally, Chapter Nine summarizes the value of the Customer Relationship Intelligence approach.

Linda backs up her argument with some wonderful metaphors. One example is the contrast between data mining and the data farming approach that she proposes with her Relationship Value metric. For data mining, we gather a large pile of data and then use advanced mathematical algorithms to determine which parts of the pile may contain some useful nuggets of information. This is like the hunter-gatherer stage of information management. When we advance into the data farming stage, we know what customer relationship metric is important and collect that data directly.

As the metaphor suggests, we are still in the early days of understanding and developing customer relationship metrics. Until now, these metrics have concentrated on measuring our own performance to see how well we are doing. Linda Sharp’s Relationship Value metric turns this on its head with a new metric that measures our whole relationship with customers. Read the book to discover a new and unified way of thinking about and measuring your customers.

Developing on a Cloud

The cloud computer is here and you can have your corner of it for as little as 10 cents an hour. This was the message that author and consultant Chris Richardson offered to the SDForum SAM SIG when he spoke on "Developing on a Cloud: Amazon's revolutionary EC2" at the SIG's February meeting.

As Chris tells it, you go to the Amazon site, sign up with your credit card, go to another screen where you describe how many cloud servers you need and a couple of minutes later you can SSH to the set of systems and start using them. In practice it is slightly more complicated than this. Firstly, you need to create an operating system configuration with all the software packages that you need installed. Amazon provides standard Linux set ups and you can extend them with your requirements and store the whole thing in the associated Amazon S3 storage array. There goes another 10 cents a month.

Next you need to consider how your cloud servers are going to be used. For example, you could configure a classic 3 tier redundant web server system with 2 cloud servers running web servers, and another 2 cloud servers running tomcat application servers and a another cloud server running the database with yet another cloud server on database standby. Chris has created a framework for defining such a network called EC2Deploy (geddit?). He has also implemented a Maven plug-in that sits on top of EC2Deploy that creates a configuration and starts applications on each server. Needless to say the configuration is defined declaratively through the Maven pom.xml files.

So why would want to use EC2 for? Chris suggested a couple of applications that are particularly interesting for impoverished start ups. Firstly, EC2 can be used to do big system tests before a new version of the software is deployed. The start up does not need buy all the hardware to replicate its production systems so that they can do a full scale system test. Big system tests are done on EC2 saving considerable resources. Another use it to have a backup solution for scaling should the startup take of in an unexpected manner. Given the unreliability of ISPs these days, having a quickly deployable backup system sounds like a good idea, and the best thing is that it does not cost you anything when you are not using it.

Develop Smarter Products

When someone asks me about Business Intelligence, I will usually say that it is about analyzing the data that a business already has, and the truth is that businesses collect huge amounts of useful data. However, there are many interesting applications where we go out and collect specific data for analysis. We heard about one such application at the February meeting of the SDForum Business Intelligence SIG where Cameron Turner, CEO of ClickStream Technologies, spoke on "Software Instrumentation: How to Develop Smarter Products with Built-in Customer Intelligence".

ClickStream Technologies has a data collector that collects user interactions with GUI based user interfaces. That data is loaded into a data warehouse for analysis of the user experience. Contrast the ClickStream method with other techniques for for analyzing program usage. The most common method is to analyze the logs generated by a program, but logs are typically recorded some distance from the user interface and tends to capture the end result of what the user did rather than how the user did it. For example, if there are several ways in which a function can be invoked, programs logs normally record that the function has been invoked, but not how it was invoked. Also collecting more information involves modifying the program to increase the log data that it produces, which is not always practical. The ClickStream data collector runs as a stand alone program and collects its data with minimum intrusion into the running of the program being instrumented. Another technique for gathering user experience data is to have someone stand behind the user and record what they do, but this is labor intensive and does not allow for large scale studies.

There are many reasons for evaluating the user experience with a program. Cameron lists them in his presentation which you can get a copy of by visiting the files area of the Business Intelligence SIG Yahoo Group. The one that is closest to my heart is providing feedback to the designers of a program that their design leaves a lot to be desired. There are many times when I have become frustrated with a program because I cannot find out how to do the the simplest and most obvious thing. This is when I wish that someone was recording my problems and feeding them back to the development team.

ClickStream Technologies started off as a consulting company. They are moving their offering towards something that is more standardized with the idea that eventually customers will be able to use it on a self service basis. Currently each engagement requires configuring the data collector and writing reports for the analysis. Also for each engagement they recruit a panel of testers who download the data collector. As such, is it more suitable for medium to large sized companies that want to do large scale studies.