Duh Typing

The meeting of the SDForum SAM SIG last week was interesting, however the discussion afterwards over a beer was just as interesting. I will write about the SIG meeting another time, here I want to capture an idea from the after session.

There is a lot going on in the world of programming languages, dynamic versus static languages, strong versus weak typing. Proponents of dynamic languages like Python and Perl claim that they are superior because you do not need to continually specify the type of an object, particularly when the type of the object is obvious.

Over a beer someone proposed that this continual restating of the type of an object should be called Duh Typing (a play on Duck Typing). I ran into an example of Duh Typing today when I found myself writing the following Java statement:

ManagedTasks[] managedTasks =
(ManagedTasks[]) tasks.toArray
(new ManagedTasks[tasks.size()]);

The dynamic language guys would laugh at the fact that I have to specify the type 3 times in a single statement. Well, it is not quite as easy all all that as I will discuss in future posts. In the mean time here is a neat name for the concept.

The Times, They Are a-Changing

"Apple’s Steve Jobs, perhaps the most important person in the music industry today, ..."

Well, it is a quote from Michael Arrington who is after all a Valley booster, however it does show how far the music business has come on its strange journey. Gone are the simple days of find and rip-off some gullible kids, now we are at the medium is the message.

The 60 Hour Data Warehouse Implementation

There was a lot of interesting stuff in the presentation by Stephen Bay to the SDForum Business Intelligence SIG on "Large Scale Detection of Irregularities in Accounting Data". However the one thing that really struck me was their 60 hour data warehouse implementation.

Stephen and his colleagues at the PricewaterhouseCoopers Center for Advanced Research have built a system called Sherlock for detecting fraud in accounting data by applying several analytic techniques. Sherlock works by looking at the general ledger of the business. A general ledger is typically several gigabytes of data and may be fed by sub-ledgers that can run into the hundreds of gigabytes. Before Sherlock can do its analytics, they have to get the accounting data into a standard form in a data warehouse. Sherlock is used during an accounting audit which typically lasts a month, so there is great pressure to get the data warehouse implemented in as short a time as possible.

So how do they do it? Firstly, the schema of the data warehouse is fixed. The PricewaterhouseCoopers team have developed a standard data warehouse design for a general general ledger that is applicable to all non-financial businesses. The data warehouse design is open source and is available from IPHIX. Secondly, the general ledger data usually comes from an SAP, Oracle or PeopleSoft ERP system so some of the connections can be prebuilt. The problem with ERP systems is that they are heavily customized for each user, so the Sherlock team have implemented a GUI tool for building a mapping between ERP content and the data warehouse. The tool is designed for business people and accountants to use so that the data warehouse can be built by people with domain knowledge but no technical knowledge.

With all this they claim that they can build a data warehouse in 60 hours for a new implementation and 20 hours for a repeat implementation. Contrast 60 hours with a typical data warehouse project that takes many months. A large data warehouse project can easily take a year or two to implement. So a 60 hour implementation is an astonishing achievement.

Music and the Limits to Consumption

Time and money are limits on the consumption of media. If I am playing a video game, or watching a video, I will not be listening to music. If I listen to less music, I will want to buy less music. Anyway, my budget is not unlimited, so if I spend money on a movie, TV Show or video games, I will not spend that money on buying music. I got to thinking this after reading a piece in the New York Times where a pair of record store owners complain that it is actions of the recording industry and RIAA that have lead to the recent decline in the sales of music.

There was a time when music was the choice media to purchase, but that was when the main alternative was books or magazines. Now we live in a much richer environment where that are many more media choices to entertain us. Music has lost its monopoly on our attention, however the music industry still behaves like it has that monopoly. Rather than market music to us, the RIAA goes out and sues the most ardent music collectors.

There are plenty of things that the recording industry could do to improve their position. For example, they should be marketing music to us as the more entertaining and lasting of the media alternatives. After all, we listen to the same music over and over again, while we usually watch an movie or TV show only once. I have never once heard a compelling argument for buying music being made by the music industry, whereas I have heard interesting arguments being made on forums like SlashDot.

Cut and Paste

It is the little details in a User Interface that make the difference between something that is straightforward and something that is frustrating. For example, lets take the simplest editing function, cut and paste. If text appears on my screen, I want to be able to select the text and paste it into a document, message or whatever. However, there is a lot of text that appears on my screen that I cannot select. There are title bars, menu items and worst of all, all the interesting text in dialog boxes.

Imagine this scenario: a modal dialog box appears on my screen with an important error message. What do I do? Well I cannot select and copy the text, and with a truly modal dialog box, I cannot do anything else until I have dismissed the box, so if I want to preserve the message in the box, I have to find a piece of PAPER to write down what it says, and later transcribe my written notes into the email sent to support. I have been on the receiving end of software errors and so I know that the first question is "What EXACTLY does the message say?". So this is why you need to keep a piece of paper and pencil handy when you use a computer!

I was reminded of this problem today by another frustrating problem with cut and paste. A colleague IMed me to ask for an email address of a third party. As I had to send the third party an email as well, I brought up Outlook (ugg) typed the first three letters of the persons name and auto-completed. Feeling happy that everything could be done in a few keystrokes, I selected the email address in the To: part of the message, copied it and pasted into the IM window. Imagine my surprise when the name of the person appeared in the IM window but not their email address. I had copied the text "name <email>" and when I pasted all that appeared was "name". After 3 goes, I gave up and had to type the email address that appeared in one window into another. Who invented this feature, and what were they thinking?

On reflection, I suppose that I could have gone through the option and disabled all the ones that call themselves intelligent or smart. I did not and I will not. As I have said before in this blog, software should do the right thing out of the box. Anyway, life is too short to customize all the software tools that I use, especially as I am expected to upgrade to new and more complicated versions every few years.

Watch This Space

Is there anyone else who did not recognise Pam Beesly on the front cover of Wired this month?

CRM is Back

There were several interesting things that came out of Jacob Taylor's "CRM is Back" presentation to the SDForum Business Intelligence SIG last week. CRM stands for Customer Relationship Management, an umbrella term for the software and systems for managing sales, marketing and service, the customer facing aspects of a business. Jacob is CTO and co-founder of SugarCRM which is the leading Open Source CRM system. During the talk, he touched on the incredible success of SugarCRM, its relationship with Open Source and the use of the PHP language. This post is about the incredible success of SugarCRM. I will discuss SugarCRM's use of Open Source and PHP in future posts.

SugarCRM was formed in April 2004 and by July had come out with its first product SugarCRM Version 1, a sales CRM module. The company also received Venture Capital funding, the first Open Source software application vendor to do so. From the start SugarCRM generated a huge amount of interest, with as Jacob described many enthusiastic users who offered input, feedback and code to improve the product. By that October, SugarCRM had risen to become Project of the month on SourceForge, the leading repository of Open Source projects.

I recall from the time that SugarCRM had generated so much interest that there were the rumblings of a backlash amongst experienced CRM professionals. The old guard did not understand how this new product that was being made available in a new and low cost way could generate such excitement. However, the excitement is there. Since Sugar has launched there have been more than 3 million downloads of SugarCRM and related projects.

Jacob attributed their rapid success to a number of factors. The first advantage is in the three founders themselves. They have complementary skills in sales, service and engineering and they work well together. Secondly, the founders had a lot of experience with building CRM systems. Before forming SugarCRM they had all been at E.piphany which had been a major CRM system vendor in its day, and prior to that they all had experience with other CRM systems. Given their experience, they all knew exactly what was needed and this allowed them to put together an extremely capable CRM system in a very short period of time.

A third advantage is Open Source and the community that SugarCRM has built around their project. From the earliest days they had an enthusiastic user community who both used and contributed to the project. Sugar soon set up its own SugarForge to accommodate what are now more than 350 Open Source extensions to the project. The Sugar Forums with more than 38,000 members allow users to help each other use SugarCRM. The recently created SugarExchange allows people to sell and exchange products and services based on SugarCRM.

As Jacob put it, Open Source is "Passionware". People want to be involved in the tools they use every day and Open Source offers them new ways in which they can be involved. SugarCRM has been a leader at involving their users and building a community. The reward is that they have created a large number of passionate advocates of the product.

The DRM Battle

The other day, someone on Slashdot commented that there had been a lot of posts on Digital Rights Management (DRM) recently. Well there have been and there will continue to be DRM posts because Intellectual Property Rights have become a huge issue caused by technology advances with computing and the Internet.

I have written about DRM frequently in the past and will continue to do so, as there is a lot to say. In the mean time, here is great quote that precisely expresses some of my feelings. It comes from Lifeline, the first published work by science fiction writer Robert Heinlein. It was written in 1939, some time before the RIAA even existed:

There has grown up in the minds of certain groups in this country the notion that because a man or a corporation has made a profit out of the public for a number of years, the government and the courts are charged with the duty of guaranteeing such profit in the future, even in the face of changing circumstances and contrary public interest. This strange doctrine is not supported by statute nor common law. Neither individuals nor corporations have any right to come into court and ask that the clock of history be stopped, or turned back.

Two Disruptive Trends

Barry Klawans gave an excellent talk to the SDForum Business Intelligence SIG when he spoke on "Two Disruptive Trends, Open Source and SaaS Meet Business Intelligence" at the February meting. Open Source is something that I have written about before. SaaS stands for Software as a Service, the idea that information technology can be delivered as a service over the internet. The best example of a successful SaaS business is Salesforce.com. Barry knows the territory well as he is CTO of JasperSoft, an Open Source BI reporting company.

Barry started with the Innovators Dilemma, a book from the 90s that describes how established technologies and products markets can be overturned by innovators who use new disruptive technology or business models. Existing BI software vendors tend to target the high end of the market and they are vulnerable to disruption from new vendors that start by targeting the under served lower end. Barry believes that Open Source and SaaS are the forces that will overthrow the old guard of established BI vendors.

Next, Barry took us through the BI stack and Open Source projects that address it. Successful Open Source projects concentrate on doing one thing and doing it well. One of the problems with using Open Source is that you have to integrate several Open Source packages to build a system. Integration is made more difficult because active Open Source projects tend to have a very short release cycle. The most active project have a new release every 6 weeks or so. (This certainly struck a chord with me.)

This brought us to the second part of Barry's talk on Software as a Service (SaaS). The point of a SaaS system is to offload the user from the responsibility of building and maintaining an IT system. The job of building a SaaS system is integrating a lot software packages to provide the service. SaaS also has to deal with transparently upgrading the service to the users as it implements new features and fix bugs. As such it complements Open Source and it rapid development cycle.

SaaS is a newer market and there are only a few emerging SaaS BI services available now. Barry touched on three, LucidEra, SeaTab and Oco, all early stage start ups. There are some real architectural challenges to providing BI as a service. For example, one issue is security. For a number of reasons, many Open Source projects put security on the back burner. On the other hand, a Saas customer needs to have solid assurances that their data is secure and safe from other customers of the SaaS service.

We will have to see how these trends play out. Barry quoted a research report that suggested that software innovation goes in a roughly 15 year cycle, and that in Business Intelligence we are just entering a new cycle that can be expected to go on to 2020 or so.

Like OMG

I overhear my daughter speaking on her new cellphone in the next room, and I get to thinking: OMG, does it mean "Object Management Group" or "Oh My God"? Well there is only one way to settle it: Google Fight. The results when they come in lean in the expected direction, but the results are not as overwhelming as anticipated.

So like any good data analyst I do a little exploration to check the results. I do not think that there is any problem with the Object Management Group side of the equation. On the other hand there are several potential spellings of Oh My God that could make a difference. A Quick test of "Oh Mi God" shows that it is not a problem and "Oh My God" is much more popular than the "O My God"spelling.

The next thing to test is whether the quotes are necessary. Removing the quotes quickly shows wobbly results, particularly with a big difference between O My God and Oh My God. Although curiously, the first page of O My God results from Google all show Oh My God as the search term. Google Fight shows Object Management Group winning over Oh my God but losing to O My God. My conclusion is that the quotes are necessary even although they make a less interesting Google Fight.

The other notable thing is that the numbers returned by Google Fight are not the same as the numbers returned by Google. Why? Well it is probably because they have a different setting of the advanced search options. You can spend hours over this stuff as my little example has shown. In the end the first Goggle Fight results seem to stand up, so perhaps the conclusion is that the world is more serious that I first though!