The evolution of the App Economy is a marvelous thing to watch. In March I questioned whether apps for the iPad would develop with the same strength as apps for the iPhone, because more content is accessible through the browser. Jacob Weisberg at Slate discussed the same thing recently in more depth. He exhorts publishers to beware of getting tangled up with Apple for both monetary and censorship reasons.
On the other hand, web content is not fully available on the iPad. Steve Jobs has denigrated Flash for being slow, buggy and inefficient, and has sworn that it will never be seen on the iPad. In its place Jobs suggests HTML5. The problem is that HTML5 does not do everything that Flash does. This recent piece on on Apple Insider explains the shortcomings of HTML5 and why Hulu will not be using it any time soon for their video distribution.
If Hulu cannot use Flash, then its only alternative is to develop an App, which it is reportedly doing. If Hulu has an App, it may charge a subscription as is being discussed. If Hulu charges a subscription, some of that revenue flows to Apple. By banning a rival development platform, Apple is encouraging the App Economy to its own advantage. Thus it is a pity that so many of the early publishing apps have received such bad reviews.
Wednesday, May 19, 2010
Sunday, May 09, 2010
Google Books Rocks
Awesome is too small a word to express what Google Books has achieved. Last year Google settled the class action law suit that allows them to index out of print books that they had digitized. As part of the settlement they also have to sell the books, which means that Google is now a bookseller. The most important part of the settlement is the Books Right Registry:
I will write more about this issue another day. For now, here is how I stumbled upon the awesomeness of Google Books. My father would often quote "but tomorrow by the living god, we'll try the game again" after some setback. I knew it was from a poem, but not much more. So the other day, I typed "but tomorrow by the living god" into Google and was astonished by the progress that has been made in search over the last few years. The first entry in the search results linked to a poetry anthology in Google Books that has the full poem by John Masefield.
Masefield is best known for his poems "Sea Fever", "I must go down to the seas again, to the lonely seas and the sky, ..." and "Cargoes", "Quinquireme of Nineveh from distant Ophir, ..." For poem collectors, here is the rarely seen poem TOMORROW by John Masefield:
"The agreement will also create an independent, not-for-profit Book Rights Registry to represent authors, publishers and other rightsholders. In essence, the Registry will help locate rightsholders and ensure that they receive the money their works earn under this agreement. You can visit the settlement administration site, the Authors Guild or the AAP to learn more about this important initiative."One of the biggest practical issue with Intellectual Property is that it is impossible to use most Intellectual Property because you do not know who owns it, and therefore you do not know who to ask for permission to use it. Laurence Lessig has been talking about this for a long time as a part of his campaign to fix copyright laws. The establishment of a Book Rights Registry goes some way to address the problem with one type of Intellectual Property. Perhaps this will be the beginning of a trend.
I will write more about this issue another day. For now, here is how I stumbled upon the awesomeness of Google Books. My father would often quote "but tomorrow by the living god, we'll try the game again" after some setback. I knew it was from a poem, but not much more. So the other day, I typed "but tomorrow by the living god" into Google and was astonished by the progress that has been made in search over the last few years. The first entry in the search results linked to a poetry anthology in Google Books that has the full poem by John Masefield.
Masefield is best known for his poems "Sea Fever", "I must go down to the seas again, to the lonely seas and the sky, ..." and "Cargoes", "Quinquireme of Nineveh from distant Ophir, ..." For poem collectors, here is the rarely seen poem TOMORROW by John Masefield:
Oh yesterday the cutting edge drank thirstily and deep,In my original search results, there was a link to Google newspapers where a Virgin Islands Daily News edition from 1950 quotes part of the poem. This time when I did the search, that link did not come up. Instead there was a link to a 1991 zine for Vietnam War vets that quotes a verse of the poem. Who knows what you may find when you do the search.
The upland outlaws ringed us in and herded us as sheep,
They drove us from the stricken field and bayed us into keep;
But tomorrow
By the living God, we'll try the game again!
Oh yesterday our little troop was ridden through and through,
Our swaying, tattered pennons fled a broken, beaten few,
And all a summer afternoon, they hunted us and slew;
But tomorrow
By the living God, we'll try the game again!
And here upon the turret-top the bale-fires glower red,
The wake-lights burn and drip about our hacked, disfigured dead,
And many a broken heart is here and many a broken head;
But tomorrow
By the living God, we'll try the game again!
Friday, April 30, 2010
More on The Big Short
When I wrote that Michael Lewis had written an almost uplifting account of the financial crisis in "The Big Short
" by concentrating on some of the winners, I did not consider that he was keeping something back. If you want to find out what he really thinks, read this interview on Bloomberg.com. He explains many of the choices that he made in the book, like for instance not including John Paulson who has been celebrated in other places for "The Greatest Trade Ever
". He also expresses his outrage over what happened and suggests that part of the reason he left Wall Street in 1989 was because his job was basically "exploiting the idiocy of my customers". It is a long interview and well worth reading in its entirety.
One issue that Lewis touches on is the fact that shorting the market is supposed to dampen the market and perhaps bring sanity into it, but in this case the structured investment vehicles like synthetic CDOs had the opposite effect of amplifying the market and making the subsequent downfall much worse. The "This American Life" radio show and podcast has a recent segment where they discuss the role of the Magnetar Hedge Fund in creating many several subprime bonds and then making huge sums of money by shorting parts of them. Again well worth hearing.
Finally, Lewis discusses the poisonous interface between the big Wall Street firms and their customers. If Goldman Sachs is responsible for defrauding its customers as the recent lawsuit suggests, there is the question of why anyone would want to do business with them. The Big Money blog posits that Goldman Sachs is losing its "Social License" to operate in an interesting post. Given their behavior, this may be a good thing.
" by concentrating on some of the winners, I did not consider that he was keeping something back. If you want to find out what he really thinks, read this interview on Bloomberg.com. He explains many of the choices that he made in the book, like for instance not including John Paulson who has been celebrated in other places for "The Greatest Trade Ever". He also expresses his outrage over what happened and suggests that part of the reason he left Wall Street in 1989 was because his job was basically "exploiting the idiocy of my customers". It is a long interview and well worth reading in its entirety.One issue that Lewis touches on is the fact that shorting the market is supposed to dampen the market and perhaps bring sanity into it, but in this case the structured investment vehicles like synthetic CDOs had the opposite effect of amplifying the market and making the subsequent downfall much worse. The "This American Life" radio show and podcast has a recent segment where they discuss the role of the Magnetar Hedge Fund in creating many several subprime bonds and then making huge sums of money by shorting parts of them. Again well worth hearing.
Finally, Lewis discusses the poisonous interface between the big Wall Street firms and their customers. If Goldman Sachs is responsible for defrauding its customers as the recent lawsuit suggests, there is the question of why anyone would want to do business with them. The Big Money blog posits that Goldman Sachs is losing its "Social License" to operate in an interesting post. Given their behavior, this may be a good thing.
Monday, April 26, 2010
Business Rules OK!
Performance Management Systems collect the data to make decisions but they do not make decisions, they do not ensure that decisions get made or even track the results of the decision so made. James Taylor (no relation) called this the "over-instrumented" enterprise when he spoke to the the April meeting of the SDForum Business Intelligence SIG on "Performance Management and Agility". James is CEO of Decision Management Solutions where he consults on using technology to better effect decision making.
James divides the decisions that an organization makes into three levels: strategic, tactical and operational. He is interested in the operation decisions, the little decisions that are taken all the time. An example of an operational decision is what offer to make to a customer that has called a call center. Every enterprise has their own set of operational decisions, however they have the characteristic that is a large number of them that in aggregate they represent a lot of value, so they are well worth managing.
Many operational decisions are or should be automated, and there are a set of principles that need to be recognized when decision making is automated. The first principle is that no decision is going to be forever, so the logic for making the decision should not be locked up into something inflexible such as program code. Much better to use a rules based decision engine which allows everybody to see the rules in a language that they can understand. Another principle is that making a decision is a business process and as such should be managed. A good business rules engine allows rules to be tested, measured and perhaps even simulated in action to understand what they are doing and how they can be optimized.
According to James, the purpose of the information gathered for a Performance Management Systems is to make decisions, so it should be used to make decisions. Too many enterprises are over-instrumented. They have spent all their effort to get and present the data, however they have no measurable ability to turn that data into actions. You can read more about these ideas in the book Smart Enough Systems: How to Deliver Competitive Advantage by Automating Hidden Decisions
by James Taylor and Neil Raden. You can also read my co-chair Paul O'Rorke's take on the meeting in his blog.
James divides the decisions that an organization makes into three levels: strategic, tactical and operational. He is interested in the operation decisions, the little decisions that are taken all the time. An example of an operational decision is what offer to make to a customer that has called a call center. Every enterprise has their own set of operational decisions, however they have the characteristic that is a large number of them that in aggregate they represent a lot of value, so they are well worth managing.
Many operational decisions are or should be automated, and there are a set of principles that need to be recognized when decision making is automated. The first principle is that no decision is going to be forever, so the logic for making the decision should not be locked up into something inflexible such as program code. Much better to use a rules based decision engine which allows everybody to see the rules in a language that they can understand. Another principle is that making a decision is a business process and as such should be managed. A good business rules engine allows rules to be tested, measured and perhaps even simulated in action to understand what they are doing and how they can be optimized.
According to James, the purpose of the information gathered for a Performance Management Systems is to make decisions, so it should be used to make decisions. Too many enterprises are over-instrumented. They have spent all their effort to get and present the data, however they have no measurable ability to turn that data into actions. You can read more about these ideas in the book Smart Enough Systems: How to Deliver Competitive Advantage by Automating Hidden Decisions
by James Taylor and Neil Raden. You can also read my co-chair Paul O'Rorke's take on the meeting in his blog.
Thursday, April 15, 2010
And the Future of Television is ...
Wait for it, wait for it ... Sports! The path to this conclusion requires a couple of steps, so bear with me. The Convergence Consulting Group just published their annual report on "The Battle for the North American Couch Potato", and several news sources and commentators immediately picked up on one element of their report. According to Convergence, by the end of 2009, 800,000 US households had cut the cable and that they expected this to double to 1.6 million households by 2011. Cutting cable means cutting subscription TV service like cable or satellite and getting all media content from the Internet, Netflix and over the air. I recently wrote about Television being in trouble because of increasing subscription fees and less content. There has been a trickle of cable cutters for some time and now Convergence Consulting tells us that the numbers are starting to swell.
So why should we not cut the cable? It turns out that sports is the only type of content where subscription television offers a compelling product that you cannot easily get if you cut the cable. I came to this conclusion after skimming through the comments on TechCrunch post on the cable cutting story. The majority of comments are either from people who have cut the cable and the only thing they miss is sports, or from people who say that they cannot cut the cable because they would not be able to get the sports that they want to see. The fact that sports is the only type of content mentioned is quite startling.
So why should we not cut the cable? It turns out that sports is the only type of content where subscription television offers a compelling product that you cannot easily get if you cut the cable. I came to this conclusion after skimming through the comments on TechCrunch post on the cable cutting story. The majority of comments are either from people who have cut the cable and the only thing they miss is sports, or from people who say that they cannot cut the cable because they would not be able to get the sports that they want to see. The fact that sports is the only type of content mentioned is quite startling.
Sunday, April 11, 2010
The Big Short
What is the best way to write an uplifting book about the recent financial crisis? In his new book, The Big Short, Michael Lewis has taken the approach of following the winners, the people who saw that the bubble would burst and made a huge sums of money by betting on it bursting. Along the way we also meet some of the people who took the other side of the bet and lost big. When you compare these two groups, the losers come across as your average every day kind of person while the winners area strange group of outsiders.
The book follows three groups of people. There is the Frontpoint Partners hedge fund led by Steve Eisman, who as a stock analyst had been best known for correctly trashing companies that he followed. Then there is Mike Burry, a one eyed doctor living in San Jose who took up investing because it allowed him to get away from having to interact with other people. Finally there are the guys at the garage band hedge fund who turn $100,000 in to more than $100 million and whose main problem is being taken seriously by the big Wall Street Firms.
Along the way we see scenes of madness from the financial machine that created the bubble. There is the explanation for why a Mexican strawberry picker with no English and an income of $14,000 per year could be loaned every penny he needed to buy a house for $724,000 in Bakersfield California. As it turns out, because he had no debt and no credit history, he has a relatively high credit rating, and that credit rating was needed to balance out the low credit rating of some deadbeat American when their home loans were packaged together with many others into a mortgage bond.
Another scene is the American Securitization Forum, the annual conference of the of the subprime mortgage industry. In early 2007 the conference takes place in the Venetian Hotel in Las Vegas. The Frontpoint Partners and the garage band hedge fund are both there trying to get more information to substantiate their huge bets against the subprime mortgage market. By this time the cracks were beginning to show. For example, the CEO of the Option One mortgage corporation gave a reassuring speech even although Option One was in trouble because they had made loans to people who could not afford to make even the first payment on the loan. When his partner asks "Who takes out a home loan and doesn't make the the first payment?" Steve Eisman responds "Who the #$%^ lends money to people who can't make the first payment?"
Michael Lewis knows what he is talking about in writing about Wall Street because he started his career as a bond salesman for Salomon Brothers as hilariously told in his first book, Liar's Poker
. In The Big Short he successfully continues that tradition. The Big Short is full of interesting characters, amusing insights, clear explanations and some genuine tension as towards the end you wonder whether the hero's will get their big payoff or whether Wall Street will totally collapse taking down everybody with them. Like of his other books, The Big Short is highly recommended.
, Michael Lewis has taken the approach of following the winners, the people who saw that the bubble would burst and made a huge sums of money by betting on it bursting. Along the way we also meet some of the people who took the other side of the bet and lost big. When you compare these two groups, the losers come across as your average every day kind of person while the winners area strange group of outsiders.The book follows three groups of people. There is the Frontpoint Partners hedge fund led by Steve Eisman, who as a stock analyst had been best known for correctly trashing companies that he followed. Then there is Mike Burry, a one eyed doctor living in San Jose who took up investing because it allowed him to get away from having to interact with other people. Finally there are the guys at the garage band hedge fund who turn $100,000 in to more than $100 million and whose main problem is being taken seriously by the big Wall Street Firms.
Along the way we see scenes of madness from the financial machine that created the bubble. There is the explanation for why a Mexican strawberry picker with no English and an income of $14,000 per year could be loaned every penny he needed to buy a house for $724,000 in Bakersfield California. As it turns out, because he had no debt and no credit history, he has a relatively high credit rating, and that credit rating was needed to balance out the low credit rating of some deadbeat American when their home loans were packaged together with many others into a mortgage bond.
Another scene is the American Securitization Forum, the annual conference of the of the subprime mortgage industry. In early 2007 the conference takes place in the Venetian Hotel in Las Vegas. The Frontpoint Partners and the garage band hedge fund are both there trying to get more information to substantiate their huge bets against the subprime mortgage market. By this time the cracks were beginning to show. For example, the CEO of the Option One mortgage corporation gave a reassuring speech even although Option One was in trouble because they had made loans to people who could not afford to make even the first payment on the loan. When his partner asks "Who takes out a home loan and doesn't make the the first payment?" Steve Eisman responds "Who the #$%^ lends money to people who can't make the first payment?"
Michael Lewis knows what he is talking about in writing about Wall Street because he started his career as a bond salesman for Salomon Brothers as hilariously told in his first book, Liar's Poker
. In The Big Short he successfully continues that tradition. The Big Short is full of interesting characters, amusing insights, clear explanations and some genuine tension as towards the end you wonder whether the hero's will get their big payoff or whether Wall Street will totally collapse taking down everybody with them. Like of his other books, The Big Short is highly recommended.
Sunday, April 04, 2010
Web Analytics 2.0
Web analytics is changing fast as we discovered at the March meeting of the SDForum Business Intelligence SIG. Avinash Kaushik, Analytics Evangelist for Google, spoke on 'Web Analytics 2.0: Rethinking Decision Making in a "2.0" World'. Avinash started off by telling us how he became well known as a web analytics guru. A few years ago he started writing his blog "Occam's Razor". It soon gathered a large readership and a publisher approached him to write a book. His first book "Web Analytics: An Hour a Day
" was a distillation of his blog posts. The book is a best seller even although much of its contents is available for free on the web. His second book "Web Analytics 2.0
" came out recently.
Avinash is an excellent communicator with a strong personal style. One aspect of that style, quite obvious from his blog posts, is the urge to create lists of ideas. For his presentation, Avinash offered us a list of simple ideas on web sites metrics and analytics. Here are some of the ideas that he presented.
The first idea is simple and direct - Don't Suck. The suckage of a web page can be measured by a metric called Bounce. This is a relatively new metric that we had not previously heard discussed at the Business Intelligence SIG. Bounce measures the users whose experience of the web page and site is, as Avinash put it "I came. I puked. I left." and he showed us some pretty pukey pages that might back up this behavior. A typical analysis is to look at the pages with the highest bounce rate, determine why they cause that behavior and what can be done about it.
His next idea is Segment or Die. Analytics is about aggregating data to make sense of large datasets, however over-aggregation results in a single number and nothing to compare it with. Segmenting the data gives us a number of data items that we can compare. Avinash showed us a simple example where he took a hospital web site and classified the content into 8 segments and then compared the amount of content against the number of page views in each segment. It was immediately apparent where the effort should go into adding and improving content.
Analyzing your web logs only tells a part of the story, you also have to worry about what the analytics cannot tell you. Ex Defense Secretary Donald Rumsfeld is infamous for having talked about "the known knowns, the known unknowns, and the unknown unknowns". The unknown unknowns are the things that you don't even know that you don't know and therefore the thing you should be most worried about. You can start to get a handle on what you do not know by looking at your performance relative to your competitors. This is known as Benchmarking, or in the case of a deep study as Competitive Intelligence. For an example of what can be done, a recent post on the Occam's Razor blog discusses 8 sources for Competitive Intelligence data.
Most web site analytics only looks for the one big conversion from a web site, however there are many other small conversions that are tracked and worth evaluating because there may be hidden value lurking in the long tail. For example, recently Avinash wanted to know what his blog was worth so that he could defend taking time away from the family to write it. After determining a value for each reader, he started adding up all the other micro-conversion like people who subscribe to the RSS feed and advertisements for his books and the non-profit organizations that he supports. Overall he came up with a figure of about $26000 per month. Now Avinash does not make a penny from his blog, so this is notional money that adds to his personal brand value, but that value seems to make the effort of writing the blog well worth the time spent.
The next idea is Fail Faster. By this Avinash means do lots of different experiment, many of which will fail, to find out what works. He led us through an example from the Obama Presidential campaign. President Obama raised huge amounts of money from many small donations on his web site. The initial web page worked well. The experiments were to try some variations on the theme. Pages with video, and stirring video at that, did very badly. A simple picture of Obama with his family did a little better than the initial picture, so that one was chosen.
Avinash showed us this example to make a number of points. The Obama analytics team was tiny. Often the best work is done by a small agile team that has the freedom to experiment. The team used free tools. Avinash believes that that good people are much more important than good tools. His suggestion for dividing up the analytics budget is to spend 90% on people and 10% on tools. Sometimes, a web site design feature starts from a HiPPO (Highest Paid Persons Opinion), which can be destructively bad, and difficult to get around because in all organizations the highest paid persons opinion is taken very seriously. The best way to counter a HiPPO is to show that other ideas work better through the results of experiments that produce hard evidence.
While some may think that web analytics is a mostly solved problem, Avinash believes we are just starting to figure out what can be done, and that there is plenty of room for more innovation. I will continue to read Occam's Razor to find out where he takes us next.
" was a distillation of his blog posts. The book is a best seller even although much of its contents is available for free on the web. His second book "Web Analytics 2.0" came out recently.Avinash is an excellent communicator with a strong personal style. One aspect of that style, quite obvious from his blog posts, is the urge to create lists of ideas. For his presentation, Avinash offered us a list of simple ideas on web sites metrics and analytics. Here are some of the ideas that he presented.
The first idea is simple and direct - Don't Suck. The suckage of a web page can be measured by a metric called Bounce. This is a relatively new metric that we had not previously heard discussed at the Business Intelligence SIG. Bounce measures the users whose experience of the web page and site is, as Avinash put it "I came. I puked. I left." and he showed us some pretty pukey pages that might back up this behavior. A typical analysis is to look at the pages with the highest bounce rate, determine why they cause that behavior and what can be done about it.
His next idea is Segment or Die. Analytics is about aggregating data to make sense of large datasets, however over-aggregation results in a single number and nothing to compare it with. Segmenting the data gives us a number of data items that we can compare. Avinash showed us a simple example where he took a hospital web site and classified the content into 8 segments and then compared the amount of content against the number of page views in each segment. It was immediately apparent where the effort should go into adding and improving content.
Analyzing your web logs only tells a part of the story, you also have to worry about what the analytics cannot tell you. Ex Defense Secretary Donald Rumsfeld is infamous for having talked about "the known knowns, the known unknowns, and the unknown unknowns". The unknown unknowns are the things that you don't even know that you don't know and therefore the thing you should be most worried about. You can start to get a handle on what you do not know by looking at your performance relative to your competitors. This is known as Benchmarking, or in the case of a deep study as Competitive Intelligence. For an example of what can be done, a recent post on the Occam's Razor blog discusses 8 sources for Competitive Intelligence data.
Most web site analytics only looks for the one big conversion from a web site, however there are many other small conversions that are tracked and worth evaluating because there may be hidden value lurking in the long tail. For example, recently Avinash wanted to know what his blog was worth so that he could defend taking time away from the family to write it. After determining a value for each reader, he started adding up all the other micro-conversion like people who subscribe to the RSS feed and advertisements for his books and the non-profit organizations that he supports. Overall he came up with a figure of about $26000 per month. Now Avinash does not make a penny from his blog, so this is notional money that adds to his personal brand value, but that value seems to make the effort of writing the blog well worth the time spent.
The next idea is Fail Faster. By this Avinash means do lots of different experiment, many of which will fail, to find out what works. He led us through an example from the Obama Presidential campaign. President Obama raised huge amounts of money from many small donations on his web site. The initial web page worked well. The experiments were to try some variations on the theme. Pages with video, and stirring video at that, did very badly. A simple picture of Obama with his family did a little better than the initial picture, so that one was chosen.
Avinash showed us this example to make a number of points. The Obama analytics team was tiny. Often the best work is done by a small agile team that has the freedom to experiment. The team used free tools. Avinash believes that that good people are much more important than good tools. His suggestion for dividing up the analytics budget is to spend 90% on people and 10% on tools. Sometimes, a web site design feature starts from a HiPPO (Highest Paid Persons Opinion), which can be destructively bad, and difficult to get around because in all organizations the highest paid persons opinion is taken very seriously. The best way to counter a HiPPO is to show that other ideas work better through the results of experiments that produce hard evidence.
While some may think that web analytics is a mostly solved problem, Avinash believes we are just starting to figure out what can be done, and that there is plenty of room for more innovation. I will continue to read Occam's Razor to find out where he takes us next.
Monday, March 29, 2010
How Data is Changing the Study of Economics
Andrew Leonard in How The World Works recently posted on how computers and the availability of data is changing the study of Economics, and I have to agree. There are a number of forces that are converging to make this happen right now.
Open Government initiatives are making more data available and the internet makes it easier to get the data. Emerging movements like the Open Data Commons emulate the Open Source movement that has made software more available. The Open Data movements are concerned to not only make the data more openly available but to make the data better by providing tools to manage it and inspection so that problems with the data can be corrected.
Web sites to make data available have been around for some time. For example, Numbary.com exists to make public data more available. Sites like Many Eyes and Swivel allow the user to upload data sets and analyze them. You do not need to find your own data sets because you can go to these sites and play around with data sets that others have uploaded.
Several popular books have shown us what can be done. The best known example is Freakonomics, which takes a number of interesting data sets and shows us how they can be analyzed to tell interesting and sometimes quite startling stories. Less flamboyant and more educational is Super Crunchers
, subtitled "why thinking-by-numbers is the new way to be smart".
Leonard suggests that the rise of large scale data analysis will displace the old guard who sit in their ivory towered and built model. I have to disagree. The economic model is the explanation of what is happening, the result of analysis. Building a model to explain some aspect of the data or behavior that is brought to light by the data is the result of Analytics. More and better data means that the models will be better, more definitive and most importantly in a fractious discipline, more defensible.
Open Government initiatives are making more data available and the internet makes it easier to get the data. Emerging movements like the Open Data Commons emulate the Open Source movement that has made software more available. The Open Data movements are concerned to not only make the data more openly available but to make the data better by providing tools to manage it and inspection so that problems with the data can be corrected.
Web sites to make data available have been around for some time. For example, Numbary.com exists to make public data more available. Sites like Many Eyes and Swivel allow the user to upload data sets and analyze them. You do not need to find your own data sets because you can go to these sites and play around with data sets that others have uploaded.
Several popular books have shown us what can be done. The best known example is Freakonomics, which takes a number of interesting data sets and shows us how they can be analyzed to tell interesting and sometimes quite startling stories. Less flamboyant and more educational is Super Crunchers
, subtitled "why thinking-by-numbers is the new way to be smart".Leonard suggests that the rise of large scale data analysis will displace the old guard who sit in their ivory towered and built model. I have to disagree. The economic model is the explanation of what is happening, the result of analysis. Building a model to explain some aspect of the data or behavior that is brought to light by the data is the result of Analytics. More and better data means that the models will be better, more definitive and most importantly in a fractious discipline, more defensible.
Wednesday, March 24, 2010
The iPad App Conundrum
While the iPad looks like it is going to be successful, I think that there is a question over how the market for iPad Apps will develop. Apps for the iPhone are a stunning success that caught many by surprise. I recall a post in TechCrunch when the iPhone App-store was about to be introduced. Using numbers that were probably leaked from Apple, the column predicted a substantial and valuable market for iPhone apps. The commenters were full of scorn, suggesting that the idea of anyone paying money for little apps was ridiculous. Apple showed us just how it could be done.
On the other hand we have the iPad. Apps for the iPhone make sense because of its small screen. Each app makes the best use of the limited screen space for its own dedicated purpose. The iPad has a much larger screen where the browser with scripting and plug-ins can support most of the experience. Therefore the need for specialized apps is less compelling.
There will still be iPad apps. For example I expect games to do well. However, media companies hoping to monetize their content through subscriptions have a tricky tightrope to walk. Most media companies now make their content available for free on the internet supported by advertising. They cannot afford to withdraw this content completely, but on the other hand if they want to monetize through the iPad, they have to provide a value added experience through their app if they expect to get people to pay. I look forward with interest to see how this all plays out.
Update: TechCrunch just posted on how magazines might work this issue with video on the cover.
On the other hand we have the iPad. Apps for the iPhone make sense because of its small screen. Each app makes the best use of the limited screen space for its own dedicated purpose. The iPad has a much larger screen where the browser with scripting and plug-ins can support most of the experience. Therefore the need for specialized apps is less compelling.
There will still be iPad apps. For example I expect games to do well. However, media companies hoping to monetize their content through subscriptions have a tricky tightrope to walk. Most media companies now make their content available for free on the internet supported by advertising. They cannot afford to withdraw this content completely, but on the other hand if they want to monetize through the iPad, they have to provide a value added experience through their app if they expect to get people to pay. I look forward with interest to see how this all plays out.
Update: TechCrunch just posted on how magazines might work this issue with video on the cover.
Saturday, March 20, 2010
Emerging Languages Face Off
New programming languages are popping up all over the place. In March the SDForum Emerging Tech SIG held an "Emerging Languages Face Off" to try and make sense out of what is going on. The new languages represented at the meeting were Clojure, Scala and Go, with Ruby as a more established control language. The panel was moderated by Steve Mezak, author of "Software without Borders" and CEO of Accelerance, Inc.
The night kicked off with Amit Rathore, Chief Software Architect at Runa, Inc. speaking for Clojure (pronounced closure). He told us that Clojure is a Lisp that runs on the Java Virtual Machine. Lisps are dynamic, functional languages with automatic garbage collection that have been around since the early 60's. Although Amit told us that Clojure programs contain less parenthesis than Java, the examples he showed us did not seem to bear this out. Clojure does try to control the amount of brackets by using both round and square ones. Apart from list, Clojure provides support for both common data structures like Map and lazy sequences.
More importantly, Amit introduced what turned out to be a major theme of the evening, support for concurrency. Clojure has a surprisingly sophisticated (read complicated) support for concurrency. The basic idea is that reads are versioned to be lock free while writes are managed to ensure that they overlap properly. Existing data is immutable, updates are made by writing the new data to new locations. Access to shared memory is delimited by transactions that correspond to the program block structure (good). There are 4 ways of referencing data in a transaction that allow the different use cases each to be handled efficiently. If a transaction fails, it is automatically retried until it succeeds. Their implementation of concurrency goes under the banner of Software Transactional Memory.
While I will give Clojure two and a half cheers for trying, I am not a great fan of Lisp like languages. Amit touched on one of my bugaboos, the ability to change the meaning of the language by writing code. In my mind, this makes Lisp a low level language as any program requires close reading to discover what it might do. Also, my only practical experience of Lisp is Emacs configuration, a scary mess of global variables and functions.
Next up, Evan Phoenix, lead developer of Rubinius spoke about Ruby. Ruby is a dynamic language with automatic garbage collection and is built on the principal of least surprise. While there are several implementations of the language, these implementations have not provided the best performance, so Rubinius is working on a high performance implementation, where more of the implementation is in Ruby itself. The genesis of Ruby was with Lisp and Smalltalk, although the actual language went in a very different direction than these two languages.
Evan admitted that Ruby does not have great support for concurrency. Ruby will work with green threads, that is cooperative multiprocessing that can exploit a single core. The problem of the "interpreter lock" means that a native threads support is not an immediate goal.
After Evan, David Pollak, author of "Beginning Scala" and Benevolent Dictator for Life of the Lift Web Framework spoke on Scala. Scala is a hybrid object oriented/functional language with static typing and garbage collection that runs on the Java Virtual Machine. In some ways it is like Java with less words, the type inference system eliminates the need to explicitly specify data types most of the time. The goal of Scala is to achieve the speed (and safety) of Java with the conciseness of Ruby.
Scala has no specific built in support for concurrency, however the Actors paradigm has been successfully implemented on top of Scala. I have written about both Scala and Actors with Scala previously, so I will say no more here. It is worth noting that both Clojure and Scala get full native threads support and many other benefits from running on the Java Virtual Machine.
Finally Robert Griesemer spoke about the new Go language. Robert is a member of the team developing Go at Google. Go is a statically typed language with automatic garbage collection that compiles down to native hardware. It is a system programming language with control over memory layout of the data. Robert listed the problems with current system programming languages. They are are verbose and repetitious, the data type system gets in the way, build time is slow, particularly compared to dynamic languages, and managing dependencies between modules is difficult. Go aims to be a simple and powerful language with fast tools.
Go does not have inheritance or type hierarchies and is not object oriented, rather it aims to be more flexible. I was somewhat disturbed by this. Although type hierarchies can be misused, they are useful for helping to organizing large projects. On the other hand, for concurrency, Go offers lightweight processes that communicate via channels, which is a welcome move away from the threads paradigm with all its problems.
The Go language is not quite complete yet. The language designers are still working on providing support for a number of features including generics, operators and exceptions. Robert told us that he expects the language to be complete and mature in 6 months to a year from now. Programming language design is not easy and needs to proceed at its own pace. It is worth remembering that the C++ language spend about 15 years in the state of being almost but not quite finished. We will have to wait and see how long it takes Go to mature.
Overall there were two themes that emerge from the the new languages presented at the meeting. One is the desire to make programming simpler and more approachable. It is easier to start writing a program in a dynamic language. Both Scala and Go are statically typed languages with the goal of making the programming experience more like writing a program in a dynamic language. The other theme is the need for new programming languages to provide better support for concurrency. In particular language need to support something that is safer and more controlled than threads.
The night kicked off with Amit Rathore, Chief Software Architect at Runa, Inc. speaking for Clojure (pronounced closure). He told us that Clojure is a Lisp that runs on the Java Virtual Machine. Lisps are dynamic, functional languages with automatic garbage collection that have been around since the early 60's. Although Amit told us that Clojure programs contain less parenthesis than Java, the examples he showed us did not seem to bear this out. Clojure does try to control the amount of brackets by using both round and square ones. Apart from list, Clojure provides support for both common data structures like Map and lazy sequences.
More importantly, Amit introduced what turned out to be a major theme of the evening, support for concurrency. Clojure has a surprisingly sophisticated (read complicated) support for concurrency. The basic idea is that reads are versioned to be lock free while writes are managed to ensure that they overlap properly. Existing data is immutable, updates are made by writing the new data to new locations. Access to shared memory is delimited by transactions that correspond to the program block structure (good). There are 4 ways of referencing data in a transaction that allow the different use cases each to be handled efficiently. If a transaction fails, it is automatically retried until it succeeds. Their implementation of concurrency goes under the banner of Software Transactional Memory.
While I will give Clojure two and a half cheers for trying, I am not a great fan of Lisp like languages. Amit touched on one of my bugaboos, the ability to change the meaning of the language by writing code. In my mind, this makes Lisp a low level language as any program requires close reading to discover what it might do. Also, my only practical experience of Lisp is Emacs configuration, a scary mess of global variables and functions.
Next up, Evan Phoenix, lead developer of Rubinius spoke about Ruby. Ruby is a dynamic language with automatic garbage collection and is built on the principal of least surprise. While there are several implementations of the language, these implementations have not provided the best performance, so Rubinius is working on a high performance implementation, where more of the implementation is in Ruby itself. The genesis of Ruby was with Lisp and Smalltalk, although the actual language went in a very different direction than these two languages.
Evan admitted that Ruby does not have great support for concurrency. Ruby will work with green threads, that is cooperative multiprocessing that can exploit a single core. The problem of the "interpreter lock" means that a native threads support is not an immediate goal.
After Evan, David Pollak, author of "Beginning Scala" and Benevolent Dictator for Life of the Lift Web Framework spoke on Scala. Scala is a hybrid object oriented/functional language with static typing and garbage collection that runs on the Java Virtual Machine. In some ways it is like Java with less words, the type inference system eliminates the need to explicitly specify data types most of the time. The goal of Scala is to achieve the speed (and safety) of Java with the conciseness of Ruby.
Scala has no specific built in support for concurrency, however the Actors paradigm has been successfully implemented on top of Scala. I have written about both Scala and Actors with Scala previously, so I will say no more here. It is worth noting that both Clojure and Scala get full native threads support and many other benefits from running on the Java Virtual Machine.
Finally Robert Griesemer spoke about the new Go language. Robert is a member of the team developing Go at Google. Go is a statically typed language with automatic garbage collection that compiles down to native hardware. It is a system programming language with control over memory layout of the data. Robert listed the problems with current system programming languages. They are are verbose and repetitious, the data type system gets in the way, build time is slow, particularly compared to dynamic languages, and managing dependencies between modules is difficult. Go aims to be a simple and powerful language with fast tools.
Go does not have inheritance or type hierarchies and is not object oriented, rather it aims to be more flexible. I was somewhat disturbed by this. Although type hierarchies can be misused, they are useful for helping to organizing large projects. On the other hand, for concurrency, Go offers lightweight processes that communicate via channels, which is a welcome move away from the threads paradigm with all its problems.
The Go language is not quite complete yet. The language designers are still working on providing support for a number of features including generics, operators and exceptions. Robert told us that he expects the language to be complete and mature in 6 months to a year from now. Programming language design is not easy and needs to proceed at its own pace. It is worth remembering that the C++ language spend about 15 years in the state of being almost but not quite finished. We will have to wait and see how long it takes Go to mature.
Overall there were two themes that emerge from the the new languages presented at the meeting. One is the desire to make programming simpler and more approachable. It is easier to start writing a program in a dynamic language. Both Scala and Go are statically typed languages with the goal of making the programming experience more like writing a program in a dynamic language. The other theme is the need for new programming languages to provide better support for concurrency. In particular language need to support something that is safer and more controlled than threads.
Subscribe to:
Posts (Atom)
