SQL is not a perfect solution as I told the audience at the SDForum Business Intelligence SIG September meeting, where I spoke about "Analytics: SQL or NoSQL". The presentation discusses the difference between SQL and structured data on the one hand versus the NoSQL movement and semi-structured data on the other hand. There is more to the presentation than I can fit in one blog post, so here is what I had to say about the SQL language itself. I will write more about the presentation at another time. You can download the presentation from the BI SIG web site.
Firstly the good. SQL has given us a model of a query language that seems so useful as to be essential. Every system that provides persistence has developed a query language. Here are a smattering of examples. The Hibernate object persistence system has Hibernate Query Language (HQL) which has been developed into the Java Persistence Query language (JPQL). Other Java based object oriented persistence systems either use JPQL or their own variant. Hive is a query interface built on top of the Hadoop Map-Reduce engine. Hive was initially developed by Facebook as a simplified way of accessing their Map-Reduce infrastructure when they discovered that many of the people who need to write queries did not have the programming skills to handle a raw Map-Reduce environment. XQuery is a language for querying a set of XML documents. It has been adopted into the SQL language and is also used with stand alone XML systems. If data is important enough to persist, there is almost always a requirement to provide a simple and easy to use reporting system on that data. A query language handles the simple reporting requirements easily.
On the other hand, SQL has many problems. Here is my thoughts on the most important ones. The first problem is that SQL is not a programming language, it is a data access language. SQL is not designed for writing complete programs, it is intended to fetch data from the database and then anything more than a simply formatted report is done in another programming language. This concept of a data access language for accessing a database goes back to the original concept of a database as promulgated by the CODASYL committee in the late 1960's.
While most implementations of SQL add extra features to make it a complete programming language, they do not solve the problem because SQL is a language unlike any of the other other programming language we have. Firstly, SQL is a relational language. Every statement in SQL starts with a table and results in a table. (Table means a table like in a document, a fixed number of columns and as many rows as are required to express the data.) This is a larger chunk of data than programmers are used to handling. The procedural languages that interface to SQL expects to deal with data at most a row at a time. Also, the rigid table of SQL does not fit well into the more flexible data structures of procedural languages.
Moreover SQL is a declarative language where you specify the desired results and the database system works out the best way to produce them. Our other programming languages are procedural where you describe to the system how it should produce the desired result. Programming SQL requires a different mindset from programming in procedural languages. Many programmers, most of whom just dabble in SQL as a sideline, have difficulty making the leap and are frustrated by SQL because it is just not like the programming languages that they are used to. The combination of a relational language and a declarative language creates a costly mismatch between SQL and our other programming systems.
Finally, SQL becomes excessively wordy, repetitive and opaque as the queries becomes more complicated. Sub-Queries start to abound and the need for correlated sub-queries, outer joins and pivoting data for presentation cause queries to explode in length and complexity. Analytics is the province of complicated queries so this is a particular problem for data analysts. In the past I have suggested that persistence is a ripe area for a new programming language, however although there are many new languages being proposed none of them are concerned with persistence or analytics. The nearest thing to an analytics programming language is R which is powerful but neither new nor easy to use.
Showing posts with label programming languages. Show all posts
Showing posts with label programming languages. Show all posts
Wednesday, October 13, 2010
Saturday, March 20, 2010
Emerging Languages Face Off
New programming languages are popping up all over the place. In March the SDForum Emerging Tech SIG held an "Emerging Languages Face Off" to try and make sense out of what is going on. The new languages represented at the meeting were Clojure, Scala and Go, with Ruby as a more established control language. The panel was moderated by Steve Mezak, author of "Software without Borders" and CEO of Accelerance, Inc.
The night kicked off with Amit Rathore, Chief Software Architect at Runa, Inc. speaking for Clojure (pronounced closure). He told us that Clojure is a Lisp that runs on the Java Virtual Machine. Lisps are dynamic, functional languages with automatic garbage collection that have been around since the early 60's. Although Amit told us that Clojure programs contain less parenthesis than Java, the examples he showed us did not seem to bear this out. Clojure does try to control the amount of brackets by using both round and square ones. Apart from list, Clojure provides support for both common data structures like Map and lazy sequences.
More importantly, Amit introduced what turned out to be a major theme of the evening, support for concurrency. Clojure has a surprisingly sophisticated (read complicated) support for concurrency. The basic idea is that reads are versioned to be lock free while writes are managed to ensure that they overlap properly. Existing data is immutable, updates are made by writing the new data to new locations. Access to shared memory is delimited by transactions that correspond to the program block structure (good). There are 4 ways of referencing data in a transaction that allow the different use cases each to be handled efficiently. If a transaction fails, it is automatically retried until it succeeds. Their implementation of concurrency goes under the banner of Software Transactional Memory.
While I will give Clojure two and a half cheers for trying, I am not a great fan of Lisp like languages. Amit touched on one of my bugaboos, the ability to change the meaning of the language by writing code. In my mind, this makes Lisp a low level language as any program requires close reading to discover what it might do. Also, my only practical experience of Lisp is Emacs configuration, a scary mess of global variables and functions.
Next up, Evan Phoenix, lead developer of Rubinius spoke about Ruby. Ruby is a dynamic language with automatic garbage collection and is built on the principal of least surprise. While there are several implementations of the language, these implementations have not provided the best performance, so Rubinius is working on a high performance implementation, where more of the implementation is in Ruby itself. The genesis of Ruby was with Lisp and Smalltalk, although the actual language went in a very different direction than these two languages.
Evan admitted that Ruby does not have great support for concurrency. Ruby will work with green threads, that is cooperative multiprocessing that can exploit a single core. The problem of the "interpreter lock" means that a native threads support is not an immediate goal.
After Evan, David Pollak, author of "Beginning Scala" and Benevolent Dictator for Life of the Lift Web Framework spoke on Scala. Scala is a hybrid object oriented/functional language with static typing and garbage collection that runs on the Java Virtual Machine. In some ways it is like Java with less words, the type inference system eliminates the need to explicitly specify data types most of the time. The goal of Scala is to achieve the speed (and safety) of Java with the conciseness of Ruby.
Scala has no specific built in support for concurrency, however the Actors paradigm has been successfully implemented on top of Scala. I have written about both Scala and Actors with Scala previously, so I will say no more here. It is worth noting that both Clojure and Scala get full native threads support and many other benefits from running on the Java Virtual Machine.
Finally Robert Griesemer spoke about the new Go language. Robert is a member of the team developing Go at Google. Go is a statically typed language with automatic garbage collection that compiles down to native hardware. It is a system programming language with control over memory layout of the data. Robert listed the problems with current system programming languages. They are are verbose and repetitious, the data type system gets in the way, build time is slow, particularly compared to dynamic languages, and managing dependencies between modules is difficult. Go aims to be a simple and powerful language with fast tools.
Go does not have inheritance or type hierarchies and is not object oriented, rather it aims to be more flexible. I was somewhat disturbed by this. Although type hierarchies can be misused, they are useful for helping to organizing large projects. On the other hand, for concurrency, Go offers lightweight processes that communicate via channels, which is a welcome move away from the threads paradigm with all its problems.
The Go language is not quite complete yet. The language designers are still working on providing support for a number of features including generics, operators and exceptions. Robert told us that he expects the language to be complete and mature in 6 months to a year from now. Programming language design is not easy and needs to proceed at its own pace. It is worth remembering that the C++ language spend about 15 years in the state of being almost but not quite finished. We will have to wait and see how long it takes Go to mature.
Overall there were two themes that emerge from the the new languages presented at the meeting. One is the desire to make programming simpler and more approachable. It is easier to start writing a program in a dynamic language. Both Scala and Go are statically typed languages with the goal of making the programming experience more like writing a program in a dynamic language. The other theme is the need for new programming languages to provide better support for concurrency. In particular language need to support something that is safer and more controlled than threads.
The night kicked off with Amit Rathore, Chief Software Architect at Runa, Inc. speaking for Clojure (pronounced closure). He told us that Clojure is a Lisp that runs on the Java Virtual Machine. Lisps are dynamic, functional languages with automatic garbage collection that have been around since the early 60's. Although Amit told us that Clojure programs contain less parenthesis than Java, the examples he showed us did not seem to bear this out. Clojure does try to control the amount of brackets by using both round and square ones. Apart from list, Clojure provides support for both common data structures like Map and lazy sequences.
More importantly, Amit introduced what turned out to be a major theme of the evening, support for concurrency. Clojure has a surprisingly sophisticated (read complicated) support for concurrency. The basic idea is that reads are versioned to be lock free while writes are managed to ensure that they overlap properly. Existing data is immutable, updates are made by writing the new data to new locations. Access to shared memory is delimited by transactions that correspond to the program block structure (good). There are 4 ways of referencing data in a transaction that allow the different use cases each to be handled efficiently. If a transaction fails, it is automatically retried until it succeeds. Their implementation of concurrency goes under the banner of Software Transactional Memory.
While I will give Clojure two and a half cheers for trying, I am not a great fan of Lisp like languages. Amit touched on one of my bugaboos, the ability to change the meaning of the language by writing code. In my mind, this makes Lisp a low level language as any program requires close reading to discover what it might do. Also, my only practical experience of Lisp is Emacs configuration, a scary mess of global variables and functions.
Next up, Evan Phoenix, lead developer of Rubinius spoke about Ruby. Ruby is a dynamic language with automatic garbage collection and is built on the principal of least surprise. While there are several implementations of the language, these implementations have not provided the best performance, so Rubinius is working on a high performance implementation, where more of the implementation is in Ruby itself. The genesis of Ruby was with Lisp and Smalltalk, although the actual language went in a very different direction than these two languages.
Evan admitted that Ruby does not have great support for concurrency. Ruby will work with green threads, that is cooperative multiprocessing that can exploit a single core. The problem of the "interpreter lock" means that a native threads support is not an immediate goal.
After Evan, David Pollak, author of "Beginning Scala" and Benevolent Dictator for Life of the Lift Web Framework spoke on Scala. Scala is a hybrid object oriented/functional language with static typing and garbage collection that runs on the Java Virtual Machine. In some ways it is like Java with less words, the type inference system eliminates the need to explicitly specify data types most of the time. The goal of Scala is to achieve the speed (and safety) of Java with the conciseness of Ruby.
Scala has no specific built in support for concurrency, however the Actors paradigm has been successfully implemented on top of Scala. I have written about both Scala and Actors with Scala previously, so I will say no more here. It is worth noting that both Clojure and Scala get full native threads support and many other benefits from running on the Java Virtual Machine.
Finally Robert Griesemer spoke about the new Go language. Robert is a member of the team developing Go at Google. Go is a statically typed language with automatic garbage collection that compiles down to native hardware. It is a system programming language with control over memory layout of the data. Robert listed the problems with current system programming languages. They are are verbose and repetitious, the data type system gets in the way, build time is slow, particularly compared to dynamic languages, and managing dependencies between modules is difficult. Go aims to be a simple and powerful language with fast tools.
Go does not have inheritance or type hierarchies and is not object oriented, rather it aims to be more flexible. I was somewhat disturbed by this. Although type hierarchies can be misused, they are useful for helping to organizing large projects. On the other hand, for concurrency, Go offers lightweight processes that communicate via channels, which is a welcome move away from the threads paradigm with all its problems.
The Go language is not quite complete yet. The language designers are still working on providing support for a number of features including generics, operators and exceptions. Robert told us that he expects the language to be complete and mature in 6 months to a year from now. Programming language design is not easy and needs to proceed at its own pace. It is worth remembering that the C++ language spend about 15 years in the state of being almost but not quite finished. We will have to wait and see how long it takes Go to mature.
Overall there were two themes that emerge from the the new languages presented at the meeting. One is the desire to make programming simpler and more approachable. It is easier to start writing a program in a dynamic language. Both Scala and Go are statically typed languages with the goal of making the programming experience more like writing a program in a dynamic language. The other theme is the need for new programming languages to provide better support for concurrency. In particular language need to support something that is safer and more controlled than threads.
Tuesday, June 30, 2009
Actors and Concurrent Computation
Carl Hewitt was in fine form when he spoke about the Actor model of concurrent computation at the SDForum SAM SIG meeting on "Actor Architectures for Concurrent Computation". The meeting was a panel with three speakers. Carl Hewitt, Emeritus Professor in Electrical Engineering and Computer Science at the Massachusetts Institute of Technology. Hewitt and his students invented Actors in the 1970's. Frank Sommers of Artima Software, is an active writer in the area of information technology and computing and is currently writing a book on Actors in the Scala programming language. Robey Pointer is a software engineer at Twitter, and is an enthusiastic user of Scala and Actors at Twitter. Bill Venners is president of Artima Software and an author of a book on Scala moderated the proceedings.
Hewitt had retired when the advent of multi-core processors and distributed parallelism renewed interest in the Actor model. He has come out of retirement and is currently visiting Stanford. Hewitt described the genesis of the Actor methodology in the Alan Kay's Smalltalk 72, where every object was autonomous and you acted on an object by sending it messages. Later versions of Smalltalk moves in a different direction. The most important aspect of the Actor model is that it decouples the sender from the communications. In practice this allows the Scala implementation to scale to millions of Actors engaged in concurrent communication (more on this later).
Hewitt spoke with great flourish on a number of other topics, including his determination that the Actor model should be properly represented on Wikipedia and spooks and the internet archive. He spent some time with the unbounded non-determinism in the Actor model versus other concurrency formalisms that only support bounded non-determinism. An audience member challenged him to explain this better and citing Map-Reduce. Amusingly, Hewitt answered by describing the parallelism in Map-Reduce as like Stalin. Stalin has three deputies and each of those deputies has three deputies. Stalin tells his deputies what to do, and those deputies tell their deputies what to do and so on. Thus the business of the state can proceed in parallel. Map-Reduce is a sequential algorithm which is speeded up by parallelism. There is no non-determinism. This is parallelism as opposed to concurrency.
Next Frank Sommers spoke on how Actors are used in Scala. The good news is that Actors are implemented in Scala and Hewitt much preferred the Scala implementation over the Erlang implementation of Actors. The bad news is that there are a number of issues with the Scala implementation. For example, a Scala program cannot exit from a "receive" statement. Another issue is that messages are supposed to be immutable, however the current implementation may not ensure this. These and other issues are being worked on, and the next version of Actors in Scala will be much better.
Finally, Robey Pointer talked about how he is using Scala. He implements message queuing systems that deals with large numbers of long lived connections where each connection is mostly idle but has sporadic bursts of activity. Robey has implemented this system in many different ways. For example, a straight thread implementation and a lot of tuning got up to 5000 thread based connections working at one time, however this fell well short of his goal of supporting millions of connections. A thread pool implementation with a few hundred threads worked better but the code became unwieldy and more complicated than it should have been. Now he has an Actor based implementation in Scala that does scale to the millions of connections and yet the code remains straightforward and small.
He also showed us how Actors can be mixed-in with thread based synchronization to solve problems for which even Actors are too heavyweight. I am in two minds about this. On the one hand, there are legitimate uses for this low level synchronization (as discussed in my PhD thesis). On the other hand, thread based concurrency is awful as I keep promising to explain in another post. Also to do it safely, you need to understand is great detail how Actors are implemented in Scala, and one reason for adopting a high level construct like Actors is that it should hide gory implementation details.
After the meeting I spoke with Carl Hewitt. We agreed that sending a message needs to have a low overhead. It should have a similar costs to calling a procedure. Computers have specialized instructions to support procedure calls and they need specialized instructions to support message passing. We did this for the Tranputer, although that was before its time, and it is eminently possible for Actors. All we need is for a high level concurrency formalism like Actors to get enough traction that the chip manufacturers become interested in supporting it.
Hewitt had retired when the advent of multi-core processors and distributed parallelism renewed interest in the Actor model. He has come out of retirement and is currently visiting Stanford. Hewitt described the genesis of the Actor methodology in the Alan Kay's Smalltalk 72, where every object was autonomous and you acted on an object by sending it messages. Later versions of Smalltalk moves in a different direction. The most important aspect of the Actor model is that it decouples the sender from the communications. In practice this allows the Scala implementation to scale to millions of Actors engaged in concurrent communication (more on this later).
Hewitt spoke with great flourish on a number of other topics, including his determination that the Actor model should be properly represented on Wikipedia and spooks and the internet archive. He spent some time with the unbounded non-determinism in the Actor model versus other concurrency formalisms that only support bounded non-determinism. An audience member challenged him to explain this better and citing Map-Reduce. Amusingly, Hewitt answered by describing the parallelism in Map-Reduce as like Stalin. Stalin has three deputies and each of those deputies has three deputies. Stalin tells his deputies what to do, and those deputies tell their deputies what to do and so on. Thus the business of the state can proceed in parallel. Map-Reduce is a sequential algorithm which is speeded up by parallelism. There is no non-determinism. This is parallelism as opposed to concurrency.
Next Frank Sommers spoke on how Actors are used in Scala. The good news is that Actors are implemented in Scala and Hewitt much preferred the Scala implementation over the Erlang implementation of Actors. The bad news is that there are a number of issues with the Scala implementation. For example, a Scala program cannot exit from a "receive" statement. Another issue is that messages are supposed to be immutable, however the current implementation may not ensure this. These and other issues are being worked on, and the next version of Actors in Scala will be much better.
Finally, Robey Pointer talked about how he is using Scala. He implements message queuing systems that deals with large numbers of long lived connections where each connection is mostly idle but has sporadic bursts of activity. Robey has implemented this system in many different ways. For example, a straight thread implementation and a lot of tuning got up to 5000 thread based connections working at one time, however this fell well short of his goal of supporting millions of connections. A thread pool implementation with a few hundred threads worked better but the code became unwieldy and more complicated than it should have been. Now he has an Actor based implementation in Scala that does scale to the millions of connections and yet the code remains straightforward and small.
He also showed us how Actors can be mixed-in with thread based synchronization to solve problems for which even Actors are too heavyweight. I am in two minds about this. On the one hand, there are legitimate uses for this low level synchronization (as discussed in my PhD thesis). On the other hand, thread based concurrency is awful as I keep promising to explain in another post. Also to do it safely, you need to understand is great detail how Actors are implemented in Scala, and one reason for adopting a high level construct like Actors is that it should hide gory implementation details.
After the meeting I spoke with Carl Hewitt. We agreed that sending a message needs to have a low overhead. It should have a similar costs to calling a procedure. Computers have specialized instructions to support procedure calls and they need specialized instructions to support message passing. We did this for the Tranputer, although that was before its time, and it is eminently possible for Actors. All we need is for a high level concurrency formalism like Actors to get enough traction that the chip manufacturers become interested in supporting it.
Wednesday, April 08, 2009
Ruby versus Scala
An interesting spat has recently emerged in the long running story of Programming Language wars. The Twitter team, who had long been exclusively a Ruby on Rails house, came out with the "shock" revelation that they were converting part of their back end code to use the Scala programming language. Several Ruby zealots immediately jumped in saying that the Twitter crew obviously did not know what they were doing because they had decided to turn their backs on Ruby.
I must confess to being amused by the frothing at the mouth from the Ruby defenders, but rather than laughing, lets take a calm look at the arguments. The Twitter developers are still using Ruby on Rails for its intended purpose of running a web site. However they are also developing back-end server software and have chosen the Scala programming language for that effort.
The Twitter crew offer a number of reasons for choosing a strongly typed language. Firstly, dynamic languages are not very good for implementing the kind of long running processes that you find in a server. I have experience with writing servers in C, C++ and Java. In all these languages there are problems with memory leaks that cause the memory footprint to grow over the days, weeks or months that the server is running. Getting rid of memory leaks is tedious and painful, but absolutely necessary. Even the smallest memory leak will be a problem with heavy usage and if you have a memory leak, the only cure is stopping and restarting the server. Note that garbage collection does not do away with memory leaks, it just changes the nature of the problem. Dynamic languages are designed for rapid implementation and hide the boring details. One detail that is missing is control over memory usage and memory usage left on its own tends to leak.
Another issue is concurrency. Server software needs to exploit concurrency, particularly now in the era of multi-core hardware. Dynamic languages have problems with concurrency. There are a bunch of issues, too many to discuss here. Sufficient to say that in the past Guido van Rossum has prominently argued against putting threads into Python, another dynamic language, and both Python and Ruby implementations suffer from poor thread performance.
A third issue is type safety. As the Twitter crew say, they found themselves building their own type manager into their server code. In a statically typed language, the type management is done at compile time, making the code more efficient and automatically eliminating the potential for a large class of bugs.
Related to this, many people commented on the revelation that the Twitter Ruby server code was full of calls to the Ruby kind_of method. It is normally considered bad form to have to use kind_of or its equivalent in other languages like the Java instanceof operator. After a few moments thought I understood what the kind_of code is for. If you look at any real server like a database server's code, it is full of assert statements. The idea is that if you are going to fail, you should fail as fast as you can and let the error management and recovery system get you out of trouble. Failing fast reduces the likelihood that the error will propagate and cause real damage like corrupting persistent data. Also with a fast fail it is easier to figure out why the error occurred. In a language with dynamic typing, checking parameters with a kind_of method is the first type of assert to put in any server code.
So the Twitter developers have opted to use Ruby on Rails for their web server and Scala for their server code. In the old days we would have said "horses for courses" and everyone would have nodded their heads in understanding. Nowadays , nobody goes racing, so nobody knows what the phrase means. Can anyone suggest a more up to date expression?
I must confess to being amused by the frothing at the mouth from the Ruby defenders, but rather than laughing, lets take a calm look at the arguments. The Twitter developers are still using Ruby on Rails for its intended purpose of running a web site. However they are also developing back-end server software and have chosen the Scala programming language for that effort.
The Twitter crew offer a number of reasons for choosing a strongly typed language. Firstly, dynamic languages are not very good for implementing the kind of long running processes that you find in a server. I have experience with writing servers in C, C++ and Java. In all these languages there are problems with memory leaks that cause the memory footprint to grow over the days, weeks or months that the server is running. Getting rid of memory leaks is tedious and painful, but absolutely necessary. Even the smallest memory leak will be a problem with heavy usage and if you have a memory leak, the only cure is stopping and restarting the server. Note that garbage collection does not do away with memory leaks, it just changes the nature of the problem. Dynamic languages are designed for rapid implementation and hide the boring details. One detail that is missing is control over memory usage and memory usage left on its own tends to leak.
Another issue is concurrency. Server software needs to exploit concurrency, particularly now in the era of multi-core hardware. Dynamic languages have problems with concurrency. There are a bunch of issues, too many to discuss here. Sufficient to say that in the past Guido van Rossum has prominently argued against putting threads into Python, another dynamic language, and both Python and Ruby implementations suffer from poor thread performance.
A third issue is type safety. As the Twitter crew say, they found themselves building their own type manager into their server code. In a statically typed language, the type management is done at compile time, making the code more efficient and automatically eliminating the potential for a large class of bugs.
Related to this, many people commented on the revelation that the Twitter Ruby server code was full of calls to the Ruby kind_of method. It is normally considered bad form to have to use kind_of or its equivalent in other languages like the Java instanceof operator. After a few moments thought I understood what the kind_of code is for. If you look at any real server like a database server's code, it is full of assert statements. The idea is that if you are going to fail, you should fail as fast as you can and let the error management and recovery system get you out of trouble. Failing fast reduces the likelihood that the error will propagate and cause real damage like corrupting persistent data. Also with a fast fail it is easier to figure out why the error occurred. In a language with dynamic typing, checking parameters with a kind_of method is the first type of assert to put in any server code.
So the Twitter developers have opted to use Ruby on Rails for their web server and Scala for their server code. In the old days we would have said "horses for courses" and everyone would have nodded their heads in understanding. Nowadays , nobody goes racing, so nobody knows what the phrase means. Can anyone suggest a more up to date expression?
Sunday, February 15, 2009
Do We Need a New Programming Language?
There is a plethora of new and emerging programming languages. Recently I have written about two: Scala and Groovy.
In many ways this era feels like the 1980's when we had a similar profusion of new programming languages. Then Niklaus Wirth seemed to come out with another new language every other year. Apart from those, there were plenty of other new languages to argue about. At the time, the language wars got so intense that most people retreated behind the shelter of a simple acronym. Their response was "JAFPL" (Just Another F-word Programming Language), where the F-word could be "Friendly" if you just wanted to dodge an excruciating time-wasting controversy.
In practice, Wirth's languages from the 80's along with many others have not survived. The reason is quite simple, these languages did not do anything more than other stronger languages of the time. Now we have the same situation. The new languages address the same problem space as better established languages. Why should we move to a new dynamic language like Groovy when we already have Perl, Python and Ruby? Likewise, is there anything of real value that Scala does that cannot already be done by Java?
In my opinion, there are two important issues in programming that current programming languages do not address well. They are:
To breakout, the next big programming language needs to do something new, different and more than existing programming languages do. There are plenty of new problems to solve, concurrency and persistence seems like the most likely problem areas to address.
In many ways this era feels like the 1980's when we had a similar profusion of new programming languages. Then Niklaus Wirth seemed to come out with another new language every other year. Apart from those, there were plenty of other new languages to argue about. At the time, the language wars got so intense that most people retreated behind the shelter of a simple acronym. Their response was "JAFPL" (Just Another F-word Programming Language), where the F-word could be "Friendly" if you just wanted to dodge an excruciating time-wasting controversy.
In practice, Wirth's languages from the 80's along with many others have not survived. The reason is quite simple, these languages did not do anything more than other stronger languages of the time. Now we have the same situation. The new languages address the same problem space as better established languages. Why should we move to a new dynamic language like Groovy when we already have Perl, Python and Ruby? Likewise, is there anything of real value that Scala does that cannot already be done by Java?
In my opinion, there are two important issues in programming that current programming languages do not address well. They are:
- Concurrency
- Persistence
To breakout, the next big programming language needs to do something new, different and more than existing programming languages do. There are plenty of new problems to solve, concurrency and persistence seems like the most likely problem areas to address.
Sunday, February 01, 2009
Groovy and the MetaObject Protocol
Bill Grosso spoke to the January meeting of the SDForum Java SIG on the Groovy programming language. An important part of the discussion was the Metaobject Protocol. I will get to that after explaining what Groovy is. Also at the meeting, Chris Richardson spoke on GORM (Grails Object Relational Mapping) an object relational mapping layer for Groovy programs. I will discuss this at another time.
A couple of months ago I posted on the Scala programming language. Like Scala, Groovy has a Java like syntax and is compiled to the Java virtual machine. That means that it meets the Java criteria of compile once, run anywhere. Also it makes the full set of Java libraries available to the programmer. Unlike Scala, Groovy is a Dynamic language. While there is some discussion as to exactly what a Dynamic programming language is, a core principal is that in a Dynamic language data type is a property of a data item, whereas in a Strongly Typed language like Scala, the data type is a property of the variable that holds the data item.
In general, it is easier to start writing a program in a dynamic language as you do not need to make so many decisions up front, the program gets written more quickly, you can test out ideas and bits of the program interactively and the program tends to be smaller. On the other hand, large programming projects, particularly with many programmers, maintainability and refactoring suffers when a dynamic language is used. I have more to say about dynamic languages at another time.
In his talk, Bill argued that another advantage of a Dynamic language is that it allows for the Metaobject Protocol. Like anything else with meta in its name, the Metaobject Protocol concept is a little hard to pin down. I think of it as the mechanisms in a programming language that allow you to build frameworks, generalized algorithms and such stuff. Bill's argument is that the ability to build these things within the language is necessary and a good thing, and that the mutability of a dynamic language's type system is required to do it. After all the name Metaobject Protocol comes from Lisp, the most mutable of programming languages (apart from assembler).
I disagree with the last part of the argument. For example, the old joke goes Q: what happens when you put more than one java programmer in a room? A: you get a framework! Java is a strongly typed language and the object of more frameworks than all other programming languages put together. In fact I am always amazed at how Java, which is a relatively small language as well as being strongly typed can be used to build such extensible systems.
There is more to my objection than that. Sometimes boundaries on what you can do are a good thing, and a well structured programming language creates boundaries. During the talk, Bill gave as an example of the Metaobject Protocol, changing the inheritance rules for an object. Inheritance, particularly multiple inheritance, is a slippery and potentially ambiguous problem at the best of times and the ability to change inheritance on the fly is frankly scary. My first reaction to any project where any kind of inheritance rules are even an issue would mean that the class structure is far to complicated. Programmers should strive to create deep and complicated data structures rather than deep and complicated class structures.
To summarize, I believe that strongly typed languages are just as capable as dynamic languages for building the kinds of tools and frameworks as the Metaobject Protocol. Moreover, strong, clear types and rules prevent us from getting lost in a semantic swamp that can arise when our language system allows us to change things on the fly. Apart from that, Groovy seems like an nice enough language, and it is probably as good as the many other dynamic languages that are around now. Bill did mention that performance is noticeably slower than Java, as might be expected for a Dynamic language running on a platform optimized for the other type of language.
Friday, December 26, 2008
Functional Programming and Concurrency
With the arrival of more and more cores on the typical processor chip these days, concurrency is becoming an screamingly important topic in programming. Unfortunately, many people seem to think that the solution to concurrent programming is a Functional Programming language. For example, in the January issue of Dr. Dobb's Journal there is an article on "It's Time to Get Good at Functional Programming" which has received a lot of comment around the web.
I have been doing concurrent programming for a long time, but I have never understood why there there is any connection between functional programming and concurrency. Rather than waste a lot of time with specious argument, let me just give a counterexample. Occam was the first programming language designed specifically for programming concurrent computer systems. It is based on the Concurrent Sequential Processes (CSP) notation of Tony Hoare. While Occam has functions, it is by no means a language for functional programming, having eschewed recursion in the first few versions of the language.
I have been doing concurrent programming for a long time, but I have never understood why there there is any connection between functional programming and concurrency. Rather than waste a lot of time with specious argument, let me just give a counterexample. Occam was the first programming language designed specifically for programming concurrent computer systems. It is based on the Concurrent Sequential Processes (CSP) notation of Tony Hoare. While Occam has functions, it is by no means a language for functional programming, having eschewed recursion in the first few versions of the language.
Saturday, November 01, 2008
The Scala Programming Language
These days feel like the 1980's as far as programming languages are concerned with new programming languages springing up all over the place. Then, the prolific Nicklaus Wirth invented a new programming language every other year. Now, the center of language design in Switzerland has moved to Lausanne where Martin Odersky at EPFL has conceived Scala. Bill Venners of Artima introduced the Scala Programming Language at the October meeting of the SDForum Java SIG.
Scala is a functional language, in the sense that is every "statement" produces a value. Also, Scala is a statically typed language although programs look like a dynamic language. The trick is that variables are declared by a 'var' declaration, and the type variable is the type of the initial value assigned to the variable. Contrast this with a dynamic language where the data type is associated with the data value and every operation on data has to look at the data types of the operands to decide what to do.
Getting the data type from the value assigned reduces the need to over specify type as is typical of statically typed languages like Java. Bill recalled the discussion of Duh typing that a group of us had after the last time he spoke to a SFDForum SIG. The other thing that Scala makes easy is declaring invariant variables. They are like 'var' variables except they are introduced by the keyword 'val'. Contrast this with Java where you put final before the declaration or C++ where you put const before the declaration. Thus a constant in Java is declared something like this:
Finally, Scala, like many other languages these days, compiles to the Java Virtual Machine. That way, it is broadly portable, and developers have access to the vast Java libraries.
Scala is a functional language, in the sense that is every "statement" produces a value. Also, Scala is a statically typed language although programs look like a dynamic language. The trick is that variables are declared by a 'var' declaration, and the type variable is the type of the initial value assigned to the variable. Contrast this with a dynamic language where the data type is associated with the data value and every operation on data has to look at the data types of the operands to decide what to do.
Getting the data type from the value assigned reduces the need to over specify type as is typical of statically typed languages like Java. Bill recalled the discussion of Duh typing that a group of us had after the last time he spoke to a SFDForum SIG. The other thing that Scala makes easy is declaring invariant variables. They are like 'var' variables except they are introduced by the keyword 'val'. Contrast this with Java where you put final before the declaration or C++ where you put const before the declaration. Thus a constant in Java is declared something like this:
final static String HELLO_WORLD = "Hello World";while in Scala the declaration looks like this:
val HELLO_WORLD = "Hello World"This leads to a more declarative style of programming, which is a good thing. Bill reported that while in Java 95% of declarations are variables and the other 5% are constants, in Scala, 95% of declarations are constant and only 5% are variables. I have used a similar style of programming in C++ when using APIs that make heavy use of const, so you have to declare variables that you are going to pass to the API as const. The only time that this is an problem is where you have to create const objects that can throw in their constructor. Then you can end up with heavily indented try blocks as you create each const object safely so that you can pass it to the API.
Finally, Scala, like many other languages these days, compiles to the Java Virtual Machine. That way, it is broadly portable, and developers have access to the vast Java libraries.
Saturday, October 25, 2008
The Google Database System
Google, Yahoo and others are taking the traditional database system and breaking it into pieces. Google has their own set of propriety database system components. Yahoo is working with the Hadoop Open Source project to make their system available to everyone. I came to this conclusion while doing research for my talk on "Models and Patterns for Concurrency" for the SDForum SAM SIG.
In this post I will talk about what is happening, why it is happening and at the end try to draw some conclusions about future directions for database systems. Note, I am using Google as an example here because their system is described in a set of widely accessible academic papers. Yahoo and many other large scale web sites have adopted a similar approach through use of Hadoop and other Open Source projects.
A traditional database system is a server. It takes care of persistent data storage, metadata management for the stored data, transaction management and querying the data, which includes aggregation. Google has developed its own set of applications which support these same functions, except instead of wrapping them into a single entity as a database server, they have been developed as a set of application programs that build on one another.
The are several reasons for Google developing their own database system. Firstly, they are dealing with managing and processing huge amounts of data. Conventional database systems struggle when the data gets really large. In particular the transaction model that underlies database operation starts to break down. This topic is worth a separate post of its own.
Secondly they their computing system is a distributed system built from thousands of commodity computer systems. Conventional database systems are not designed or tuned to run on this type of hardware. One issue is that at this scale the hardware cannot be assumed to be reliable and the database system has to be designed to work around the unreliable hardware. A final issue is that the cost of software licenses for running a conventional database system on the Google hardware would be prohibitive.
The Google internal application look like this. A the bottom is Chubby, a lock service that also provides reliable storage for small amounts of data. The Google File System provides file data storage in a distributed system of thousands of computers with local disks. It uses Chubby to ensure that there is a single master in the face of system failures. Bigtable is a system for storing and accessing structured data. While it is not exactly relational, it is comparable to storing data in a single large relational database table. Bigtable stores its data in the Google File System and uses Chubby for several purposes including metadata storage and to ensure atomicity of certain operations.
Finally Map Reduce is a generalized aggregation engine (and here I mean aggregation in the technical database sense). It uses the Google File System. Map Reduce is surprisingly closely related to database aggregation as found in the SQL language although it is not usually described in that way. I will discuss this in another post. In the mean time, it is interesting to note that Map Reduce has been subject to a rather intemperate attack by database luminaries David DeWitt and Michael Stonebreaker.
In total, these four applications: Chubby, Google File System, Bigtable and Map Reduce provide the capabilities of a database system. In practice there are some differences. The user writes program in a language like C++ that integrated the capabilities of these components as they need them. They do not need to use all the components. For example, Google can calculate the page rank for each page on the web as a series of 6 Map Reduce steps none of which necessarily uses Bigtable.
The concept of a Database System was invented in the late 60's by the CODASYL committee, shortly after their achievement of inventing the COBOL programming language. The Relational model and Transactions came later, however the concept of a server system that owns and manages data and much of the terminology originated with CODASYL. Since then, the world has changed.
Nowadays databases are often hidden behind frameworks such as Hibernate or Ruby on Rails that try to paper over the impedance mismatch between the database model on one side and an object oriented world looking for persistence on the other. These are mostly low end systems. At the other end of scale are the huge data management problems of Google, Yahoo and other web sites. New companies with new visions of database systems to meet these challenges are emerging. It is an exciting time.
In this post I will talk about what is happening, why it is happening and at the end try to draw some conclusions about future directions for database systems. Note, I am using Google as an example here because their system is described in a set of widely accessible academic papers. Yahoo and many other large scale web sites have adopted a similar approach through use of Hadoop and other Open Source projects.
A traditional database system is a server. It takes care of persistent data storage, metadata management for the stored data, transaction management and querying the data, which includes aggregation. Google has developed its own set of applications which support these same functions, except instead of wrapping them into a single entity as a database server, they have been developed as a set of application programs that build on one another.
The are several reasons for Google developing their own database system. Firstly, they are dealing with managing and processing huge amounts of data. Conventional database systems struggle when the data gets really large. In particular the transaction model that underlies database operation starts to break down. This topic is worth a separate post of its own.
Secondly they their computing system is a distributed system built from thousands of commodity computer systems. Conventional database systems are not designed or tuned to run on this type of hardware. One issue is that at this scale the hardware cannot be assumed to be reliable and the database system has to be designed to work around the unreliable hardware. A final issue is that the cost of software licenses for running a conventional database system on the Google hardware would be prohibitive.
The Google internal application look like this. A the bottom is Chubby, a lock service that also provides reliable storage for small amounts of data. The Google File System provides file data storage in a distributed system of thousands of computers with local disks. It uses Chubby to ensure that there is a single master in the face of system failures. Bigtable is a system for storing and accessing structured data. While it is not exactly relational, it is comparable to storing data in a single large relational database table. Bigtable stores its data in the Google File System and uses Chubby for several purposes including metadata storage and to ensure atomicity of certain operations.
Finally Map Reduce is a generalized aggregation engine (and here I mean aggregation in the technical database sense). It uses the Google File System. Map Reduce is surprisingly closely related to database aggregation as found in the SQL language although it is not usually described in that way. I will discuss this in another post. In the mean time, it is interesting to note that Map Reduce has been subject to a rather intemperate attack by database luminaries David DeWitt and Michael Stonebreaker.
In total, these four applications: Chubby, Google File System, Bigtable and Map Reduce provide the capabilities of a database system. In practice there are some differences. The user writes program in a language like C++ that integrated the capabilities of these components as they need them. They do not need to use all the components. For example, Google can calculate the page rank for each page on the web as a series of 6 Map Reduce steps none of which necessarily uses Bigtable.
The concept of a Database System was invented in the late 60's by the CODASYL committee, shortly after their achievement of inventing the COBOL programming language. The Relational model and Transactions came later, however the concept of a server system that owns and manages data and much of the terminology originated with CODASYL. Since then, the world has changed.
Nowadays databases are often hidden behind frameworks such as Hibernate or Ruby on Rails that try to paper over the impedance mismatch between the database model on one side and an object oriented world looking for persistence on the other. These are mostly low end systems. At the other end of scale are the huge data management problems of Google, Yahoo and other web sites. New companies with new visions of database systems to meet these challenges are emerging. It is an exciting time.
Thursday, July 17, 2008
A Gentle Introduction to R
We were given a gentle introduction to the R statistical programming language and its application in Business Intelligence at the July meeting of the SDForum Business Intelligence SIG. The speakers were Jim Porzac ( Senior Director of Analytics at Responsys) and Michael Driscoll (Principal at Dataspora). Jim has posted the presentation here.
R is an Open Source project that uses the GNU license. It has a growing user base with a strong support community and a user group (called UseR Group - try Googling that). There are now almost 1500 packages for the languages that supports various statistical techniques and specialized application areas. Packages include: Bayesian, Econometrics, Genetics, Machine Learning, Natural Language Processing, Pharmacokinetics, Psycometrics, which gives some idea of the range of subjects and techniques that R covers.
Jim did most of the talking, introducing the language and showing us some examples of its use. One example is his data quality package that he uses on each new dataset that he receives for analysis at Responsys. Another example showed how reporting capabilities while a third showed sophisticated graphs and plots used for customer segmentation analysis. Michael showed us how he used R to do some interesting and very practical analyzes of Baseball statistics.
The audience probed R's strength and weakness. R has the connectivity to get data for analysis from databases and other sources. R also has excellent graphing and reporting capabilities. Currently R works by reading data into memory where it is manipulated, which limits the maximum size of data set that can be analyzed to the many Gigabyte range.
One person asked for a comparison with SAS. R has the advantages of being free with an enthusiastic user base to keeps it on the cutting edge. Also R is a more coherent language than SAS, which is a collection of libraries, each of which may be very good but they do not necessarily make a whole.
Jim and Michael are starting a Bay Area chapter of the UseR Group. If you are interested, contact Jim Porzac at Responsys.
R is an Open Source project that uses the GNU license. It has a growing user base with a strong support community and a user group (called UseR Group - try Googling that). There are now almost 1500 packages for the languages that supports various statistical techniques and specialized application areas. Packages include: Bayesian, Econometrics, Genetics, Machine Learning, Natural Language Processing, Pharmacokinetics, Psycometrics, which gives some idea of the range of subjects and techniques that R covers.
Jim did most of the talking, introducing the language and showing us some examples of its use. One example is his data quality package that he uses on each new dataset that he receives for analysis at Responsys. Another example showed how reporting capabilities while a third showed sophisticated graphs and plots used for customer segmentation analysis. Michael showed us how he used R to do some interesting and very practical analyzes of Baseball statistics.
The audience probed R's strength and weakness. R has the connectivity to get data for analysis from databases and other sources. R also has excellent graphing and reporting capabilities. Currently R works by reading data into memory where it is manipulated, which limits the maximum size of data set that can be analyzed to the many Gigabyte range.
One person asked for a comparison with SAS. R has the advantages of being free with an enthusiastic user base to keeps it on the cutting edge. Also R is a more coherent language than SAS, which is a collection of libraries, each of which may be very good but they do not necessarily make a whole.
Jim and Michael are starting a Bay Area chapter of the UseR Group. If you are interested, contact Jim Porzac at Responsys.
Thursday, June 12, 2008
Flex, ActionScript, MXML?
It has been over a week since James Ward and Chet Haase of Adobe gave a talk on Flex to the SDForum Java SIG, and I am still trying to get my mind around what it all means. Adobe has a bunch of technologies and products in the Rich Internet Application (RIA) area, but it is difficult to work out what they are, how they fit together and which one I should use for any particular application. Here is the story as I understand it.
Lets start with the programming language ActionScript. ActionScript is ECMAScript which is JavaScript. There are differences in implementation between ActionScript and other forms of JavaScript, however most of the difference is in the Document model, which can be called the API , but is more like the object environment in which the program executes. JavaScript programs execute in a web page defined by the Document Object Model (DOM). ActionScript started as the language of the Flash player so it is more oriented to construction an environment and this leads to some differences in the objects that it can use.
MXML is an XML based declarative language that compiles into ActionScript. Basically it is a shorthand for defining the static parts of an ActionScript environment. By the way, when I entered MXML into the Adobe site search engine, the first thing that came back was the question "Do you mean MSXML?", where MSXML is a MicroSoft technology.
Next we come to the runtime environments. The Flash player is lightweight client that executes compiled ActionScript and is most commonly deployed as a browser plug in (as opposed to, for example, a Browser which contains a JavaScript interpreter). Adobe AIR is a larger and more capable stand alone client for executing compiled ActionScript as well as HTML, Java etc. (as opposed to, for example, a Browser which is a client for interpreting HTML, JavaScript, et al).
Flex is the framework which means that it is a overarching name for the whole pile of technology. The one piece of technology called Flex is Flex Builder, the Eclipse based development environment for ActionScript and MXML. As they have done with other products, Adobe has open sourced a lot of technology surrounding Flex to bring more developers to the platform.
Overall, I am not sure which is more impressive, the melange of technology in Adobe Flex or the marketing effort that tries to make the whole melange of technology seem like one coherent whole.
Lets start with the programming language ActionScript. ActionScript is ECMAScript which is JavaScript. There are differences in implementation between ActionScript and other forms of JavaScript, however most of the difference is in the Document model, which can be called the API , but is more like the object environment in which the program executes. JavaScript programs execute in a web page defined by the Document Object Model (DOM). ActionScript started as the language of the Flash player so it is more oriented to construction an environment and this leads to some differences in the objects that it can use.
MXML is an XML based declarative language that compiles into ActionScript. Basically it is a shorthand for defining the static parts of an ActionScript environment. By the way, when I entered MXML into the Adobe site search engine, the first thing that came back was the question "Do you mean MSXML?", where MSXML is a MicroSoft technology.
Next we come to the runtime environments. The Flash player is lightweight client that executes compiled ActionScript and is most commonly deployed as a browser plug in (as opposed to, for example, a Browser which contains a JavaScript interpreter). Adobe AIR is a larger and more capable stand alone client for executing compiled ActionScript as well as HTML, Java etc. (as opposed to, for example, a Browser which is a client for interpreting HTML, JavaScript, et al).
Flex is the framework which means that it is a overarching name for the whole pile of technology. The one piece of technology called Flex is Flex Builder, the Eclipse based development environment for ActionScript and MXML. As they have done with other products, Adobe has open sourced a lot of technology surrounding Flex to bring more developers to the platform.
Overall, I am not sure which is more impressive, the melange of technology in Adobe Flex or the marketing effort that tries to make the whole melange of technology seem like one coherent whole.
Thursday, October 25, 2007
The Facebook Platform Internals
Ari Steinberg and Charlie Cheever of Facebook gave a fascinating talk about the design of the Facebook platform when they spoke on Tuesday to the SDForum Web Services SIG. As might be expected, the presentation drew a large audience including renowned Facebook fanboy Dave Maclure.
The Facebook platform allows outside developers to build applications and offer them to Facebook users. These applications can reach into Facebook data to do their work so security and privacy concerns are paramount. Also with Facebook approaching 50 million users and reportedly 85,000 applications, performance and scaling are big concerns.
Ari spoke first on the Facebook API. After looking at other Web 2.0 APIs and finding them wanting, they decided to use a query language approach which would give applications much greater flexibility and at the same time have the possibility of being more efficient by only fetching data that is needed. The language, FBQL is basically a simplified and restricted version of SQL.
The linguistic approach is also used in Facebook Markup Language (FBML) the language that applications and users use to define and customize their pages. As Charlie told us, it is much easier to validate and sanitize markup and style sheets from a parse tree than with regular expressions or any other technique. Also the result is much less likely to have the security loopholes that seem to plague so many Web 2.0 sites.
The Facebook platform allows outside developers to build applications and offer them to Facebook users. These applications can reach into Facebook data to do their work so security and privacy concerns are paramount. Also with Facebook approaching 50 million users and reportedly 85,000 applications, performance and scaling are big concerns.
Ari spoke first on the Facebook API. After looking at other Web 2.0 APIs and finding them wanting, they decided to use a query language approach which would give applications much greater flexibility and at the same time have the possibility of being more efficient by only fetching data that is needed. The language, FBQL is basically a simplified and restricted version of SQL.
The linguistic approach is also used in Facebook Markup Language (FBML) the language that applications and users use to define and customize their pages. As Charlie told us, it is much easier to validate and sanitize markup and style sheets from a parse tree than with regular expressions or any other technique. Also the result is much less likely to have the security loopholes that seem to plague so many Web 2.0 sites.
Sunday, October 21, 2007
OLAP with Mondrian
I got three things out of Julian Hyde's talk the the SDForum SIG last week. Julian is the lead developer on the Open Source Mondrian project and his talk was entitled "Building scalable OLAP applications with Mondrian and Pentaho".
The first thing is the nature of Mondrian. While most OLAP servers are database like servers that persistently store the OLAP cubes, Mondrian is a cache. At the back end, Mondrian uses JDBC to fetch persistent data from a database. At the front end Mondrian accepts queries in the MDX language and replies to these queries with data cubes. In the middle, Mondrian caches data cubes in memory so that it can respond to queries faster. Mondrian is a component. It is packaged as a single Java .jar file, and its schema is defined by an XML file. Mondrian needs both a back end database and a front end, either something like JPivot that can display OLAP cubes or an XML/A driver that can communicate MDX queries with a remote front end.
This brings us to the second thing. While there have been several attempts to create a standard interface to OLAP servers, none have really succeeded. Julian has spearheaded a new Open Source initiative called olap4j. The concept is that olap4j will be the JDBC of OLAP. It is based on JDBC, and uses the MDX language to express OLAP queries. MDX is the universal language of OLAP, just as SQL is the universal language of relational databases. The only curiosity is MDX is owned by Microsoft and that other implementations of MDX like Mondrian have to scramble to match each Microsoft release. I think that it is about time that MDX is managed by an independent standards committee.
Finally Julian talked about a "real-time" feature that he has recently added to Mondrian. This is a cache control API that allows you to invalidate parts of the Mondrian cache when new data is loaded into the underlying database. Having invalidated part of the cache, new queries fetch the new data and display up to date results. Initially the example looked complicated, and the audience reacted with some skepticism. On looking back at the example in the presentation, it seems less forbidding, however the long term goal should be to automatically manage the OLAP cache from an ETL tool. When pressed, Julian told us that the API is being used, for example, in applications for securities analysis.
In all, it was an satisfyingly informative evening.
The first thing is the nature of Mondrian. While most OLAP servers are database like servers that persistently store the OLAP cubes, Mondrian is a cache. At the back end, Mondrian uses JDBC to fetch persistent data from a database. At the front end Mondrian accepts queries in the MDX language and replies to these queries with data cubes. In the middle, Mondrian caches data cubes in memory so that it can respond to queries faster. Mondrian is a component. It is packaged as a single Java .jar file, and its schema is defined by an XML file. Mondrian needs both a back end database and a front end, either something like JPivot that can display OLAP cubes or an XML/A driver that can communicate MDX queries with a remote front end.
This brings us to the second thing. While there have been several attempts to create a standard interface to OLAP servers, none have really succeeded. Julian has spearheaded a new Open Source initiative called olap4j. The concept is that olap4j will be the JDBC of OLAP. It is based on JDBC, and uses the MDX language to express OLAP queries. MDX is the universal language of OLAP, just as SQL is the universal language of relational databases. The only curiosity is MDX is owned by Microsoft and that other implementations of MDX like Mondrian have to scramble to match each Microsoft release. I think that it is about time that MDX is managed by an independent standards committee.
Finally Julian talked about a "real-time" feature that he has recently added to Mondrian. This is a cache control API that allows you to invalidate parts of the Mondrian cache when new data is loaded into the underlying database. Having invalidated part of the cache, new queries fetch the new data and display up to date results. Initially the example looked complicated, and the audience reacted with some skepticism. On looking back at the example in the presentation, it seems less forbidding, however the long term goal should be to automatically manage the OLAP cache from an ETL tool. When pressed, Julian told us that the API is being used, for example, in applications for securities analysis.
In all, it was an satisfyingly informative evening.
Wednesday, July 04, 2007
Design Patterns in Python
Alex Martelli is a leading light of the Python programming language community. He is a leader in the development of the language, author of Python in a Nutshell and has written extensively on Python in other books and articles. Last week he spoke to the SDForum Software Architecture and Modeling SIG on "Design Patterns in Python". Alex has posted the presentation slides here.
Design patterns are a useful concept in programming. A programming language consists of a set of basic constructs that are used in various ways to build a program. Design patterns capture higher level constructs that commonly appear in programs. As Alex explained, an important part of each design pattern is its name. Having a good and well known name for each pattern makes it easy to explain a program and discuss how it works at a higher level than the programming language constructs.
Although design patterns are supposed to be programming language agnostic, many of the books on design patterns are based on statically typed languages such as C++ and Java. There are some classic design patterns that do not translate well into dynamically types languages like Python. One example is the Singleton design pattern. This uses strong typing to ensures that there can be only one instance of a particular type of object.
Python is a dynamically typed language, and in these languages the type system is used to make sure that the right thing happens at run time rather than to do any enforcement. The consequence is that you cannot really do a singleton in Python or any other dynamically typed language. If I were in the dynamic language camp, I would argue that if you only want one instance of an object, make sure that you only create one instance of the object. This is clearly in the spirit of dynamic typing.
However Alex is clearly concerned by the absence of these design patterns from Python. First he claimed that these design patterns do not exist in Python because it is a higher level language and that the design patterns are somehow subsumed by the language system. Then he went on to show us his Python answer to the singleton design pattern, which he called the Borg. In the Borg, all instances of the class share the same state, and so they all appear to be the same instance.
While the Borg pattern works in Python, I do not see it a either necessary or interesting. Alex told us, Guido van Rossum, the father of Python, does not like it either.
Design patterns are a useful concept in programming. A programming language consists of a set of basic constructs that are used in various ways to build a program. Design patterns capture higher level constructs that commonly appear in programs. As Alex explained, an important part of each design pattern is its name. Having a good and well known name for each pattern makes it easy to explain a program and discuss how it works at a higher level than the programming language constructs.
Although design patterns are supposed to be programming language agnostic, many of the books on design patterns are based on statically typed languages such as C++ and Java. There are some classic design patterns that do not translate well into dynamically types languages like Python. One example is the Singleton design pattern. This uses strong typing to ensures that there can be only one instance of a particular type of object.
Python is a dynamically typed language, and in these languages the type system is used to make sure that the right thing happens at run time rather than to do any enforcement. The consequence is that you cannot really do a singleton in Python or any other dynamically typed language. If I were in the dynamic language camp, I would argue that if you only want one instance of an object, make sure that you only create one instance of the object. This is clearly in the spirit of dynamic typing.
However Alex is clearly concerned by the absence of these design patterns from Python. First he claimed that these design patterns do not exist in Python because it is a higher level language and that the design patterns are somehow subsumed by the language system. Then he went on to show us his Python answer to the singleton design pattern, which he called the Borg. In the Borg, all instances of the class share the same state, and so they all appear to be the same instance.
While the Borg pattern works in Python, I do not see it a either necessary or interesting. Alex told us, Guido van Rossum, the father of Python, does not like it either.
Thursday, June 14, 2007
Programming Language Wars
I have been doing a lot of programming recently, using a wide variety of programming languages. In the last year I have written programs or modules in the following programming languages: C, C++, Java, Perl, PHP, Python, Shell and various dialects of SQL. Tiobe Software maintains an interesting Community Index that tracks programming language popularity. From their data I calculate that I am using 65% of current programming languages (by volume).
For anyone new to the game, Programming Language Wars have a long and storied history. As soon as the first programming was implemented, there were wars about whether to use it or assembler. Shortly thereafter there were two programming languages and the battle commenced with earnest. For example, this article remembers some of the more ridiculous wars from the 70's and 80's. Newbies, get over it, we are at least on the 423 "Great Programming Language War".
I am going to write a series of posts on programming languages that I hope will provide a more balanced view that the usual blast.
For anyone new to the game, Programming Language Wars have a long and storied history. As soon as the first programming was implemented, there were wars about whether to use it or assembler. Shortly thereafter there were two programming languages and the battle commenced with earnest. For example, this article remembers some of the more ridiculous wars from the 70's and 80's. Newbies, get over it, we are at least on the 423 "Great Programming Language War".
I am going to write a series of posts on programming languages that I hope will provide a more balanced view that the usual blast.
Wednesday, May 02, 2007
Save Time, Generate Code
I mentioned the after party in the previous post, now is time to talk about the SDForum SAM SIG meeting "Writing Code Generators For Quality, Productivity, and Fun" by Bill Venners of Artima. In short, Bill's talk was about designing domain specific languages and using code generation to solve programming problems. Code generation can build better, more reliable code faster, particularly in situations where you find yourself doing a lot of cut and paste style .
As befits the technique, Bill got the audience to do a lot of the lifting by describing how they had solved various programming problems through code generation. I have used it a couple of times as well as some real language design, and I shared one of my experiences along with several other audience members. It was definitely encouraging to hear that many audience members had successfully used code generation.
Bill characterized a number of situations where code generation is useful and showed us a specific example of a domain specific language that he had designed. From my experience language design is comparable to API design except that there are more degrees of freedom so there are more ways to go wrong. Designing a real programming language is hard. There are issues at the lexical level, problems with keywords and extensibility, the need to make the language regular and unsurprising while dealing with special case and a thousand other little details to get right.
On the other hand, if a domain specific language can be kept simple enough, it can skirt these problems. Bill's example language was simple. Each statement consisted of a set of keywords followed by a name. In my opinion, domain specific languages are a great technique provided that you firstly design a clean simple language and secondly remember the principals of API design.
As befits the technique, Bill got the audience to do a lot of the lifting by describing how they had solved various programming problems through code generation. I have used it a couple of times as well as some real language design, and I shared one of my experiences along with several other audience members. It was definitely encouraging to hear that many audience members had successfully used code generation.
Bill characterized a number of situations where code generation is useful and showed us a specific example of a domain specific language that he had designed. From my experience language design is comparable to API design except that there are more degrees of freedom so there are more ways to go wrong. Designing a real programming language is hard. There are issues at the lexical level, problems with keywords and extensibility, the need to make the language regular and unsurprising while dealing with special case and a thousand other little details to get right.
On the other hand, if a domain specific language can be kept simple enough, it can skirt these problems. Bill's example language was simple. Each statement consisted of a set of keywords followed by a name. In my opinion, domain specific languages are a great technique provided that you firstly design a clean simple language and secondly remember the principals of API design.
Monday, April 30, 2007
Duh Typing
The meeting of the SDForum SAM SIG last week was interesting, however the discussion afterwards over a beer was just as interesting. I will write about the SIG meeting another time, here I want to capture an idea from the after session.
There is a lot going on in the world of programming languages, dynamic versus static languages, strong versus weak typing. Proponents of dynamic languages like Python and Perl claim that they are superior because you do not need to continually specify the type of an object, particularly when the type of the object is obvious.
Over a beer someone proposed that this continual restating of the type of an object should be called Duh Typing (a play on Duck Typing). I ran into an example of Duh Typing today when I found myself writing the following Java statement:
There is a lot going on in the world of programming languages, dynamic versus static languages, strong versus weak typing. Proponents of dynamic languages like Python and Perl claim that they are superior because you do not need to continually specify the type of an object, particularly when the type of the object is obvious.
Over a beer someone proposed that this continual restating of the type of an object should be called Duh Typing (a play on Duck Typing). I ran into an example of Duh Typing today when I found myself writing the following Java statement:
ManagedTasks[] managedTasks =The dynamic language guys would laugh at the fact that I have to specify the type 3 times in a single statement. Well, it is not quite as easy all all that as I will discuss in future posts. In the mean time here is a neat name for the concept.
(ManagedTasks[]) tasks.toArray
(new ManagedTasks[tasks.size()]);
Subscribe to:
Posts (Atom)
