EU

Open Source Visionaries

May 20, 2011

 

Creating, extending and maintaining modern computational tools is really hard. It used to be that a motivated person could envision a solution, knock out some code, and have a useful prototype running in a short time (days or weeks). The resulting code was simple (and short), the build environment was typically a brief makefile (if that), cross-platform issues were ignored, and the narrowly focused solution didn’t require much of an architecture. While this approach is still in limitied use today, serious users and developers of computational software are addressing daunting new challenges: algorithmic complexity, cross-platform development, extensive integration requirements, new computing models (distributed, multi-core, GPU) and computing environments (client/server, cloud, web-based, mobile), and a plethora of computing languages. What used to be a relatively quick process to create something useful has now become a focused effort of months or years.
As a result I think we’ve come to the point where open source approaches are the only viable way to build and sustain large-scale computational tools. Many others have made similar claims arguing along the lines of intellectual freedom and engineering process. For example, it has become increasingly evident that the lofty philosophical roots of open source freedom are absolutely essential to the practice of Open Science (and the resulting innovation). And if you are a pragmatist, there are clear engineering benefits including scalable software development (ref cathedral and the bazaar), agile software processes, and community maintenance. However, there is another critical argument to make in favor of open source — its power to create, implement and sustain a long term Collaborative Vision — which I believe is underappreciated yet essential to the future of scientific computing software.
Let me put it this way: How are we going to create and pay for the advanced computational tools of the future? While the long answer is complex, I think there are two key issues that must be addressed in any solution: the need to create and sustain a long-term vision, and the ability to pay for it. In both these areas, I believe that open approaches carry decided advantages.
Take vision for example. Without clear vision a software system is doomed to cluttered, fragmented and clumsy implementations. However with too rigid a vision, or a vision that cannot adapt and change, a system soon becomes obsolete and eventually dies. Large systems often begin with a modest vision and a few developers, but as it succeeds and grows the corresponding vision must also grow (not to mention the community). To accomplish this you can’t do it behind closed doors (well maybe you can if you’re big enough and rich enough but this is not common). For most of us, the only way to really grow the vision is to mix the visionary technologies and ideas that others have created into our software stew and make it part of the recipe. In other words you need a community to provide its collective intelligence to hold and grow the vision, which of course is a hallmark of open source practices.
Now how to pay to manifest the vision? For example, tools like VTK and ITK are conservatively estimated to have cost tens of millions of dollars to develop over decades [ref]. How many venture firms, angel investors, funding agencies, and companies do you think would have the patience and resources for that kind of effort? Especially when we are talking about complex computational tools whose commercial potential is murky at best (unless you are a company that is smart enough to turn computational infrastructure into a service ;-)). In an open approach, once you have developed the vision, it is possible for thousands of users and developers, for their own particular reasons, to contribute to implementing the vision–some simply volunteering for the intellectual challenge; others committed to building a scientific and engineering infrastructure to avoid reinventing the wheel over and over again; still others joining forces and attacking grand scientific challenges; and even commercial entitites who wish to reduce maintenance costs by contributing code to the community. Whatever the reason, each of these parties contributes time, money and knowledge which inexorably builds towards realization of the vision. Of course, this only works in an open environment.
So I would add to the current list of advantages of open source systems, intellectual freedom and scalable engineering process, Collaborative Vision–the ability to envision and manifest advanced technologies over extended periods while efficiently combining the resources of many people and organizations. I believe it is the ability of open source communities to create, hold, and evolve their combined vision, along with the efficient capture of contributions, that will eventually overtake closed approaches and create an open, sustainable computing infrastructure.

Creating, extending and maintaining modern computational tools is really hard. It used to be that a motivated person could envision a solution, knock out some code, and have a useful prototype running in a short time (days or weeks). The resulting code was simple (and short), the build environment was typically a brief makefile (if that), cross-platform issues were ignored, and the narrowly focused solution didn’t require much of an architecture. While this approach is still in limited use today, serious users and developers of computational software are addressing daunting new challenges: algorithmic complexity, cross-platform development, extensive integration requirements, new computing models (distributed, multi-core, GPU) and computing environments (client/server, cloud, web-based, mobile), and a plethora of computing languages. What used to be a relatively quick process to create something useful has now become a focused effort of months or years.

As a result I think we’ve come to the point where open source approaches are the only viable way to build and sustain large-scale computational tools. Many others have made similar claims arguing along the lines of intellectual freedom and engineering process. For example, it has become increasingly evident that the lofty philosophical roots of open source freedom are absolutely essential to the practice of Open Science (and the resulting innovation). And if you are a pragmatist, there are clear engineering benefits including scalable software development (The Cathedral and the Bazaar), agile software processes, and community maintenance. However, there is another critical argument to make in favor of open source — its power to create, implement and sustain a long term Collaborative Vision — which I believe is under appreciated yet essential to the future of scientific computing software.

Let me put it this way: How are we going to create and pay for the advanced computational tools of the future? While the long answer is complex, I think there are two key issues that must be addressed in any solution: the need to create and sustain a long-term vision, and the ability to pay for it. In both these areas, I believe that open approaches carry decided advantages.

Take vision for example. Without clear vision a software system is doomed to cluttered, fragmented and clumsy implementations. However with too rigid a vision, or a vision that cannot adapt and change, a system soon becomes obsolete and eventually dies. Large systems often begin with a modest vision and a few developers, but as it succeeds and grows the corresponding vision must also grow (not to mention the community). To accomplish this you can’t do it behind closed doors (well maybe you can if you’re big enough and rich enough but this is not common). For most of us, the only way to really grow the vision is to mix the visionary technologies and ideas that others have created into our software stew and make it part of the recipe. In other words you need a community to provide its collective intelligence to hold and grow the vision, which of course is a hallmark of open source practices.

Now how to pay to manifest the vision? For example, tools like VTK and ITK are conservatively estimated to have cost tens of millions of dollars to develop over decades (see the bottom of the page describing the NA-MIC Kit). How many venture firms, angel investors, funding agencies, and companies do you think would have the patience and resources for that kind of effort? Especially when we are talking about complex computational tools whose commercial potential is murky at best (unless you are a company that is smart enough to turn computational infrastructure into a service ;-)). In an open approach, once you have developed the vision, it is possible for thousands of users and developers, for their own particular reasons, to contribute to implementing the vision–some simply volunteering for the intellectual challenge; others committed to building a scientific and engineering infrastructure to avoid reinventing the wheel over and over again; still others joining forces and attacking grand scientific challenges; and even commercial entitites who wish to reduce maintenance costs by contributing code to the community. Whatever the reason, each of these parties contributes time, money and knowledge which inexorably builds towards realization of the vision. Of course, this only works in an open environment.

So I would add to the current list of advantages of open source systems, intellectual freedom and scalable engineering process, Collaborative Vision–the ability to envision and manifest advanced technologies over extended periods while efficiently combining the resources of many people and organizations. I believe it is the ability of open source communities to create, hold, and evolve their combined vision, along with the efficient capture of contributions, that will eventually overtake closed approaches and create an open, sustainable computing infrastructure.

Tags:
Scientific Computing

Leave a Reply