Newer posts are loading.
You are at the newest post.
Click here to check if anything new just came in.

February 17 2012

Documentation strategy for a small software project: launching VoIP Drupal introductions

VoIP Drupal is a window onto the promises and challenges faced by a new open source project, including its documentation. At O'Reilly, we've been conscious for some time that we lack a business model for documenting new collaborative projects--near the beginning, at the stage where they could use the most help with good materials to promote their work, but don't have a community large enough to support a book--and I joined VoIP Drupal to explore how a professional editor can help such a team.

Small projects can reach a certain maturity with poor and sparse document. But the critical move from early adopters to mainstream requires a lot more hand-holding for prospective users. And these projects can spare hardly any developer time for documentation. Users and fans can be helpful here, but their documentation needs to be checked and updated over time; furthermore, reliance on spontaneous contributions from users leads to spotty and unpredictable coverage.

Large projects can hire technical writers, but what they do is very different from traditional documentation; they must be community managers as well as writers and editors (see Anne Gentle's book Conversation and Community: The Social Web for Documentation). So these projects can benefit from research into communities also.

I met at the MIT Media Lab this week with Leo Burd, the inventor of VoIP Drupal, and a couple other supporters, notably Micky Metts of We worked out some long-term plans for firming up VoIP Drupal's documentation and other training materials. But we also had to deal with an urgent need for materials to offer at DrupalCon, which begins in just over one month.


One of the difficulties of explaining VoIP Drupal is that it's just so versatile. The foundations are simple:

  • A thin wrapper around PHP permits developers to write simple scripts that dial phone numbers, send SMS messages, etc. These scripts run on services that initiate connections and do translation between voice and text (Tropo, Twilio, and the free Plivo are currently supported).

  • Administrators on Drupal sites can use the Drupal interface to configure VoIP Drupal modules and add phone/SMS scripts to their sites.

  • Content providers can use the VoIP Drupal capabilities provided by their administrators to do such things as send text messages to site users, or to enable site users to record messages using their phone or computer.

Already you can see one challenge: VoIP Drupal has three different audiences that need very different documentation. In fact, we've thought of two more audiences: decision-makers who might build a business or service on top of VoIP Drupal, and potential team members who will maintain and build new features.

Some juicy modules built on top of VoIP Drupal's core extend its versatility to the point where it's hard to explain on an elevator ride what VoIP Drupal could do. Leo tosses out a few ideas such as:

  • Emergency awareness systems that use multiple channels to reach out to a population who live in a certain area. That would require a combination of user profiling, mapping and communication capabilities tend to be extremely hard to put together under one single package.

  • Community polling/voting systems that are accessible via web, SMS, email, phone, etc.

  • CRM systems that keep track (and even record) phone interactions, organize group conference calls with the click of a button, etc.

  • Voice-based bulletin boards.

  • Adding multiple authentication mechanisms to a site.

  • Sending SMS event notifications based on Google Calendars.

In theory you could create a complete voice and SMS based system out of VoIP Drupal and ignore the web site altogether, but that would be a rather cumbersome exercise. VoIP Drupal is well-suited to integrating voice and the Web--and it leaves lots of room for creativity.

Long-term development

A community project, we agreed, needs to be incremental and will result in widely distributed documents. Some people like big manuals, but most want a quickie getting-started guide and then lots of chances to explore different options at their own pace. Communities are good for developing small documents of different types. The challenge is finding someone to cover any particular feature, as well as to do the sometimes tedious work of updating the document over time.

We decided that videos would be valuable for the administrators and content providers, because they work through graphical interfaces. However, the material should also be documented in plain text. This expands access to the material in two ways. First, VoIP Drupal may be popular in part of the world where bandwidth limitations make it hard to view videos. Second, the text pages are easier to translate into other languages.

Just as a video can be worth a thousand words, working scripts can replace a dozen explanations. Leo will set up a code contribution site on Github. This is more work than it may seem, because malicious or buggy scripts can wreak havoc for users (imagine someone getting a thousand identical SMS messages over the course of a single hour, for instance), so contributions have to be vetted.

Some projects assign a knowledgeable person or two to create an outline, then ask community members to fill it in. I find this approach too restrictive. Having a huge unfilled structure is just depressing. And one has to grab the excitement of volunteers wherever it happens to land. Just asking them to document what they love about a project will get you more material than presenting them with a mandate to cover certain topics.

But then how do you get crucial features documented? Wait and watch forums for people discussing those features. When someone seems particularly knowledgeable and eager to help, ask him or her for a longer document that covers the feature. You then have to reward this person for doing the work, and a couple ways that make sense in this situation include:

  • Get an editor to tighten up the document and work with the author to make a really professional article out of it.

  • Highlight it on your web site and make sure people can find it easily. For many volunteers, seeing their material widely used is the best reward.

We also agreed that we should divide documentation into practical, how-to documents and conceptual documents. Users like to grab a hello-world document and throw together their first program. As they start to shape their own projects, they realize they don't really understand how the system fits together and that they need some background concepts. Here is where most software projects fail. They assume that the reader understands the reasoning behind the design and knows how best to use it.

Good conceptual documentation is hard to produce, partly because the lead developers have the concepts so deeply ingrained that they don't realize what it is that other people don't know. Breaking the problems down into small chunks, though, can make it easier to produce useful guides.

Like many software projects, VoIP Drupal documentation currently starts the reader off with a list of modules. The team members liked an idea of mine to replace these with brief tutorials or use cases. Each would start with a goal or question (what the reader wants to accomplish) and then introduce the relevant module. In general, given the flexibility of VoIP Drupal, we agreed we need a lot more "why and when" documentation.

Immediate preparations

Before we take on a major restructuring and expansion of documentation, though, we have a tight deadline for producing some key videos and documents. Leo is going to lead a development workshop at DrupalCon, and he has to determine the minimum documentation needed to make it a productive experience. He also wants to do a webinar on February 28 or 29, and a series of videos on basic topics such as installing VoIP Drupal, a survey of successful sites using it, and a nifty graphical interface called Visual VoIP Drupal. Visual VoIP Drupal, which will be released in a few weeks, is one of the new features Leo would like to promote in order to excite users. It lets a programmer select blocks and blend them into a script through a GUI, instead of typing all the code.

The next few weeks will bring a flurry of work to realize our vision.

December 22 2011

Developer Year in Review: 2011 Edition

This year brought us triumphs and tragedies, new companies born and old ones burning out. Before DWiR takes a holiday hiatus, we're going to look back on the high points of the year that was.

Mobile gains ground


Lost in all the news about lawsuits, patents and speculation was the overarching theme for mobile this year: it has become the primary software platform for many users. The desktop may not be dead, but it's definitely showing its age, and as smartphones and tablets become ubiquitous, the amount of time the average consumer spends in front of a keyboard is declining rapidly.

The good news for software developers is that the maturing app store model has opened up software distribution to a much larger pool of potential software makers. The bad news is that it has also drastically reset the expectation of how much consumers are willing to spend for apps, although prices are climbing marginally. A $1 app can make you a lot of money if you can get millions of users to buy it, but it won't even get you a nice night on the town if you're writing for a niche market.

With RIM's Blackberry market share doing a good imitation of an Olympic high diver, and the new Windows mobile platform not yet gaining significant traction, 2011 was essentially a two-horse race, with Android passing iOS for the first time in new sales. Apple is crying all the way to the bank, though, as the profit margin on iOS devices is pushing Apple's bottom line to new highs and overall unit sales continue to climb steadily. At least for the moment, the smartphone market is not a zero-sum game.

This year also marked the release of Ice Cream Sandwich (ICS) for Android and iOS 5 for the iPhone/iPad/iPod. ICS is the first version of Android that is making serious efforts to tame the tablet situation, but there have been widespread complaints that carriers are slow to pick it up, even in new models. Objective-C developers are finally getting to say goodbye to old friends like retain, release and autorelease, as Apple rolled out the automatic reference count compiler. Few tears were shed for their passing.

Strata 2012 — The 2012 Strata Conference, being held Feb. 28-March 1 in Santa Clara, Calif., will offer three full days of hands-on data training and information-rich sessions. Strata brings together the people, tools, and technologies you need to make data work.

Save 20% on registration with the code RADAR20

The year of HTML5

In future years, 2011 will be remembered as the year Adobe put up the white flag and joined the HTML5 bandwagon, which started an industry death-watch for Flash. Microsoft also sent out signals that Silverlight was being put out to pasture and that it planned to embrace HTML5 as well.

The stampede to adopt HTML5 was prompted, in part, by the increasing robustness of the standard and the implementations of the standard in browsers. It also didn't hurt that it is the only Rich Internet Application platform that will run on the iPad.

Dru-who and Ha-what?

Two packages with funny names became the hot skills to have on your resume this year. Drupal continued to gain popularity as a content management platform, while Apache Hadoop was the must-have technology for data crunching. By the end of the year, developers with experience in either were in short supply and could basically write their own tickets.

Languages emerge, but few stick

It seems like every year, there's a new batch of languages that promise to be the next Big Thing. In past years, the crown has been worn by Scala, Erlang, Clojure and others. But when it comes time to start a project or hire developers, skills in new languages are rarely high on the list of priorities for companies.

This year, Google joined the fun, promoting both Go and Dart. Like most new languages, they face an uphill battle, even with Google's massive resources behind them. Few have what it takes to fight the institutional inertia of existing development decisions and to join winners such as Ruby in the pantheon of well-adopted emerging languages.

Some general thoughts to end the year

The computer industry, more than most others, can make you feel very old at a relatively young age. I've been hacking, in one form or another, for nearly 35 years, and the technology I used in my youth seems like it belongs in another universe.

The flip side of this is that I'm constantly amazed by what science and technology brings forth on a seemingly daily basis. Whether it's having a conversation with a device I can hold in the palm of my hand or watching the aurora light up the heavens, seen from above by occupants of the ISS, I often seem to be living in the future I read about as a kid.

As a species, we may be prone to pettiness, violence, willful ignorance and hatred, but once in a while, we manage to pull ourselves out of the muck and do something insanely great. Let's attempt to honor the vision of an admittedly imperfect man we lost this year and try to make 2012 insanely greater.

Got news?

Please send tips and leads here.


November 21 2011

VoIP Drupal reaches out to the developing world

I don't know why so few of us turned up on Saturday for the VoIP Drupal hackathon. As a way to integrate voice and SMS into a Drupal site, the VoIP modules form a door throught which Drupal can move into a vast world of touch tone telephones, smart telephones, and text messaging, and therefore toward integrating a huge range of users in developing regions who use those technologies instead of desktop or laptop computers. Perhaps Boston isn't the right place or November the right month for a workshop (although the weather was quite nice), but just four of us gathered to get the low-down on VoIP Drupal from Leo Burd, a research scientist at MIT's Media Lab and Center for Civic Media.

Together with just a couple other developers, he is putting together modules that support Twilio and Tropo, two cloud platforms that are highly scalable and provide telephone and SMS capabilities accessible from different countries. For cases where those services are not available or desirable, VoIP Drupal provides support for Free/SWITCH, an open source telephony platform, via the Plivo communication framework/API.

Most of the time we played with the scripting language using the VoIP Drupal sandbox. The scripting language is a domain-specific language for VoIP built on top of Drupal's module language, PHP. It has about 15 commands to create interactive calls doing such things as recording and playing back audio, handling input from the telephone keypad, managing conference calls, and sending and receiving SMS messages. A trivial script I created went like this:

$script->addSetVoice('woman'); $script->addSetLanguage('fr');

$script->addSay('Voiçi un message. Ne répondez pas.');


(The scripting language requires you to create the $script object first, but the sandbox does that for you silently.) When I played this back, I got a pretty authentic sounding Parisian voice, having even the suitably cavalier tone when she told me not to talk in return (although she was tolerant of my spelling mistakes).

Of course, much richer applications are available through the scripting language. It is mostly linear, although you can define and call subroutines, you can set and retrieve variables, and there is a primitive assembly-language-like statement that lets you branch to a label based on a condition. Furthermore, the modules' full power is available through a PHP API. The Drupal administration menu allows you to specify a script to play when the site makes an outgoing call, a script to play when someone calls the site's phone number, and a script to play when someone sends a text message to the site's phone number.

Burd showed off a site put together by a non-profit in Dorchester (a low-income area of Boston, Mass.) together with the MIT Center for Civic Media. A group of young students recorded some descriptions of nearby locations of interest. These locations display plaques with a phone number for the web site and an extension unique for their location. Someone dialing in hears the message and is invited to record his or her own opinions or stories about that part of the city. Attendees today were so impressed that they said, if this application could be released as a drop-in module, it would boost the use of the VoIP modules immediately.

Just a few of the many uses for VoIP in Drupal include:

  • The equivalent of mass mailings via voice calls and SMS, so you can send messages, for instance, to people who sign up for political campaigns

  • Letting visitors leave voice mail or add verbal comments to the site

  • Embedding a phone interface on the web page so people can make VoIP calls directly from your site, with no extra stand-alone software such as Skype

  • Providing a conference call service through your site

  • Letting people sign up for groups to receive SMS messages on chosen topics of interest

More modules are under construction; an overview is available on the Drupal web site. A messaging module lets you send a voice message that is delivered in by phone, email, or SMS, as preferred by the site's visitor. The modules are developed for Drupal version 6, but Burd plans to create Drupal 7 modules as soon as the version 6 ones reach their 1.0 release, depending on interest from the community.

Of the underlying services supported, Twilio offers voice generation for four languages. Tropo supports voice generation for 24 languages, and can also do speech-to-text. Both of those companies have been very friendly to the VoIP Drupal project and promote it at Drupal conferences. Free/SWITCH and Plivo require you to do a lot of the work that Twilio and Tropo will do for you. But Free/SWITCH is useful for areas without Twilio or Tropo support, and for high volume use because it tends to cost less under those circumstances. Free/SWITCH also give the programmer more control over the server and allows you to run everything from the same box. Overall, VoIP Drupal represents another step toward an Internet where communicating by voice is taken for granted.

August 04 2011 relaunches using open source and the cloud

In April, Radar reported that was moving to Drupal. This morning, the Energy Department launched a redesigned as an interactive open platform that enables information exchange, open data and localized information for citizens. The new uses a combination of open source technology and cloud computing will save an estimated $10 million dollars annually, according to Energy Department officials.

"Our goal is to make easier to use, more transparent and more participatory," said Secretary of Energy Steven Chu in a prepared statement. "This next phase is part of our ongoing commitment to empower consumers and businesses with the information, tools and services they need to save money, create jobs and find opportunities in the new energy economy."

The new is built using Drupal 7, the same open source content management system used at,,, and it's the system that supported the reboot of as an open government platform. Drupal distributions are now supporting a growing number of open government platforms in local, state and federal government.

Strata Conference New York 2011, being held Sept. 22-23, covers the latest and best tools and technologies for data science -- from gathering, cleaning, analyzing, and storing data to communicating data intelligence effectively.

Save 20% on registration with the code STN11RAD

Saving money through open source

The new site was implemented by several different firms. The Treehouse Agency built the backend, HUGE Inc designed the front end, Acquia helped with Drupal support and Energy Enterprise Solutions served as the integrator. is hosted in the cloud by BlackMesh.

[Disclosure: O'Reilly AlphaTech Ventures is an investor in Acquia.]

"The initial investment for the project was nearly paid for by consolidating other sites into this platform and not building new stand-alone sites in other places," wrote Cammie Croft, senior adviser and director of new media and citizen engagement at the Department of Energy, in an email this morning.

"We strategically invested our resources into open source and cloud solutions where possible," she wrote. "We anticipate more cost savings as we consolidate more sites into this platform and eliminate duplicative and out-dated website infrastructure elsewhere. As we consolidate more sites into the platform, we anticipate cost savings or avoidance of $10 million in a year. As of this launch, we're over $1 million in cost savings or avoidance."

The redesigned site has several notable bells and whistles around localization, data visualizations and open source mapping tools that use Node.js. features interactive maps built from open government data using MapBox, a map design suite from Washington, D.C.-based development firm Development Seed. For instance, an "alternative fuel locator" dataset is mapped and embedded below:

"This is about telling complex stories with data, and beautiful maps matter," said Development Seed founder Eric Gunderson in an interview this morning. "It just makes data a lot more consumable for citizens. The best part is that agencies are now able to do this for free on their own using open source tools."

For more ways that the Energy Department is tapping technology to deliver on its mission, including fuel economy, a solar decathlon, ARPA-E and more, make sure to read Aliza Sherman's excellent Mashable article.

Croft emphasized that the relaunch shouldn't be seen simply through the prism of cost savings alone. "This isn't about reducing the bottom line," she wrote. "It's about being more strategic with our investments in digital communications and technology."

The site allows Energy staff to create new sites without needing to go to developers. They'll "own" their own platform and will be able to add more functionality from the open source community in the future and contribute code back as well.

In that context, open source is playing an important role in open government, although it's hardly a precondition for it. Whether it's, coding the middleware for open government data or codesharing with CivicCommons, open source matters more than ever. As we moved together into the 21st century, open source technology and collaborative models will matter in media, mapping, education, smarter cities, national security, disaster response and much more in 2011 and beyond. The success of open source in building systems that work at scale offers an important lesson to government leaders as well: to meet grand national challenges and create standards for the future, often it's best to work on them together.

April 27 2011

Four short links: 27 April 2011

  1. Aaargh! Physicists! --the dangers of venturing outside your area of expertise is that someone will mercilessly point out your overconfident missteps, as happens here. Unless, of course, your new field is social media, in which case there are hundreds of thousands of sycophantic circlejerkers ready to retweet, link back, and Like your misbegotten ill-conceived content-free mindless dribblings.
  2. Crowdsourcing to Improve Sales -- products sell better if their reviews have good spelling and grammar, so one retailer used MTurk to copyedit reviews and thus improve sales. (via Matt Biddulph on Delicious)
  3. Converting the Virtual Economy into Development Potential -- World Bank reports that online microwork earned $3B for poor countries in 2009, and encourages third world countries to invest in infrastructure to support this. Interestingly, the authors include both gold farming and "cherry blossoming" (clickfraud) as microwork. (via BoingBoing)
  4. Misery -- Drupal module that makes life difficult for trolls--randomly wrong redirections, slow page loads, white screens, forms that don't submit, etc. Brilliant! (via Andy Baio)

February 22 2011

Phase2 acquires two more Drupal distributions

DC-based open source development shop Development Seed and Phase2 Technology announced that Phase2 has acquired two Drupal distributions, Open Atrium and Managing News.

Phase2 now has the top four Drupal distributions in use in government under its wings. It already hosted the OpenPublic and OpenPublish distributions, both of which are in use by numerous government and publishing clients. The ability to share code between government bodies — something that better interoperability between distributions might facilitate — could be a considerable cost saver for budget-strapped governments at all levels.

The new .gov websites emerging as government platforms include support for rich media, ideation capabilities, accessibility, and application programming interfaces (APIs) that enable developers to build upon them. That's why, for instance, rebooting the FCC online will mean moving to Drupal. Later this year, will move to Drupal. Notably, the websites for dozens of freshman representatives on moved to Drupal. Of course, these recent moves all follow, which switched to Drupal in 2009.

January 13 2011 moves to Drupal

capitol-winter.jpg In another win for open source, the United States House of Representatives has chosen Drupal as the web content management system for That makes the "People's House" one of the largest government institutions to move to the open source web content management platform.

The move follows a series of high profile moves to Drupal in the federal government, including and, more recently,

In a request for sources posted to, embedded below, the House of Representatives chief administrative officer indicated the institution's interest in assembling a list of vendors to support Drupal development. According to the document, there are some 520 websites at for Drupal developers to work on.

Drupal creator Dries Buytaert offered insight into how the House of Representatives will use Drupal, including background and examples of recently deployed Drupal sites, on his blog. (Note: Buytaert is currently the CTO at Acquia, one of the vendors involved in the House's Drupal effort.)

When the House went looking for a platform to standardize their web presence, they chose Drupal because it met their requirements to:

  1. Accommodate hundreds of independent websites, each with different sets of features.

  • Provide the ability to deploy new sites quickly and efficiently.

  • Enable House Members to use the web designer or developer of their choice by leveraging the Drupal community.

  • As the new House continues to move forward with an e-transition under the GOP, open source and Drupal developers are looking like increasingly important players in the Washington IT ecosystem.

    Drupal Sources Sought


    December 17 2010 to move to Drupal

    Rack up another win for open source in government. According to Cammie Croft (@cammiecroft), the director of new media at the Department of Energy, and all of the websites from the department's program offices will relaunch using Drupal next spring.

    For those who track such things, the U.S. Department of Energy has hundreds of websites, all of which are currently maintained through a variety of content management systems (CMS). Astonishingly, in 2010, a few are still being manually maintained. Earlier this week, I caught up with Croft at an Armed Forces Communications and Electronics Association (AFCEA) breakfast. Croft was there to discuss the use of social media and open government for citizen engagement. Video of our interview follows (she discusses Drupal after the second question):

    In a follow-up interview, Croft said that is primarily using proprietary content management systems, but it has already begun a transition toward Drupal. The National Nuclear Security Administration and the Office of Scientific and Technical Information have rebuilt their program sites using Drupal.

    "We did build the Energy Blog in Drupal as it was cheaper and quicker to do so than trying to make the proprietary CMS do something it didn't want to do," said Croft.

    "The Energy Department is not unique here — many departments and agencies across government have a similar variety [of CMS]," she pointed out. "The we are launching in Drupal in the spring will include all of what we're calling our 'phase one sites' — the main site plus program offices completely within the current branding and proprietary CMS. Secondary phases of the project are planned following that initial launch. Our goal is to build a platform that can better serve the public as well as our needs, and can scale over time."

    News that another federal agency is moving to an open source platform is an important data point. As Phase2 Consulting group founder Jeff Walpole wrote at Govfresh, the federal government has been using Drupal for open government initiatives for years. In September, the F.C.C. announced that part of the reboot of would involve using Drupal. The system is famously deployed at as well.

    For those interested in learning more about Drupal, the O'Reilly community has a great guide to getting started.

    June 24 2010

    April 22 2010

    Four short links: 22 April 2010

    1. Whitehouse Released Open Source Code -- four modules for Drupal with features the White House needed, including integration with the Akamai CDN.
    2. Android on iPhone -- it's like constructing an apartment building out of lasagne: an astonishing feat of engineering, even if it's not ultimately useful for anything. (via waxy)
    3. A Practical Guide to Geostatistical Mapping -- covers R, SAGA, Google Earth, and other tools. (via Flowing Data)
    4. Open Data Saves Canada $3.2B -- interesting case of charities fraud, where official institutions were slow to respond but opening the data that revealed the fraud prompted action. Notable to me because the investigators as outsiders didn't have power but the data gave power to the rest of the industry, who had a stake in making sure the fraud was fixed.

    Older posts are this way If this message doesn't go away, click anywhere on the page to continue loading posts.
    Could not load more posts
    Maybe Soup is currently being updated? I'll try again automatically in a few seconds...
    Just a second, loading more posts...
    You've reached the end.

    Don't be the product, buy the product!