Newer posts are loading.
You are at the newest post.
Click here to check if anything new just came in.

March 08 2012

Profile of the Data Journalist: The Hacks Hacker

Around the globe, the bond between data and journalism is growing stronger. In an age of big data, the growing importance of data journalism lies in the ability of its practitioners to provide context, clarity and, perhaps most important, find truth in the expanding amount of digital content in the world. In that context, data journalism has profound importance for society.

To learn more about the people who are doing this work and, in some cases, building the newsroom stack for the 21st century, I conducted a series of email interviews during the 2012 NICAR Conference. This interview followed the conference and featured a remote participant who diligently used social media and the World Wide Web to document and share the best of NICAR:

Chrys Wu (@MacDiva) is a data journalist and user engagement strategist based in New York City. Our interview follows.

Where do you work now? What is a day in your life like?

I work with clients through my company, Matchstrike, which specializes in user engagement strategy. It's a combination of user experience research, design and program planning. Businesses turn to me to figure out how to keep people's attention, create community and tie that back to return on investment.

I also launch Hacks/Hackers chapters around the world and co-organize the group in New York with Al Shaw of ProPublica and Jacqui Cox of The New York Times.

Both things involve seeking out people and ideas, asking questions, reading, wireframing and understanding what motivates people as individuals and as groups.

How did you get started in data journalism? Did you get any special degrees or certificates?

I had a stats class in high school with a really terrific instructor who also happened to be the varsity basketball coach. He was kind of like our John Wooden. Realizing the importance of statistics, being able to organize and interpret data — and learning how to be skeptical of claims (e.g., where "4 out of 5 dentists agree" comes from)— has always stayed with me.

Other than that class and studying journalism at university, what I know has come from exploring (finding what's out there), doing (making something) and working (making something for money). I think that's pretty similar to most journalists and journalist-developers currently in the field.

Though I've spent several years in newsrooms (most notably with the Los Angeles Times and CBS Digital Media Group), most of my journalism and communications career has been as a freelancer. One of my earliest clients specialized in fundraising for Skid Row shelters. I quantified the need cases for her proposals. That involved working closely with the city health and child welfare departments and digging through a lot of data.

Once I figured that out, it was important to balance the data with narrative. Numbers and charts have a much more profound impact on people if they're framed by an idea to latch onto and compelling story to share.

Did you have any mentors? Who? What were the most important resources they shared with you?

I don't have individual mentors, but there's an active community with a huge body of work out there to learn from. It's one of the reasons why I've been collecting things on Delicious and Pinboard, and it's why I try my best to put everything that's taught at NICAR on my blog.

I always try look beyond journalism to see what people are thinking about and doing in other fields. Great ideas can come from everywhere. There are lots of very smart people willing to share what they know.

What does your personal data journalism "stack" look like? What tools could you not live without?

I use Coda and TextMate most often. For wireframing, I'm a big fan of OmniGraffle. I code in Ruby, and a little bit in Python. I'm starting to learn how to use R for dataset manipulation and for its maps library.

For keeping tabs on new but not urgent-to-read material, I use my friend Samuel Clay's RSS reader, Newsblur.

What data journalism project are you the most proud of working on or creating?

I'm most proud of working with the Hacks/Hackers community. Since 2009, we've grown to more than 40 groups worldwide, with each locality bringing journalists, designers and developers together to push what's possible for news.

As I say, talking is good; making is better — and the individual Hacks/Hackers chapters have all done some version of that: presentations, demos, classes and hack days. They're all opportunities to share knowledge, make friends and create new things that help people better understand what's happening around them.

Where do you turn to keep your skills updated or learn new things?

MIT's open courses have been great. There's also blogs, mailing lists, meetups, lectures and conferences. And then there's talking with friends and people they know.

Why are data journalism and "news apps" important, in the context of the contemporary digital environment for information?

I like Amanda Cox's view of the importance of reporting through data. She's a New York Times graphics editor who comes from a statistics background. To paraphrase: Presenting a pile of facts and numbers without directing people toward any avenue of understanding is not useful.

Journalism is fundamentally about fact-finding and opening eyes. One of the best ways to do that, especially when lots of people are affected by something, is to interweave narrative with quantifiable information.

Data journalism and news apps create the lens that shows people the big picture they couldn't see but maybe had a hunch about otherwise. That's important for a greater understanding of the things that matter to us as individuals and as a society.

This interview has been edited and condensed for clarity.

December 12 2011

Can the People's House become a social platform for the people?

Congressional hackathon
InSourceCode developers work on "Madison" with volunteers.

There wasn't a great deal of hacking, at least in the traditional sense, at the "first congressional hackathon." Given the general shiver that the word still evokes in many a Washingtonian in 2011, that might be for the best. The attendees gathered together in the halls of the United States House of Representatives didn't create a more interactive visualization of how laws are made or a mobile health app. As open government advocate Carl Malamud observed, the "hack" felt like something even rarer in the "Age of the App for That:"

In a time when partisanship and legislative gridlock have defined Congress for many citizens, seeing the leadership of the United States House of Representatives agree on the importance of using the power of data and social networking to open government was an early Christmas present.

"Increased access, increased connection with our constituents, transparency, openness is not a partisan issue," said House Majority Leader Eric Cantor.

"The Republican leader and I may debate vigorously on many issues, but one area where we strongly agree is on making Congress more transparent and accessible," said House Democratic Whip Steny Hoyer in his remarks. "First, Congress took steps to open up the Capitol building so citizens can meet with their representatives and see the home of their legislature. In the same way, Congress is now taking steps to update how it connects with the American people online."

An open House

While the event was branded as a "Congressional Facebook Developer Hackathon," what emerged more closely resembled a loosely organized conference or camp.

Facebook executives and developers shared the stage with members of Congress to give keynotes to the 200 or so attendees before everyone broke into discussion groups to talk about constituent communications, press relations and legislative data. The event might be more aptly described as a "wonk-a-thon," as Sunlight Foundation's Daniel Schuman put it last week.

This "hackathon" was organized to have some of the feel of an unconference, in the view of Matt Lira, digital director for the House Majority Leader. Lira sat down for a follow-up interview last Thursday.

"There's a real model to CityCamp," he said. "We had 'curators' for the breakout. Next time, depending on how we structure it, we might break out events that are designed specifically for programming, with others clustered around topics. We want to keep it experimental."

Why? "When Aneesh Chopra and I did that session at SXSW, that personally for me was what tripped my thinking here," said Lira. "We came down from the stage and formed a circle. I was thinking the whole time that it would have been a waste of intellectual talent to have Tim O'Reilly and Clay Shirky in the audience instead of engaging in the conversation. I was thinking I never want to do a panel again. I want it to be like this."

Part of the challenge, so to speak, of Congress hosting a hackathon in the traditional sense, with judging and prizes, lies in procurement rules, said Lira."There are legal issues around challenges or prizes for Congress," he explained. "They're allowed in the executive branch, under DARPA, and now every agency under the COMPETES Act. We can't choose winners or losers, or give out prizes under procurement rules."

Whatever you call it, at the end of the event, discussion leaders from the groups came back and presented on the ideas and concepts that had been hashed out. You can watch a short video that EngageDC produced for the House Majority Leader's office below:

What came out of this unprecedented event, in other words, won't necessarily be measured in lines of code. It's that Congress got geekier. It's that the House is opening its doors to transparency through technology.

Given the focus on Facebook, it's not surprising that social media took center stage in many of the discussions. The idea for it came from a trip to Silicon Valley, where Representative Cantor said he met with Facebook founder Mark Zuckerberg and COO Sheryl Sandberg, and discussed how to make the House more social. After that conversation, Lira and Steve Dwyer, director of online communications and technology for the House Democratic Whip, organized the event.

For a sense of the ideas shared by the working groups, read the story of the first congressional "hackathon" on Storify.

"For government, I don't think we could have done anything more purposeful than this as a first meeting," said Lira in our interview. "Next, we'll focus on building this group of people, strengthening the trust, which will prove instrumental when we get into the pure coding space. I have 100% confidence that we could do a programming-only event now and would have attendance."

A Likeocracy in alpha

As the Sunlight Foundation's John Wonderlich observed earlier this year, access to legislative data brings citizens closer to their representatives.

"When developers and programmers have better access to the data of Congress, they can better build the databases and tools that let the rest of us connect with the legislature," he wrote.

If more open legislative data goes online, when we talk about what's trending in Congress, those conversations will be based upon insight into how the nation is reacting to them on social networks, including Facebook, Twitter, and Google+.

Facebook developers Roddy Lindsay, Tyler Brock, Eric Chaves, Porter Bayne, and Blaise DiPersia coded up a simple proof of concept of what making legislative data might look like. "LikeOcracy" pulls legislation from a House XML feed and makes it more social. The first version added Facebook's ubiquitous "Like" buttons to bill elements. A second version of the app adds more opportunities for reaction by integrating ReadrBoard, which enables users to rate sections or individual lines as "Unnecessary, Problematic, Great Idea or Confusing." You can try it out on three sample bills, including the Stop Online Piracy Act.

Would "social legislation" in a Facebook app catch on? The growth of civic startups like PopVox, OpenCongress and Votizen suggests that the idea has legs. [Disclosure: Tim O'Reilly was an early angel investor in PopVox.]

Likeocracy doesn't tap into Facebook's Open Graph, but it does hint at what integration might look like in the future. Justin Osofsky, Facebook's director of platform partnerships, described how the interests of constituents could be integrated with congressional data under Facebook's new Timeline. Citizens might potentially be able to simply "subscribe" to a bill, much like they can now for any web page, if Facebook's "Subscribe" plug-in was applied to the legislative process.

Opening bill markup online

The other app presented at the hackathon came not from the attendees but from the efforts of InSourceCode, a software development firm that's also coded for Congressman Mike Pence and the Republican National Committee.

Rep. Darrell Issa, chairman of the House Committee on Oversight and Government Reform, introduced the beta version of MADISON on Wednesday, a new online tool to crowdsource legislative markup. The vision is that MADISON will work as a real-time markup engine to let the public comment on bills as they move through the legislative process. "The assumption is that legislation should be open in Congress," said Issa. "It should be posted, interoperable and commented upon."

As Nick Judd reported at techPresident, the first use of MADISON is to host Issa and Sen. Ron Wyden's "OPEN bill," which debuted on the app. Last week, the congressmen released the Online Protection and Enforcement of Digital Trade Act (OPEN) at The OPEN legislation removes one of the most controversial aspects of SOPA, using the domain name system for enforcement, and instead places authority with the International Trade Commission to address enforcement of IP rights on websites that are primarily infringing upon copyright.

Issa said that his team had looked at the use of wikis by Rep. John Culberson, who put the healthcare reform bill online in a wiki. "There are some problems with editors who are not transparent to all of us," said Issa. "That's one of the challenges. We want to make sure that if you're an editor, you're a known editor."

MADISON includes two levels of authentication: email for simple commenting and a more thorough vetting process for organizations or advocacy groups that wish to comment. "Like most things that are a 1.0 or beta, our assumption is that we'll learn from this," said Issa. "Some members may choose to have an active dialog. Others may choose to have it be part of pre-markup record."

Issa fielded a number of questions on Wednesday, including one from web developer Brett Stubbs: "Will there be open access or an API? What we really want is just data." Issa indicated that future versions might include that.

Jayson Manship, the "chief nerd" at InSourceCode, said that MADISON was built in four days. According to Manship, the idea came from conversations with Issa and Seamus Kraft, director of digital strategy for the House Committee on Oversight and Government Reform. MADISON is built with PHP and MySQL, and hosted in RackSpace's cloud so it can scale with demand, said Manship.

"It's important to be entrepreneurial," said Lira in our interview. "There are partners throughout institutions that would be willing to do projects of different sizes and scopes. MADISON is something that Issa and Seamus wanted to do. They took it upon themselves to get the ball rolling. That's the attitude we need."

"We're working to hold the executive accountable to taxpayers," said Kraft last week. "Opening up what we do here in these two halls of Congress is equally important. MADISON is our first shot at it. We're going to need a lot of help to make it better."

Kraft invited the remaining developers present to come to the Rayburn Office Building, where Manship and his team had brought in half a dozen machines, to help get MADISON ready for launch. While I was there, there were conversations about decisions, plug-ins and ideas about improving the interface or functionality, representing a bona fide collaboration to make the app better.

There's a larger philosophical issue relating to open government that Nick Judd touched upon over at techPresident in a follow-up post on MADISON:

The terms for the site warn the user that anything they write on it will become public domain — but the code itself is proprietary. Meanwhile, OpenCongress' David Moore points out that the code that powers his organization's website, which also allows users to comment on individual provisions of bill text, is open source and has been available for some time. In theory, this means the Oversight staff could have started from that code and built on it instead of beginning from scratch. The code being proprietary means that while people like Moore might be able to make suggestions, they can't just download it, make their own changes and submit them for community review — which they'd happily do at little or no cost for a project released under an open-source license.

As Moore put it, "Get that code on GitHub, we'll do OpenID, fix the design."

When asked about whether the team had considered making MADISON code open source, Manship said that "he didn't know, although they weren't opposed to it."

While Moore welcomed MADISON, he also observed that Open Congress has had open-source code for bill text commenting for years.

The decision by Issa's office to fund the creation of an app that was already available as open-source software is one that's worth noting, so I asked Kraft why they didn't fork OpenCongress' code, as Judd suggests. "While there was no specific budget expense for MADISON, it was developed by the Oversight Committee," said Kraft.

"While we like and support OpenCongress' code, it didn't fit the needs for MADISON," Kraft wrote in an emailed statement.

What's next is, so to speak, an "OPEN" question, both in terms of the proposed SOPA alternative and the planned markup of SOPA itself on December 15. The designers of OPEN are actively looking for feedback from the civic software development community, both in terms of what functionality exists now and what could be built in future iterations. as a platform

What Moore and long-time open-government advocates like Carl Malamud want to see from Congress is more structural change:

They're not alone. Dan Schuman listed many other ways the House has yet to catch up with 21st century technology:

We have yet to see bulk access to THOMAS or public access to CRS reports, important legislative and ethics documents are still unavailable in digital format, many committee hearings still are not online, and so on.

As Schuman highlighted, the Sunlight Foundation has been focused on opening up Congress through technology since the organization was founded. To whit: "There have been several previous collaborative efforts by members of the transparency community to outline how the House of Representatives can be more open and accountable, of which an enduring touchstone is the Open House Project Report, issued in May 2007," wrote Schuman.

The notion of making into a platform received high-level endorsement from a congressional leader when House Minority Whip Steny Hoyer remarked on how technology is affecting Congress, his caucus and open government in the executive branch:

For Congress, there is still a lot of work to be done, and we have a duty to make the legislative process as open and accessible as possible. One thing we could do is make — where people go to research legislation from current and previous Congresses — easier to use, and accessible by social media. Imagine if a bill in Congress could tweet its own status.

The data available on should be expanded and made easily accessible by third-party systems. Once this happens, developers, like many of you here today, could use legislative data in innovative ways. This will usher in new public-private partnerships that will empower new entrepreneurs who will, in turn, yield benefits to the public sector.

One successful example is how cities have made public transit data accessible so developers can use it in apps and websites. The end result has been commuters saving time every day and seeing more punctual trains and buses as a result of the transparency. Legislative data is far more complex, but the same principles apply. If we make the information available, I am confident that smart people like you will use it in inventive ways.

Hoyer's specific citation of the growth of open data in cities and an ecosystem of civic applications based upon it is further confirmation that the Gov 2.0 meme is moving into the mainstream.

Making into a platform for bulk data would change what's possible for all civic developers. What I really want is "data on everything," Stubbs told me last week. "THOMAS is just a visual viewer of the internal stuff. If we could have all of this, we could do something with it. What I would like is a data broker. I'd like a RESTful API with all of the data that I could just query. That's what the government could learn from Facebook. From my point of view, I just want to pull information and compile it."

If Hoyer and the House leadership would like to see act as a platform, several attendees at the hackathon suggested to me that Congress could take a specific action: collaborate with the Senate and send the Library of Congress a letter instructing it to provide bulk legislative data access to in structured formats so that developers, designers and citizens around the nation can co-create a better civic experience for everyone.

"The House administration is working on standards called for by the rule and the letter sent earlier this year," said Lira. "We think they will be satisfactory to people. The institutions of the House have been following through since the day they were issued. The first step was issuing an XML feed daily. Next year, there will be a steady series of incremental process improvements. When the House Administrative Committee issues standards, the House Clerk will work on them. "

Despite the abysmal public perception of Congress, genuine institutional changes in the House of Representatives driven by the GOP embracing innovation and transparency are incrementally happening. As Tim O'Reilly observed earlier this year, the current leadership of the House on transparency is doing a better job than their predecessors.

In April, Speaker Boehner and Majority Leader Cantor sent a letter to the House Clerk regarding legislative data release. Then, in September, a live XML feed for the House floor went online. Yes, there's a long way to go on open legislative data quality in Congress. That said, there's support for open-government data from both the White House and the House.

"My personal view is that what's important right now is that the House create the right precedents," said Lira. "If we create or adopt a data standard, it's important that it be the right standard."

Even if open government is in beta, there needs to be more tolerance for experiments and risks, said Lira. "I made a mistake in attacking We the People as insufficient. I still believe it is, but it's important to realize that the precedent is as important as the product in government. In technology in general, you'll never reach an end. We The People is a really good precedent, and I look forward to seeing what they do. They've shown a real commitment, and it's steadily improving."

A social Congress

While Sean Parker may predict that social media will determine the outcome of the 2012 election, governance is another story entirely. Meaningful use of social media by Congress remains challenged by a number of factors, not least an online identity ecosystem that has not provided Congress with ideal means to identify constituents online. The reality remains that when it comes to which channels influence Congress, in-person visits and individual emails or phone calls are far more influential with congressional staffers.

As with any set of tools, success shouldn't be measured solely by media reports or press releases but by the outcomes from their use. The hard work of bipartisan compromise between the White House and Congress, to the extent it occurs, might seem unlikely to be publicly visible in 140 characters or less.

"People think it's always an argument in Washington," said Lira in our interview. "Social media can change that. We're seeing a decentralization of audiences that is built around their interests rather than the interests of editors. Imagine when you start streaming every hearing and making information more digestible. All of a sudden, you get these niche audiences. They're not enough to sustain a network, but you'll get enough of an audience to sustain the topic. I believe we will have a more engaged citizenry as a result."

Lira is optimistic. "Technology enables our republic to function better. In ancient Greece, you could only sustain a democracy in the size of city. Transportation technology limited that scope. In the U.S., new technologies enabled global democracy. As we entered the age of mass communication, we lost mass participation. Now with the Internet, we can have people more engaged again."

There may be a 30-year cycle at play here. Lira suggested looking back to radio in the 1920s, television in the 1950s, and cable in the 1980s. "It hasn't changed much since; we're essentially using the same rulebook since the '80s. The changes made in those periods of modernization were unique."

Thirty years on from the introduction of cable news, will the Internet help reinvigorate the founders' vision of a nation of, by and with the people? "I do think that this is a transformational moment," said Lira. "It will be for the next couple of years. When you talk to people — both Republicans and Democrats — you sense we're on the cusp of some kind of change, where it's not just communicating about projects but making projects better. Hearings, legislative government and executive government will all be much more participatory a decade from now. "

In that sweep of history, the "People's House" may prove to be a fulcrum of change. "If any place in government is going to do it, it's the House" said Lira. "It's our job to be close to the public in a way that no other part of government is. In the Federalist Papers, that's the role of the House. We have an obligation to lead the way in terms of incorporating technology into real processes. We're not replacing our system of representative government. We're augmenting it with what's now possible, like when the telegraph let people know what the votes were faster."

Strata 2012 — The 2012 Strata Conference, being held Feb. 28-March 1 in Santa Clara, Calif., will offer three full days of hands-on data training and information-rich sessions. Strata brings together the people, tools, and technologies you need to make data work.

Save 20% on registration with the code RADAR20
Sponsored post

December 01 2011

Gov 2.0 enters the mainstream on NPR and the AP

Regular Radar readers know that "Gov 2.0 has gone local," as local governments look for innovative ways to use technology cooperatively with citizens to deliver smarter government. This week, NPR listeners learned more about the open-government movement around the country when the Kojo Nnamdi Show hosted an hour-long discussion on local Gov 2.0 on WAMU in Washington, D.C.

You can listen to the audio archive of the program and read the transcript at

I was happy to join Bryan Sivak, chief innovation officer of the state of Maryland; Tom Lee, director of Sunlight Labs; and Abhi Nemani, director of strategy and communications at Code For America, as a guest on the show.

Heather Mizeur, a delegate to the Maryland State Assembly, called in to the show to share what her state has been working on with respect to open government, including streaming video, budget transparency and online access. Mizeur had the one-liner of the day: Commenting on the need to improve, she observed that "our state website is an eight-track tape player in an iPhone universe."

An open government linkology

During the program, the @KojoShow producer shared links to sites and services that were mentioned by the guests. These included:

  • Maryland's solicitation for feedback on helpful or hurtful business regulations at
  • An NPR News feature on the American Legislative Exchange Council and channels of influence in state legislatures.
  • Churnalism, an app to discover PR masquerading as original journalism. Could a churnalism model be used to detect similar subtle influences in state legislatures? Sunlight Labs has an ongoing project at OpenStates. Stay tuned.
  • Civic engagement platform Change By Us launched in Philadelphia and was open sourced into the Civic Commons.
  • Sivak cited as a model for well designed government websites. The key is that it's adapted for mobile visitors.
  • Nemani cited Open Data Philly as a local open government platform that uses open standards. It's open sourced, so that other cities, like Chattanooga, Tenn., can use it to stand up their own open-data efforts.
  • The Sportaneous location-aware mobile app uses open-government data to help people find pick-up sports games.
  • The StreetBump app uses a smartphone's accelerometer to automatically report potholes in Boston.
Moving to Big Data: Free Strata Online Conference — In this free online event, being held Dec. 7, 2011, at 9AM Pacific, we'll look at how big data stacks and analytical approaches are gradually finding their way into organizations as well as the roadblocks that can thwart efforts to become more data driven. (This Strata Online Conference is sponsored by Microsoft.)

Register to attend this free Strata Online Conference

Civic applications enter the mainstream

Recently, civic applications and open data pushed further into the national consciousness with a widely syndicated Associated Press story by Marcus Wohlsen. Here's how Wohlsen described what's happening:

Across the country, geeks are using mountains of data that city officials are dumping on the web to create everything from smartphone tree identifiers and street sweeper alarms to neighborhood crime notifiers and apps that sound the alarm when customers enter a restaurant that got low marks on a recent inspection. The emergence of city apps comes as a result of the rise of the open-data movement in U.S. cities, or what advocates like to call "Government 2.0."

The AP covered Gov 2.0 and the open-government data movement in February, when it looked at how cities were crowdsourcing ideas from citizens, or "citizensourcing."

It's great to see what's happening around the country receive more mainstream attention. Over on Google+, Tim O'Reilly commented on the AP story:

Of all the things that made up the "gov 2.0" meme, open data may be one of the most important. It's a key part of government thinking like a platform player rather than an application provider. At Code for America, the work ended up being about liberating data as much as about writing apps. We're just at the beginning of a really interesting new approach to government services.

Wohlsen captured the paradigm behind Gov 2.0 at the end of his article:

New York, San Francisco and other cities are now working together to develop data standards that will make it possible for apps to interact with data from any city. The idea, advocates of open data say, is to transform government from a centralized provider of services into a platform on which citizens can build their own tools to make government work better.

Open311 is a data standard of this sort. So is GTFS. "So much can flow from so little," noted O'Reilly. "Consider how Google Transit began with outreach from the city of Portland to create GTFS, a standard format for transit data, which was subsequently adopted by other cities. Now, you can get transit arrival times from Google as well as from hundreds of smartphone apps, none of which needed to be written by city government."

What lies ahead for Gov 2.0 in 2012 has the potential to improve civic life in any number of interesting ways. If the Gov 2.0 movement is to have a lasting, transformative effect, however, what's described above needs to be the beginning of the story, not the end. That arc will include the results of HHS CTO Todd Park's efforts to revolutionize the healthcare industry or the work of the Alfred brothers at BrightScope to bring more transparency to financial advisors.

Making Gov 2.0 matter will also mean applying different ways of thinking and new technology to other areas, as FutureGov founder Dominic Campbell commented on Google+:

There aren't enough of us working to transform, challenge and change the inside of government. Not enough taking on the really sticky issues beyond relatively quick and easy wins, such as transit data or street-scene related apps. This needs to change before anything can be said to have gone mainstream. Disclaimer: this is exactly what we're looking to do with apps like PatchWorkHQ and CasseroleHQ, starting to hone in on priority, challenging, socially important and costly areas of government, such as child protection and supporting older people to live better independent lives. The journey is far longer and harder, but (we're hoping) even more rewarding.

More awareness of what's possible and available will lead to more use of civic applications and thereby interest and demand for open-government data. For instance, on the AP's Twitter feed, an editor asked more than 634,000 followers this question: "Hundreds of new apps use public data from cities to improve services. Have you tried any?" I'll ask the same of Radar readers: have you used a civic app? If so, what and where? Did it work? Did you keep it? Please let us know in the comments.

July 31 2011

App outreach and sustainability: lessons learned by Portland, Oregon

Having decided to hang around Portland for a couple days after the Open Source convention, I attended a hackathon sponsored by the City of Portland and a number of local high tech companies, and talked to Rick Nixon (program manager for technology initiatives in the Portland city government) about the two big problems faced by contests and challenges in government apps: encouraging developers to turn their cool apps into sustainable products, and getting the public to use them.

It's now widely recognized that most of the apps produced by government challenges are quickly abandoned. None of the apps that won awards at the original government challenge--Vivek Kundra's celebrated Apps for Democracy contest in Washington, DC--still exist.

Correction: Alex Howard tells me one of the Apps for Democracy
winners is still in use, and points out that other cities have found
strategies for sustainability.

And how could one expect a developer to put in the time to maintain an app, much less turn it into a robust, broadly useful tool for the general public? Productizing software requires a major investment. User interface design is a skill all its own, databases have to be maintained, APIs require documentation that nobody enjoys writing, and so forth. (Customer service is yet another burden--one that Nixon finds himself taking on for apps developed by private individuals for the city of Portland.) Developers quit their day jobs when they decide to pursue interesting products. The payoff for something in the public sphere just isn't there.

If a government's goal is just to let the commercial world know that a data set is available, a challenge may be just the thing to do, even if no direct long-term applications emerge. But as Nixon pointed out, award ceremonies create a very short blip in the public attention. Governments and private foundations may soon decide that the money sunk into challenges and awards is money wasted--especially as the number of challenges proliferate, as I've seen them do in the field of health.

Because traditional incentives can never bulk up enough muscle to make it worthwhile for a developer to productize a government app, the governments can try taking the exact opposite approach and require any winning app to be open source. That's what Portland's CivicApps does. Nixon says they also require a winning developer to offer the app online for at least a year after the contest. This gives the app time to gain some traction.

Because nearly any app that's useful to one government is useful to many, open source should make support a trivial problem. For instance, take Portland's city council agenda API, which lets programmers issue queries like "show me the votes on item 506" or "what was the disposition of item 95?" On the front end, a city developer named Oscar Godson created a nice wizard, with features such as prepopulated fields and picklists, that lets staff quickly create agendas. The data format for storing agendas is JSON and the API is so simple that I started retrieving fields in 5 minutes of Ruby coding. And at the session introducing the API, several people suggested enhancements. (I suggested a diff facility and a search facility, and someone else suggested that session times be coded in standard formats so that people could plan when to arrive.) Why couldn't hundreds of governments chip in to support such a project?

Code for America, a public service organization for programmers supported by O'Reilly and many other institutions, combines a variety of strategies. All projects are open source, but developers are hooked up with projects for a long enough period to achieve real development milestones. But there may still be a role for the macho theatrics of a one-day hackathon or short-term challenge.

Enhancing the platform available to developers can also stimulate more apps. Nixon pointed out that, when Portland first released geographic data in the form of Shapefiles, a local developer created a site to serve them up more easily via an API, mobilizing others to create more apps. He is now part of the Code For America effort doing exactly the same thing--serving up geographic data--for other large municipalities.

Public acceptance is the other big problem. A few apps hit the big time, notably the Portland PDX bus app that tells you how soon a bus is coming so you can minimize the time you wait out in the rain. But most remain unknown and unappreciated. Nixon and I saw no way forward here, except perhaps that one must lead the way with increasing public involvement in government, and that this involvement will result in an increased use of software that facilitates it.

The wealth of simple APIs made a lot of people productive today. The applications presented at the end of the Portland hackathon were:

  • A mapping program that shows how much one's friends know each other, clustering people together who know each other well

  • An information retrieval program that organizes movies to help you find one to watch

  • A natural language processing application that finds and displays activities related to a particular location

  • An event planner that lets you combine the users of many different social networks, as well as email and text messaging users (grand prize winner)

  • A JSON parser written in Lua communicating with a GTK user interface written in Scheme (just for the exercise)

  • A popularity sorter for the city council agenda, basing popularity on the number of comments posted

  • A JavaScript implementation of LinkedIn Circles

  • A geographic display of local institutions matching a search string, using the Twilio API

  • A visualization of votes among city council members

  • An aggregator for likes and comments on Facebook and (eventually) other sites

  • A resume generator using LinkedIn data

  • A tool for generating consistent location names for different parts of the world that call things by different terms

Approximately 130 man-and-woman hours went into today's achievements. A project like Code for America multiplies that by hundreds.

Older posts are this way If this message doesn't go away, click anywhere on the page to continue loading posts.
Could not load more posts
Maybe Soup is currently being updated? I'll try again automatically in a few seconds...
Just a second, loading more posts...
You've reached the end.
No Soup for you

Don't be the product, buy the product!

YES, I want to SOUP ●UP for ...