Newer posts are loading.
You are at the newest post.
Click here to check if anything new just came in.

March 08 2012

Profile of the Data Journalist: The Hacks Hacker

Around the globe, the bond between data and journalism is growing stronger. In an age of big data, the growing importance of data journalism lies in the ability of its practitioners to provide context, clarity and, perhaps most important, find truth in the expanding amount of digital content in the world. In that context, data journalism has profound importance for society.

To learn more about the people who are doing this work and, in some cases, building the newsroom stack for the 21st century, I conducted a series of email interviews during the 2012 NICAR Conference. This interview followed the conference and featured a remote participant who diligently used social media and the World Wide Web to document and share the best of NICAR:

Chrys Wu (@MacDiva) is a data journalist and user engagement strategist based in New York City. Our interview follows.

Where do you work now? What is a day in your life like?

I work with clients through my company, Matchstrike, which specializes in user engagement strategy. It's a combination of user experience research, design and program planning. Businesses turn to me to figure out how to keep people's attention, create community and tie that back to return on investment.

I also launch Hacks/Hackers chapters around the world and co-organize the group in New York with Al Shaw of ProPublica and Jacqui Cox of The New York Times.

Both things involve seeking out people and ideas, asking questions, reading, wireframing and understanding what motivates people as individuals and as groups.

How did you get started in data journalism? Did you get any special degrees or certificates?

I had a stats class in high school with a really terrific instructor who also happened to be the varsity basketball coach. He was kind of like our John Wooden. Realizing the importance of statistics, being able to organize and interpret data — and learning how to be skeptical of claims (e.g., where "4 out of 5 dentists agree" comes from)— has always stayed with me.

Other than that class and studying journalism at university, what I know has come from exploring (finding what's out there), doing (making something) and working (making something for money). I think that's pretty similar to most journalists and journalist-developers currently in the field.

Though I've spent several years in newsrooms (most notably with the Los Angeles Times and CBS Digital Media Group), most of my journalism and communications career has been as a freelancer. One of my earliest clients specialized in fundraising for Skid Row shelters. I quantified the need cases for her proposals. That involved working closely with the city health and child welfare departments and digging through a lot of data.

Once I figured that out, it was important to balance the data with narrative. Numbers and charts have a much more profound impact on people if they're framed by an idea to latch onto and compelling story to share.

Did you have any mentors? Who? What were the most important resources they shared with you?

I don't have individual mentors, but there's an active community with a huge body of work out there to learn from. It's one of the reasons why I've been collecting things on Delicious and Pinboard, and it's why I try my best to put everything that's taught at NICAR on my blog.

I always try look beyond journalism to see what people are thinking about and doing in other fields. Great ideas can come from everywhere. There are lots of very smart people willing to share what they know.

What does your personal data journalism "stack" look like? What tools could you not live without?

I use Coda and TextMate most often. For wireframing, I'm a big fan of OmniGraffle. I code in Ruby, and a little bit in Python. I'm starting to learn how to use R for dataset manipulation and for its maps library.

For keeping tabs on new but not urgent-to-read material, I use my friend Samuel Clay's RSS reader, Newsblur.

What data journalism project are you the most proud of working on or creating?

I'm most proud of working with the Hacks/Hackers community. Since 2009, we've grown to more than 40 groups worldwide, with each locality bringing journalists, designers and developers together to push what's possible for news.

As I say, talking is good; making is better — and the individual Hacks/Hackers chapters have all done some version of that: presentations, demos, classes and hack days. They're all opportunities to share knowledge, make friends and create new things that help people better understand what's happening around them.

Where do you turn to keep your skills updated or learn new things?

MIT's open courses have been great. There's also blogs, mailing lists, meetups, lectures and conferences. And then there's talking with friends and people they know.

Why are data journalism and "news apps" important, in the context of the contemporary digital environment for information?

I like Amanda Cox's view of the importance of reporting through data. She's a New York Times graphics editor who comes from a statistics background. To paraphrase: Presenting a pile of facts and numbers without directing people toward any avenue of understanding is not useful.

Journalism is fundamentally about fact-finding and opening eyes. One of the best ways to do that, especially when lots of people are affected by something, is to interweave narrative with quantifiable information.

Data journalism and news apps create the lens that shows people the big picture they couldn't see but maybe had a hunch about otherwise. That's important for a greater understanding of the things that matter to us as individuals and as a society.

This interview has been edited and condensed for clarity.

March 06 2012

Profile of the Data Journalist: The Data Editor

Around the globe, the bond between data and journalism is growing stronger. In an age of big data, the growing importance of data journalism lies in the ability of its practitioners to provide context, clarity and, perhaps most important, find truth in the expanding amount of digital content in the world. In that context, data journalism has profound importance for society.

To learn more about the people who are doing this work and, in some cases, building the newsroom stack for the 21st century, I conducted a series of email interviews during the 2012 NICAR Conference.

Meghan Hoyer (@MeghanHoyer) is a data editor based in Virginia. Our interview follows.

Where do you work now? What is a day in your life like?

I work in an office within The Virginian Pilot’s newsroom. I’m a one-person team, so there’s no such thing as typical.

What I might do: Help a reporter pull Census data, work with IT on improving our online crime report app, create a DataTable of city property assessment changes, and plan training for a group of co-workers who’d like to grow their online skills. At least, that’s what I’m doing today.

Tomorrow, it’ll be helping with our online election report, planning a strategy to clean a dirty database, and working with a reporter to crunch data for a crime trend story.

How did you get started in data journalism? Did you get any special degrees or certificates?

I have a journalism degree from Northwestern, but I got started the same way most reporters probably got started - I had questions about my community and I wanted quantifiable answers. How had the voting population in a booming suburb changed? Who was the region’s worst landlord? Were our localities going after delinquent taxpayers? Anecdotes are nice, but it’s an amazingly powerful thing to be able to get the true measure of a situation. Numbers and analysis help provide a better focus - and sometimes, they upend entirely your initial theory.

Did you have any mentors? Who? What were the most important resources they shared with you?

I haven’t collected a singular mentor as much as a group of people whose work I keep tabs on, for inspiration and follow-up. The news community is pretty small. A lot of people have offered suggestions, guidance, cheat sheets and help over the years. Data journalism - from analysis to building apps -- is definitely not something you can or need to learn in a bubble all on your own.

What does your personal data journalism "stack" look like? What tools could you not live without?

In terms of daily tools, I keep it basic: Google docs, Fusion Tables and Refine, QGIS, SQLite and Excel are all in use pretty much every day.

I’ve learned some Python and JavaScript for specific projects and to automate some of the newsroom’s daily tasks, but I definitely don’t have the programming or technical background that a lot of people in this field have. That’s left me trying to learn as much as I can as quick as I can.

In terms of a data stack, we keep information such as public employee salaries, land assessment databases and court record databases (among others) updated in a shared drive in our newsroom. It’s amazing how often reporters use them, even if it’s just to find out which properties a candidate owns or how long a police officer caught at a DUI checkpoint has been on the force.

What data journalism project are you the most proud of working on or creating?

A few years ago, I combined property ownership records, code enforcement citations, real estate tax records and rental inspection information from all our local cities and found a company with hundreds of derelict properties.

Their properties seemed to change hands often, so a partner and I then hand-built a database from thousands of land deeds that proved the company was flipping houses among investors in a $26 million mortgage fraud scheme. None of the cities in our region had any idea this was going on because they were dealing with each parcel as a separate entity.

That’s what combining sets of data can get you - a better overall view of what’s really happening. While government agencies are great at collecting piles of data, it’s that kind of larger analysis that’s missing.

Where do you turn to keep your skills updated or learn new things?

To be honest - Twitter. I get a lot of ideas and updates on new tools there. And the NICAR conference and listserv. Usually when you hit up against a problem - whether it’s dealing with a dirty dataset or figuring out how to best visualize your data -- it’s something that someone else has already faced.

I also learn a lot from the people within our newsroom. We have a talented group of web producers who all are eager to try new things and learn.

Why are data journalism and "news apps" important, in the context of the contemporary digital environment for information?

Data is everywhere, but in most cases it’s just stockpiled and warehoused without a second thought to analysis or using it to solve larger problems.

Journalists are in a unique position to make sense of it, to find the stories in it, to make sure that governments and organizations are considering the larger picture.

I think, too, that people in our field need to truly push for open government in the sense not of government building interfaces for data, but for just releasing raw data streams. Government is still far too stuck in the “Here’s a PDF of a spreadsheet” mentality. That doesn’t create informed citizens, and it doesn’t lead to innovative ways of thinking about government.

I’ve been involved recently in a community effort to create an API and then apps out of the regional transit authority’s live bus GPS stream. It has been a really fun project - and something that I hope other local governments in our area take note of.

Sponsored post

August 30 2011

How to create sustainable open data projects with purpose

mySocietyThere has been much hand-wringing of late about whether the explosion of government-run app contests over the last couple of years has generated any real value for the public. With only one of the Apps for Democracy projects still running, it's easy to see the entire movement being written off as an overly optimistic fad.

The organisation that I'm lucky enough to lead — mySociety — didn't come from the world of app contests, but it does build the kind of open-source, open-data-grounded civic apps that such contests are suppose to produce. I believe that mySociety's story shows that it's possible to build meaningful, impactful civic and democratic web apps, to grow them to a scale where they're unambiguously a good use of time and money, then sustain them for years at a time. Right now we're launching a new site, FixMyTransport, that is trying to try to raise the bar for the ambition and scale of civic apps, so this seems a good moment to share some thoughts about what it takes to build good services and get them to last more than a few months.

You have to be just as focused on user needs as any company (and perhaps more so)

People have needs. Sometimes they need to eat, sometimes they need to sleep. And sometimes they need to send an urgent message to a local politician, or get a dangerous hanging branch cleared off of a road.

What people never, ever do is wake up thinking, "Today I need to do something civic," or, "Today I will explore some interesting data via an attractive visualisation." MySociety has always been unashamed about packaging civic services in a way that appeals directly to real people with real, everyday needs. I gleefully delete the two or three emails a year that land in our inbox suggesting that FixMyStreet should be renamed to FixOurStreet. No, dude, when I'm pissed it's definitely my street, which is why people have borrowed the name around the world.

We learned this lesson most vividly from Pledgebank, a sputtering site with occasional amazing successes and lots and lots of "meh." The reason it never took off was because, unlike the later (and brilliant) Kickstarter, we didn't make it specific enough. We didn't say "use this site to raise money for your first album," or "use this site to organise a march." We said it was a platform for "getting things done," and the users walked away in confusion. That's why our new site is called FixMyTransport, even though it's actually the first instance of a general civic-problem-fixing platform that could handle nearly any kind of local campaigning.

Being focused on user needs means not starting things you think you probably can't finish

In mySociety's history we have run four calls for proposals, asking the whole world what we should build next. Like most idea gathering processes, there's about 100 bad ideas for every good one, but the bad ideas have value in that they reveal a habitual digital era trait — being insanely optimistic about the effort required to build things to a high standard.

Now, clearly, I'm not saying it is impossible to hack brilliant things without piles of VC gold. But if you are going to hack something really, genuinely valuable in just a couple of weeks, and you want it to thrive and survive in the real Internet, you need to have an idea that is as simple as it is brilliant. Matthew Somerville's accessible Traintimes fits into this category, as does, and But ideas like this are super rare — they're so simple and powerful that really polished sites can be built and sustained on volunteer-level time contributions. I salute the geniuses who gave us the four sites I just mentioned. They make me feel small and stupid.

If your civic hack idea is more complicated than this, then you should really go hunting for funding before you set about coding. Because the Internet is a savagely competitive place, and if your site isn't pretty spanking, nobody is going to come except the robots and spammers.

To be clear — FixMyTransport is not an example of a super-simple genius idea. I wish it were. Rather it's our response to the questions "What's missing in the civic web?" and "What's still too hard to get done online?" But we didn't start building it until we knew we had the money, and we didn't try to fit it into evenings and weekends. It was painful to wait and not rush with it, but it was the right thing to do to build something up to the expectations of an Internet-using public habituated to websites with billion-dollar budgets. And we are emotionally and financially prepared for the six months of rapid iteration that will follow once the public arrives.

Strata Conference New York 2011, being held Sept. 22-23, covers the latest and best tools and technologies for data science — from gathering, cleaning, analyzing, and storing data to communicating data intelligence effectively.

Save 30% on registration with the code ORM30

Data is your servant, not your master

I love open data. I love structured data. I love data, full stop. But my love of data is not the same as respecting our users' needs. There are more than 300,000 bus stops, train stations, ferry routes and so on in the FixMyTransport back end, munged together over months of hard work from dirty, dirty public data sources. Can you see any sign of this on the homepage? No sir, because users want to fix transport problems, not revel in our mastery of databases.

Demand fewer, larger grants from government and funders

MySociety got lucky. It was born into a period of high public spending, 2003/4, and its second ever grant was for 0.02% of a government funding pot worth more than a billion dollars — about a quarter of a million dollars. It was amazing luck for a small organisation with no track record, possible only because so much money was being thrown around. Those days are gone on both sides of the Pond, but governments everywhere should note that that funding of this scale got us right through our first couple of years, until sites like WriteToThem were mature and had proved their public value (and picked up an award or two).

In the subsequent few years, we saw the "thousand flowers bloom" mentality really take over the world of public-good digital funding, and we saw it go way beyond what was sensible. Time and again, we'd see two good ideas get funding and eight bad ones at the same time because of the sense that it was necessary to spread the money around. It would be great if someone could make the case to public grant funders that good tech ideas — and the teams that can implement them — are vanishingly rare. There is nothing to be ashamed about dividing the pot up two or three ways if there are only a few ideas or proposals or hacks that justify the money. The larger amounts this would produce wouldn't mean champagne parties for grantees, it would mean the best ideas surviving long enough to grow meaningful traffic and learn how to make money other ways.

After a long road supported by public grant funding, mySociety is now 50% commercially funded and 50% private-grant funded, but we'd never have arrived there without being 100% public-grant funded for the first couple of years. Now our key donors are philanthropic, with Indigo Trust in particular covering most of the core development cost for FixMyTransport.

Respect the geeks

All great technology projects have one or more über geeks at the heart of them. If you find the right über geeks, they'll understand politics, society and users just as much as they understand their code. If you find someone as ferociously multi-talented as, say, Louise Crow, who built FixMyTransport almost single-handedly, listen to them and change your plans when they say "no." Luckily, she said "yes" to building this project, and I hope those of you who care about civic tech give her the props appropriate to building something on this scale. Respect her, and respect the geeks like her, and you'll be one step closer to civic app success.


June 02 2011

Civic Commons taps tech to make government work better and cost less

Civic CommonsLast September, an ambitious code-sharing initiative named Civic Commons was launched at the Gov 2.0 Summit in a bid to help city governments use information technology better. This week, Civic Commons took a big step forward with a new management team in place and $250,000 of funding from Omidyar Network.

Former White House deputy CTO Andrew McLaughlin will be the first executive director and Nick Grossman, former director of Civic Works at nonprofit Open Plans, will be its first managing director. Grossman was one of the lead architects of Civic Commons from its inception.

The benefits of adopting a Gov 2.0 approach in cities include improved e-services to the public, resident participation in government, and collaboration between agencies. That snapshot of Gov 2.0 evolution offers ample perspective on the challenges for Gov 2.0 at the federal level.

Cities like New York, Boston, San Francisco, Portland, Ore. and the District of Columbia, have all been hailed in the media for innovative uses of open data, new urban mechanics, adoption of Open311, and improved e-services. Chicago's new chief technology officer, John Tolva, recently shared a vision of a digital second city that would embrace open data and innovation. "Forget thinking about computers or even websites. We want to think of the city itself as a platform for interaction — as a computing platform," Tolva told the Chicago Sun-Times. "There are makers, builders and developers who see the city as a new domain, as a new problem set," he said. "How might we become as digitally literate a place as we are architecturally literate?"

Not all city governments are as forward thinking or visionary in their approach. Most have significant cost constraints or well earned cynicism about the utility of bloated IT projects. While promoting government transparency through technology is a leading topic of interest for local government officials, implementation still lags that interest in many counties.

The architects of Civic Commons are well aware of this context. In the video below, McLaughlin delivers a lightning talk at the 2011 Transparency Camp in Washington, D.C., where he talks about Civic Commons, Code for America, and the challenges that face cities around the world.

In his first post at the Civic Commons blog, McLaughlin reflected further upon the core principles that the nonprofit is built upon:

In sum, Civic Commons is built around two central convictions: first, that wave after wave of innovation is delivering amazing new capabilities to the people and organizations that can take advantage of them, and second, that, with a little help, governments can absolutely understand and seize the opportunities created by the rapid evolution of information technology.

So how are we going to do it? Civic Commons will operate as a neutral and expert non-profit that (a) helps cities and other governments understand the possibilities and pitfalls around shared technologies, (b) provides technical assistance, (c) facilitates the creation and management of collaborative technology projects, (d) connects interested cities with peers, collaborators, experts, vendors, and other supporters, and (e) creates high-quality information — such as guides, checklists, how-to's, and a comprehensive catalog of civic technology — that is as comprehensible and useful to mayors, city managers, and citizens as it is to software engineers.

OSCON 2011 — Join today's open source innovators, builders, and pioneers July 25-29 as they gather at the Oregon Convention Center in Portland, Ore.

Save 20% on registration with the code OS11RAD

McLaughlin also wrote in to Radar in response to questions about his plans for Civic Commons and what civic codesharing could mean for cities. His email interview follows.

What attracted to you to working for Civic Commons after your experiences in law, at ICANN, Google and the White House?

Andrew McLaughlinAndrew McLaughlin: The thread that runs through my grown-up work life, starting when I was a junior lawyer on the legal team that challenged the insidious Communications Decency Act in 1996, is that I get highly motivated — irrationally exuberant, according to my wife — as I marvel at the ability of the Internet to shift power into the hands of individuals. As computers keep getting cheaper and more beefy, and connectivity gets more and more ubiquitous, the Internet enables individuals to publish and communicate and create and analyze information on a global scale. As we've seen on a spectacular scale in North Africa this year, information is power; and so the Internet's potential to deliver meaningful, practical, useful empowerment to individuals around the world is incredible to me. The Internet can make things work better and cheaper, but it can also change the balance of power between active individuals and remote, unresponsive institutions.

So then at ICANN what motivated me was the objective of keeping the Internet's distributed, decentralized infrastructure out of the hands of potentially restrictive governments. At Google I was motivated to protect freedom of expression from governmental censorship. And at the White House, I was motivated by the potential to transform government agencies from the inside through the embrace of open data, Silicon Valley-style technology platforms, and more nimble organizational management techniques. My White House experience was fantastic, but sobering: federal agencies, for a host of reasons, are very difficult organizations to change.

With Civic Commons, I'm motivated by the observation that cities are noticeably eager to put technology to work to improve public services, transparency, accountability, public participation, and organizational effectiveness. They also want to spend much less money on technology. And it's possible. The same dynamics that have driven amazing, free or low-cost products and services (Facebook, iPads, webmail, Skype, online docs, smart phones, etc.) can be applied to drive transformative changes and improvements in the ways cities, towns, counties, and state agencies do their work. That's my ambition for Civic Commons, and for myself.

Why do you think Civic Commons could make a difference? What problems is it trying to solve?

Andrew McLaughlin: If we do our job right, Civic Commons will help cities (and towns and counties and state agencies — I'm just going to say "cities", though) collaborate with each other to build and use technologies that improve their services at lower cost. Currently, when it comes to technology, cities function in isolation, reinventing (or re-procuring) the wheel with every costly, customized technology purchase they make. But cities all basically face the same problems, their citizens have the same needs, and they're trying to provide the same types of services. So it would be smart for them, in some areas, to pool their resources (talents and budgets) to build technologies in common, which can then be shared and reused globally. The beauty of this approach is that it works — we know it works, because it works in other areas. Just take a look at the various open source communities, especially Kuali, which is a consortium of higher-education institutions that work together to build shared technology systems (e.g., finance, HR, curriculum management) for universities and colleges.

How does open source relate to the issues that cities face in 2011? What are the biggest barriers to adoption or implementation?

Andrew McLaughlin: Open source isn't the answer to every city tech need. But in some key areas of widely shared demand, open source development can provide the best path to new, low-cost, high-reliability, high-security options for cities. For example, if dozens of cities work together to build, tune, and polish a comprehensive, secure, and Open 311 system for citizen interactions (which is exactly what the Open311 community is doing), they will get a vastly better system for vastly less money than if each city did its own procurement in isolation. The biggest barriers to adoption and implementation are pretty predictable: cities don't have experience in working together on tech projects; they often lack in-house technical expertise; and they many not even know that open source options and projects exist.

There have been other attempts to create code repositories. What's wrong with using GitHub or SourceForge? Why does there need to be a nonprofit?

Andrew McLaughlin: We don't feel a driving need to create a new code repository. GitHub and SourceForge are both terrific. We're going to keep talking with city CIOs and technologists to see whether their projects have peculiar needs that counsel in favor of a civic tech repository. We've heard mixed input on that so far.

One thing we are going to build is a comprehensive catalog/index/guide to all the civic technologies in use or under construction — all the technology options for management, operations/finance, citizen engagement, open data and analytics, resource planning, etc. We want to aim a living database that enables non-technical mayors, city managers, citizen activists, to see what's out there, what's being built, how to use it, where to go for support or advice or implementation, and so on. That doesn't seem to exist anywhere right now, and we think there's a huge need (and potential demand) for it.

What are the biggest challenges ahead for Civic Commons?

One big challenge is to keep our open source code projects vibrant, with active ownership and online participation by a real community. Another is to persuade political and civil service leadership that our pitch is realistic, and that supporting their technologists to work on collaborative or shared technology projects is in their self interest.

How can the Gov 2.0 and open source community help?

There are many ways to help: contributing to specific projects like Open311; helping us scope out and foster new projects; contributing to our wiki and catalog (when it's up); joining our mailing list; introducing us to civic leaders who might be interested in building or using shared technologies.


March 08 2011

Citizensourcing smarter government in New York City

NYC 311 map"Even the biggest cities have small towns buried within them." When Steven B. Johnson made that observation in his Wired article on what 311 calls can tell us about New York City, he referred to an unexpected connection that Mayor Bloomberg made with a former colleague in a city call center. As Gov 2.0 goes local, the neighborhoods of New York City will be a prime place to watch this year.

Similar connections and reconnections, made intentionally or serendipitously through social networks, phones and in-person meetings, are part of the warp and weft of the fabric of civil society in every major city in the United States. When it comes to big cities, few have the energy and dynamism of Gotham, with its ceaseless activity, millions of citizens and global heft in the financial, media, fashion and entertainment industries.

Whether cities like New York can evolve to embrace more collaborative government isn't entirely a matter of choice, given grim budget pressures and citizens demanding better services. In the face of those challenges, the NYC mayor's office is choosing to pursue an enlightened strategy: using modern technology and the collective wisdom of citizens to find better ways of getting things done. In an interview last month, Steven Goldsmith, the deputy mayor of New York City, talked with me about open government, crowdsourcing, data and much more.

"I was involved in e-government decades ago," said Goldsmith, the former mayor of Indianapolis. "E-government tools that feel really well constructed do make it easier for citizens." The opportunities that technology affords government today go beyond traditional e-government, however, as Goldsmith explained. He suggested thinking of it in terms of version numbers:

  • 0.5 is putting information online.
  • 1.0 is an electronic way to fill out a form.
  • 1.5 is providing citizens with ways to complain to government about an issue.
  • 2.0 is creating platforms for citizens to collaborate around information to improve outcomes.

Gov 2.0 makes all of those things "necessary but insufficient," said Goldsmith. It's "when collaboration around information enhances the quality of decision making on the behalf of citizens."

One area where New York City government is exploring that kind of collaboration is in rulemaking. "We're working on opening up the process for comments to individuals and even small businesses that feel their voices are not heard, focusing on rulemaking transparency and crowdsourcing tools," he said.

Outreach is critical, given that many small business owners won't know about new online resources or platforms. "I remember when, in an in-person meeting, a small business owner complained about a new regulation," said Goldsmith. "I asked him what he or his colleagues had contributed during rulemaking process. He stared at me blankly. That was the first hint that digital outreach and crowdsourcing had a long way to go."

Putting open data to work

In some ways, thinking about government this way follows the five-star system that Tim Berners-Lee has defined for open government data. New York City has a government data repository, the NYC DataMine, that Berners-Lee might rate reasonably high on that scale, given that data sets are posted in raw form machine readable data, including in XLS, XML, CSV, and RSS formats.

"Our call center received 100,000 calls an hour during the snowstorm [last year]," said Goldsmith. "We're working to convert a large number of those to text, and then convert to personalized outbound messaging. We have an advantage over smaller cities, given the huge volume of calls. We're looking at all of the ways that we can use that data."

Goldsmith said that New York City has been moving toward making more useful public data available, including 311, geocoding, performance and regulatory data. Recently, New York launched an online 311 service request map. The city is also working to grow its community of civic entrepreneurs and developers. "We have a Big Apps program, and it's successful," said Goldsmith. As this post went live, the second version of New York City's civic application contest, NYC Big Apps, was open to public voting.

One way to gauge that success is in a recent development: one of the winners of the first iteration of the contest, MyCityWay, now is a growing company: it recently landed $5 million in funding and a partnership with BMW. The location-based technology startup provides citizens with information about public transportation, parking and entertainment.

As Justin Houk suggests at Programmable Web, in many ways the NYC BigApps 2.0 competition might be one of the first "second generation" apps contests. Houk's key insight, reflected in Mark Headd's view of application contests, is that that these efforts can help to catalyze more benefits than just free apps. Application contests can unlock innovation not only by stimulating app creation but by establishing a community of public servants, designers, librarians and civic developers who can work together on making open government data more useful in the future.

Closing the loop between transparency and accountability will be an important focus for NYC and every government that's opening its data. "We have a transparency initiative, which is on its way," said Goldsmith. "What we don't have is a transparency effort to make it better. We need better ways of organizing information and data that makes it easy for the public to discover ways to improve their way of life."

One way that discovery can happen now will be through the use of QR or "quick response" codes. On Feb. 22, Mayor Bloomberg and Department of Buildings Commissioner Limandri introduced the use of QR codes on all future NYC construction permits. "QR codes help us add context and dynamic info to NYC's physical environment," wrote Rachel Sterne, New York City's first chief digital officer on her Tumblr. The city government is now considering other ways that QR codes could be used in NYC.

NYC QR code
An example of the QR codes (upper right corner) now featured on New York City's construction permits. More information is available at Fast Company.

Lessons from the Snowpocalypse

In 2010, New York City was hit by an unexpectedly heavy winter storm that buried the Big Apple under a blanket of snow. Many streets were left unplowed for days. Ambulances couldn't get through to emergencies. And while Newark mayor Cory Booker earned national attention for his heavy use of social media to listen and engage with citizens affected by the snowfall, New York City government came under withering scrutiny from its famously critical citizens. In the wake of the first snowstorm, the Mayor's office changed its digital strategy in a number of ways.

"We knew some of the issues already, in terms of people who can't shovel or elderly who can't get out," said Goldsmith. "We need to be able to hold drivers accountable in snow emergencies. We're working on a large number of linkages between all of those issues and technology. We're also looking at the use of social media in minority populations. We have a long way to go with exclusivity, particularly for people who aren't conversant with the tools."

By the time a second major snowstorm rolled in, the Mayor's Office had set up a number of digital initiatives that created platforms for citizens to give feedback to city government. (In Boston and other cities, that activity can be described as "citizensourcing," where citizens are co-creators of solutions to community problems.)

"Between the first and second big snowstorms, we organized a number of solutions that involved crowdsourcing," said Goldsmith. "We launched a snow website with real-time information connected to it. We created a citizen-facing blog to report issues. We set up a Flickr feed where citizens could post pictures about unplowed streets. Because plowing reporting wasn't entirely accurate, we encouraged citizens to send in information. We used citizens as the eyes and ears of this performance monitoring process."

Running a civic surplus in NYC

The philosophy where a city works to increase transparency, participation and collaboration with its citizens now has a name that has gained increasing prominence over the past few years: open government. Collaboration and participation are key aspects of "We government" and open government, where a "civic surplus" of citizens' passion, expertise and patriotism is put to work fixing their own communities.

"The underlying premise here is that life is complicated," said Goldsmith. "Government is particularly difficult. The hierarchical structure where 'government knows what's best for you' is outdated. Co-creation and collaboration are the models for this century. Digital tools can dramatically improve the exchange of information and improve the quality of services. Public employees can dramatically improve this exchange, whether it's in idea markets organized by vertical topics or horizontally across agencies. Those devices are some of the best examples where citizens collaborating with government work together for better outcomes."

These efforts are going to receive a cynical response from New Yorkers unless they see actual results, not soaring rhetoric or shiny technologies. This is, after all, the city where corruption persists long after Boss Tweed held sway. At this moment in New York's history, however, there's a willingness to approach governance differently in the Mayor's office that's worth observing closely.

January 14 2011

4 civic innovation organizations to watch in 2011

What lies ahead for Gov 2.0 in 2011 is worth watching, whether it's international, federal or in states. Given the growth of citizen engagement platforms in 2010, expect more of the same in 2011. One of the most fascinating areas to track will be in the evolution of smarter cities. The technology landscape looks like a hybrid future, with the adoption of new cloud computing platforms from corporate giants like IBM, Microsoft, Cisco, Siemens and Google alongside the efforts of civic developers using open source components and smaller, scrappy startups.

Picture taken and posted to by Abhi Nemani, Code for America's director of strategy and communications.

Philip Ashlock, open government program manager at OpenPlans in New York, concisely pointed out four key civic innovation organizations to watch in 2011 in a single tweet recently. 2011 trends have been on the docket this month, so his pointer fit right in. If there's a civic surplus to be applied to smarter government, look for players from these organizations to apply it. Anyone interested in open government, open source software and civic entrepreneurship should keep an eye on the work of the following initiatives in 2011.

Code for America

In his inaugural address in January of 1961, President John F. Kennedy challenged his fellow Americans to "ask not what your country can do for you - ask what you can do for your country." In 2010, the question has been updated: ask not what your country can code for you - ask what you can code for your country. Toward that end, Jennifer Pahlka has empowered developers to become civic coders through Code for America. The nonprofit allows some of the brightest technical minds of this generation to create applications that will help government deliver better services to citizens. It offers its fellows a new kind of public service.

Code For America has begun its work to create better government through code. Its first slate of projects were introduced late last year in a webinar, embedded below. Last week, Code for America kicked off, with great coverage in Fast Company and others. For some insight into what's happening, read Pahlka's report from Day One or Dan Melton on following the data, iterating and the $1,200 problem. [Disclosure: Tim O'Reilly is on Code for America's board.]

Civic Commons

The Civic Commons code-sharing initiative is an effort to reduce government IT costs. Around the United States, city governments have created a multitude of software. Unfortunately, most of the time the code from those projects has not been shared between municipalities, which results in duplication of effort, and redundant or static software.

Civic Commons, launched at the Gov 2.0 Summit in Washington last year, is aimed squarely at helping city governments share the software they've developed. Like many civic innovation projects, the idea is relatively simple in conception but tricky in practice: track, archive, improve and re-share code developed by cities for use in other municipalities.

At launch, CivicCommons catalogued existing projects like the District of Columbia's App Store and the Federal IT Dashboard. The long-term goal of the project, according to its founders, is "to develop the app catalog into an open 'Civic Stack' -- a streamlined collection of software that cities can use to run core services."

As 2011 begins, CivicCommons had some news: the city of San Francisco open sourced its enterprise addressing system. As Karl Fogel explained at

EAS is a web-based system for managing the city's master database of physical addresses, tied to assessor's parcels and the City's street centerline network. We posted a short screencast of EAS in action a couple of months ago, and since then there's been a lot of interest in it from other jurisdictions.

Keep an eye on the evolution of CivicCommons as more technologists come on board and cities become aware of the resource. is a new public resource in the Gov 2.0 world that's set to launch later this month. The free, open source web application is based on, the nonprofit-backed website for tracking the United States Congress. will serve a similar function for state and local governments.

OpenGovernment is a joint project of the Participatory Politics Foundation and the Sunlight Foundation. The beta version of will launch with information for five state legislatures: California, Louisiana, Maryland, Texas, and Wisconsin. The nonprofit is actively seeking funding to expand to all 50 U.S. states and major cities.

OpenGovernment is also looking for a "few good civic coders." (See a theme here? To quote Microsoft CEO Steve Ballmer, "developers, developers, developers.") Here's their pitch to the open government community:

We're open-source to the core, evangelists for open standards, and activists for liberation of public data. We're building the site in Ruby on Rails and we'd love your help -- our code is available now on GitHub, and please join our brand-new Google Group to stay in touch. Our limiting factor isn't ideas, or a lack of valuable government info to make accessible, but rather web development time -- we have a huge wish list of data & features for this open-source community site, so let us know your skills & interests. We hang out in #opengovernment on

Read more about our projects, we're easy to reach and eager to talk -- don't forget to sign up for our low-volume email list above. We hope you'll help us grow and bring user-friendly transparency to every level of government, including your local community.

Urban Scale

Urban Scale brings another important focus to the virtual table: design for networked cities and citizens. Smarter use of technology is important, but good design is crucial for citizens to be able to navigate urban labyrinths, untangle regulations or understand critical instructions in crises. Designing for how people live and work is a crucial lesson from Web 2.0 for Gov 2.0, and it's one that Adam Greenfield, the founder of Urban Scale, has clearly internalized. Greenfield, previously head of design direction for service and user interface design at Nokia and lead information architect at Razorfish Tokyo, founded Urban Scale to "bring a user-centered approach to the design of urban architectures." You can read "Urban Computing and its Discontents," written with Mark Shephard, as a PDF.

The @urbnscl Twitter account is proving to be a skillful curation of ideas and news related to urban design, with links to traffic indicators, DeConcrete ("everyday urbanisms without architects' architectures"), a "parker app" from StreetLine Networks, and turning a city into an equation. With a core focus on the design of bus rapid transit (BRT) systems, data visualization for that class of services they call "citizen intelligence engines" and data collection from sensors deployed in public space," UrbanScale and — similar urban design shops — are worth watching.

More to come

There are no shortage of other civic innovation organizations to watch, in addition to those listed above.

OpenPublic is a distribution of the open source Drupal content management system specifically tailored to the needs of government. OpenPublic comes from the Phase2 Consulting Group. There's going to be a lot of focus on Drupal in government in 2011 (read up on moving to Drupal here at Radar) and this community and distribution will be one to watch.

Open Source Cities, slated to launch in the first quarter of 2011, will focus on "citizen urbanism," ecological design, urban planning, open data and collaboration. Follow @Open_Cities on Twitter for updates.

GovHub is a new nonprofit out of Portland, Ore., that's focused on "providing government agencies, non-profits and developers a place to collaborate on open source software." Yes, that sounds a lot like what Civic Commons does or what Github provides right now, on some levels. And yes, the proposals for a "" could results in the creation a government open source repository. It's worth keeping an eye on GovHub because of its location within Portland's dynamic open source community.


December 24 2010

Citizen engagement platforms grow in 2010

During the 2008 election, then Senator Barack Obama said that "the challenges we face today -- from saving our planet to ending poverty -- are simply too big for government to solve alone. We need all hands on deck." As President, finding solutions to grand challenges means that Obama is looking again to the technology community for answers. Whether he finds them will be a defining element in judging whether a young Senator from Illinois that leveraged Web 2.0 to become President can tap into that collective intelligence to govern in the Oval Office.

Will citizens collaborate with officials, workers and one another to apply a civic surplus to open government? 2011 might be a make or break year for White House efforts to create platforms for citizen engagement. There were two notable news stories on that front this month. First, the passage of the COMPETES Act in Congress means that every federal agency can create prizes and competitions. Look for more to come on

Second, the White House has made a new, ambitious request for the American people: help design digital democracy with ideas for creating a platform for citizen consultation. For now, the project is called "ExpertNet" and is being hosted on a wiki.

Both underpin significant parts of White House deputy CTO Beth Noveck's vision for open government, where stakeholders collaborate in "wiki government." In that context, making community health data as useful as weather data has potential, where companies and citizens engage with data in ways that help them make better decisions. Open source may improve healthcare through NHIN Direct, now the Direct Project.

The growth of citizen engagement platforms, however, extends far beyond Washington. Civic developers are helping government by standardizing application programming interfaces and empowering others by coding the middleware for open government. Working with developers can be a crucial complement to publishing open data online. Those citizens matter a lot there, but only if engaged.

There's a growth in "do it ourselves (DIO) government," or as the folks at techPresident like to say, "We government." That's likely to be important in the context of crises in state budgets. It was an important enough topic that I was happy to accept Adriel Hampton's invitation to join him on Gov 2.0 Radio to talk about the growth in citizen engagement in 2010. Our conversation is available in the following audio clip:

You can break down what's happening into categories:

  • Platforms
  • Data-driven decisions
  • Crowdsourcing
  • Helping one another with the digital divide and information literacy
  • Guerrilla design efforts to make better cities
  • Sharing information with one another with respect to how to do things

Social media is a big part of that, but far from the only one. The "army of techies" taking on City Hall are using apps like SeeClickFix and Citysourced to report issues and track their resolution.

The growth of CityCamp is fascinating to watch, in that context. 2010 was the year when many of these camps did something more than talk and share information, though those activities have value. While the federal government, particularly the Coast Guard and EPA, used social media to get information out and set up, citizens used platforms like Ushahidi at the Lousiana Bucket Brigade, or tried grassroots mapping. Citizens can do a lot on their own. Look at what people did with the CrisisCamp Haiti work, particularly Open Street Map.

That's important, but when you look at what happened with CrisisCommons, you can see the germ of something bigger in the Oil Reporter app. That's one reason why the Sloan Foundation gave them a grant. CrisisCommons uses the Internet to act as a platform for people to help one another. As Andy Carvin from NPR has pointed out, now citizens can do more than donate blood or money. They can share information and donate their skills and time.

2010 was also notable for the growth of civic hackathons, like the International open data hackathon, the Times Open Hack Day, and Random Hacks of kindness.

As the new year beckons, there are more ways for the citizens of the United States to provide feedback to their federal government than perhaps there ever have been in its history. In 2011, the open question is whether "We the people" will use these new participatory platforms to help government work better. The evolution of these kinds of platforms aren't U.S.-centric, either. Ushahidi, for example, started in Africa and has been deployed worldwide. The crowd matters more now in every sense: crowdfunding, crowdsourcing, crowdmapping, collective intelligence, group translation, and human sensor networks.

I'll close with a prediction for 2011: Watch for important innovation to come out of India in the mobile space, as open government for transparency moves ahead.

November 23 2010

Coding the middleware for government data

Cities, states and agencies are publishing more government data online, but that's just the tip of the iceberg. Much government data is still in paper form, locked away in file cabinets, or in closed formats on obscure servers. For instance, the data-driven story of BrightScope, which uses government data to clarify 401(k) plans, started with boxes upon boxes of printouts. The Department of Labor is just now starting to put that data online. That's why reporting on the progress of open government data initiatives is a key pillar of Gov 2.0. For those who have been working toward more transparent government, that issue is central to their work.

"Embracing Tim O'Reilly's concept of 'Government as a Platform' is easier said than done," wrote Max Ogden in his pitch for the first Ignite Gov at the Government Open Source Conference (GOSCON) in Portland. During a five-minute presentation, Ogden offered up a refreshing personal perspective on what it takes for civic hackers to put open data to good use. Here's his talk:

Under the Open Government Directive, a PDF qualifies as an open format. BrightScope uses government data, but it's not "open" in the sense that technologists use the term, nor did BrightScope's business result from the open government initiative. Put in the context of Tim Berners-Lee's definition for open linked data or the principles at, PDFs on CD might not merit even one star, although BrightScope has been able to move forward with their business in the meantime.

IT officials from the Office of Management and Budget, the White House Office of Science and Technology Policy, or open government technologists like Clay Johnson or Noel Hidalgo have expressed a preference for data published in structured formats. Federal CIO Vivek Kundra said at the International Open Government Data Conference that releasing open data can be seen through three lenses: accountability, citizen utility and creating economic value.

"Putting out data is not enough," said Beth Noveck, White House deputy CTO for open government, at the Open Cities Conference earlier this month. "It's what we do with that data to make it useful." It's civic hackers like Ogden who have done just that, with a little help from the government. Ogden wants local government to act as a data supplier, providing the means for civic hackers to make things that help citizens to make better decisions. Ogden was instrumental in connecting data and developers through Portland's Civic Apps contest.

More examples of open government and civic innovation will depend on similar public-private partnerships of open data, developers and entrepreneurs. Portland, San Francisco and Boston have shown how open data can be used to spur economic activity. Real-time transit data in Boston has created a whole new ecosystem of apps. The Apps for California contest that featured mashups of government data is behind a new startup, Zonability. And the healthcare apps coming out of a community health data initiative at the Department of Health and Human Services are continuing to evolve.

If governments wish to provide citizens with better understanding of how government works and what it's doing with taxpayer funds, that means using the most efficient, cost-effective means to provide that transparency, and the right data sets for real accountability. Consider how the British government released spending: spreadsheets. That choice enabled the Guardian to download the data and help citizens analyze it.

The story of open data and Max Ogden is evolving too: he's one of the inaugural fellows of Code for America. As Code for America founder Jen Pahlka notes:

They will be the first participants in our experiment to help city governments better leverage the power of the web. Starting in January, it will be their challenge to not only build innovative apps for each of our cities, but also become the leaders of the ongoing movement to make government more open and efficient.

Based on Ogden's track record, there will be more open government middleware to come.


November 09 2010

Local government: data supplier

What are Skip Newberry's lessons learned from a regional approach to open data and civic apps? Newberry, who serves as economic development policy adviser to Portland, Ore. mayor Sam Adams, offered practical insights that other cities can apply to their own open data initiatives during his Ignite Gov talk at the Government Open Source Conference (GOSCON).

"In the near future, collaboration among different jurisdictions in standardizing data across local, county, state, and international boundaries will pose significant challenges," said Newberry. "I do not think these are insurmountable."

His talk, embedded below, focused on the open data and app design contest in the Portland area called Civic Apps.

Newberry is well placed to speak to the progress of Portland's initiatives involving software and digital media. Last year, he helped to draft an open source procurement and open data policy that was subsequently adopted by the Portland City Council.

Open government and civic innovation were on display at an awards ceremony the day after Newberry spoke. Among the winners of the second round of Civic Apps awards, stood out for its potential for broader use in crisis or disaster response around the country or world.

Will open data initiatives in Portland lead to more economic activity and improve the life of citizens? Mayor Adams believes so. "In Portland, like I think most cities, when people are armed with knowledge, they make wiser choices," Adams said in an interview at the awards ceremony. He pointed toward PDX Bus as an example of innovation that helps citizens better navigate the city.

"We are seeing folks that are sole proprietorships hire folks to help them build their business, their app business, their online business with our data sets," Adams said. "For us, data has always been there, in some cases for decades. Putting it to use for the public and helping people make money while they do it -- we intend to be the open source capitol of the nation -- and this is one contribution we can make, with our data sets."

Older posts are this way If this message doesn't go away, click anywhere on the page to continue loading posts.
Could not load more posts
Maybe Soup is currently being updated? I'll try again automatically in a few seconds...
Just a second, loading more posts...
You've reached the end.
No Soup for you

Don't be the product, buy the product!

YES, I want to SOUP ●UP for ...