Newer posts are loading.
You are at the newest post.
Click here to check if anything new just came in.

March 22 2013

Sensoring the news

When I went to the 2013 SXSW Interactive Festival to host a conversation with NPR’s Javaun Moradi about sensors, society and the media, I thought we would be talking about the future of data journalism. By the time I left the event, I’d learned that sensor journalism had long since arrived and been applied. Today, inexpensive, easy-to-use open source hardware is making it easier for media outlets to create data themselves.

“Interest in sensor data has grown dramatically over the last year,” said Moradi. “Groups are experimenting in the areas of environmental monitoring, journalism, human rights activism, and civic accountability.” His post on what sensor networks mean for journalism sparked our collaboration after we connected in December 2011 about how data was being use in the media.

AP Beijing Air Quality graphicAP Beijing Air Quality graphic

Associated Press visualization of Beijing air quality. See related feature.

At a SXSW panel on “sensoring the news,” Sarah Williams, an assistant professor at MIT, described how the Civic Data Design Project had partnered with the Associated Press to independently measure air quality in Beijing.

Prior to the 2008 Olympics, the coaches of the Olympic teams had expressed serious concern about the impact of air pollution on the athletes. That, in turn, put pressure on the Chinese government to take substantive steps to improve those conditions. While the Chinese government released an index of air quality, explained Williams, they didn’t explain what went into it, nor did they provide the raw data.

The Beijing Air Tracks project arose from the need to determine what the conditions on the ground really were. AP reporters carried sensors connected to their cellphones to detect particulate and carbon monoxide levels, enabling them to report air quality conditions back in real-time as they moved around the Olympic venues and city.

The sensor data helped the AP measure the effect of policy decisions that the Chinese government made, said Williams, from closing down factories to widespread shutdowns of different kinds of industries. The results from the sensor journalism project, which showed a decrease in particulates but conditions 12 to 25 times worse than New York City on certain days, were published as an interactive data visualization.

AP Beijing mash-up of particulate levels and photography in Beijing.AP Beijing mash-up of particulate levels and photography in Beijing.

Associated Press mashup of particulate levels and photography at the Olympic stadium in Beijing over time.

This AP project is a prime example of how sensors, data journalism, and old-fashioned, on-the-ground reporting can be combined to shine a new level of accountability on official reports. It won’t be the last time this happens, either. Around the world, from the Amazon to Los Angeles to Japan, sensor data is now being put to use by civic media and journalists.

Sensing civic media

There are an increasing number of sensors in our lives, said John Keefe, a data news editor for WNYC, speaking at his SXSW panel in Austin. From the physical sensors in smartphones to new possibilities built with Arduino or Raspberry Pi hardware, Keefe highlighted how journalists could seize hold of new possibilities.

“Google takes data from maps and Android phones and creates traffic data,” Keefe said. “In a sense, that’s sensor data being used live in a public service. What are we doing in journalism like that? What could we do?”

The evolution of Safecast offers a glimpse of networked accountability, collecting and publishing radiation data through sensors, citizen science and the Internet. The project, which won last year’s Knight News Challenge on data, is now building the infrastructure to enable people to help monitor air quality in Los Angeles.

Sensor journalism is also being applied to make sense of the world in using remote sensing data and satellite imagery. The director of that project, Gustavo Faleiros, recently described how environmental reporting can be combined with civic media to collect data, with relevant projects in Asia, Africa and the Americas. For instance, Faleiros cited an environmental monitoring project led by Eric Paulos of the University of California at Berkeley’s Center for New Media, where sensors on taxis were used to gather data in Accra, Ghana.

Another direction that sensor data could be applied lies in social justice and education. At SXSW, Sarah Williams described [slides] how the Air Quality Egg, an open source hardware device, is being used to make an argument for public improvements. At the Cypress Hills Community School, kids are bringing the eggs home, measuring air quality and putting data online, said Williams.

Air Quality Eggs at Cypress Hill Community SchoolAir Quality Eggs at Cypress Hill Community School

Air Quality Eggs at Cypress Hill Community School.

“Health sensors are useful when they can compare personal real-time data against population-wide data,” said Nadav Aharony, who also spoke on our panel in Austin.

Aharony talked about how Behavio, a startup based upon his research on smartphones and data at MIT, has created funf, an open source sensing toolkit for Android devices. Aharony’s team has now deployed an integration with Dropbox that requires no coding ability to use.

According to Aharony, the One Laptop Per Child project is using funf in tablets deployed in Africa, in areas where there are no schools. Researchers will use funf as a behavioral tool to sense how children are interacting with the devices, including whether tablets are next to one another.

Sensing citizen science

While challenges lie ahead, it’s clear that sensors will be used to create data where there was none before. At SXSW, Williams described a project in Nairobi, Kenya, where cellphones are being used to map informal bus systems.

The Digital Matatus project is publishing the data into the General Transit Feed Standard, one of the most promising emerging global standards for transit data. “Hopefully, a year from now [we] will have all the bus routes from Nairobi,” Williams said.

Map of Matatus stops in Nairobi, KenyaMap of Matatus stops in Nairobi, Kenya

Map of Matatus stops in Nairobi, Kenya

Data journalism has long depended upon official data released by agencies. In recent years, data journalists have begun scraping data. Sensors allow another step in that evolution to take place, where civic media can create data to inform the public interest.

Matt Waite, a professor of practice and head of the Drone Journalism Lab at the University of Nebraska-Lincoln, joined the panel in Austin using a Google Hangout and shared how he and his students are experimenting with sensors to gather data for projects.

Journalists are going to run up against stories where no one has data, he said. “The old way was to give up,” said Waite. “I don’t think that’s the way to do it.”

Sensors give journalists a new, interesting way to enlist a distributed audience in gathering needed data, he explained. “Is it ‘capital N’ news? Probably not,” said Waite. “But it’s something people are really interested in. The easy part is getting a parts list together and writing software. The hard part is the creative process it takes to figure out what we are going to measure and what it means.”

In an interview with the Nieman Journalism Lab on sensor journalism, Waite also raised practical concerns with the quality of data collection that can be gathered with inexpensive hardware. “One legitimate concern about doing this is, you’re talking about doing it with the cheapest software you can find,” Waite told the Nieman Lab’s Caroline O’Donovan. “It’s not expertly calibrated. It’s not as sensitive as it possibly could be.”

Those are questions that will be explored practically in New York in the months ahead, when New York City’s public radio station will be collaborating with the Columbia School of Public Health to collect data about New York’s environmental conditions. They’ll put particulate detectors, carbon dioxide monitors, leg motion sensors, audio monitors, cameras and GPS trackers on bicycles and ride around the city collecting pollution data.

“At WNYC, we already do crowdsourcing, where we ask our audience to do something,” said Keefe. “What if we could get our audience to do something with this? What if you could get an audience to work with you to solve a problem?”

Keefe also announced the Cicada Project, where WNYC is inviting its listeners to build homemade sensors and track the emergence of cicadas this spring across New Jersey, New York and the Northeast region.

This cicada tracker project is a 21st century parallel to the role that birders have played for decades in the annual Christmas Bird Count, creating new horizons for citizen science and public media.

Update: WNYC’s public is responding in interesting ways that go beyond donations. On Twitter, Keefe highlighted the work of a NYC-based hacker, Guan, who was able to make a cicada tracker for $20, 1/4 the cost of WNYC’s kit.

Sensing challenges ahead

Just as civic technologists need to be mindful of “solutionism,” so too will data journalists need to be aware of the “sensorism” that exists in the health care world, as John Wilbanks pointed out this winter.

“Sensorism is rife in the sciences,” Wilbanks wrote. “Pick a data generation task that used to be human centric and odds are someone is trying to automate and parallelize it (often via solutionism, oddly — there’s an app to generate that data). What’s missing is the epistemic transformation that makes the data emerging from sensors actually useful to make a scientific conclusion — or a policy decision supposedly based on a scientific consensus.”

Anyone looking to practice sensor journalism will face interesting challenges, from incorrect conclusions based upon faulty data to increased risks to journalists carrying the sensors, to gaming or misreporting.

“Data accuracy is both a real and a perceived problem,” said Moradi at SXSW. “Third-party verification by journalists or other non-aligned groups may be needed.”

Much as in the cases of “drone journalism” and data journalism, context, usage and ethics have to be considered before you launch a quadcopter, fire up a scraper or embed sensors around your city. The question you come back to is whether you’re facing a new ethical problem or an old ethical problem with new technology, suggested Waite at SXSW. “The truth is, most ethical issues you can find with a new analogue.”

It may be, however, that sensor data, applied to taking a “social MRI” or other uses, may present us with novel challenges. For instance, who owns the data? Who can access or use it? Under what conditions?

A GPS device is a form of sensor, after all, and one that’s quite useful to law enforcement. While the Supreme Court ruled that the use of a GPS device for tracking a person without a warrant was unconstitutional, sensor data from cellphones may provide law enforcement with equal or greater insight into a target’s movements. Journalists may well face unexpected questions about protecting sources if their sensor data captures the movements or actions of a person of interest.

“There’s a lot of concern around privacy,” said Moradi. “What data can the government request? Will private companies abuse personal data for marketing or sales? Do citizens have the right to personal data held by companies and government?”

Aharony outlined many of the issues in a 2011 paper on stealing reality, exploring what happens when criminals become data scientists.

“It’s like a slow-moving attack if you attach yourself to someone’s communication,” said Aharony, in a follow-up interview in Austin. “‘iPhonegate‘ didn’t surprise people who know about mobile app data or how the cellular network is architected. Look at what happened to Path. You can make mistakes without meaning to. You have to think about this and encrypt the data.”

This post is part of our series investigating data journalism.

March 29 2011

4 SXSWi themes reveal the story within the story

For those who went to Austin, Texas this month in search of the next big thing in technology, the South by Southwest Interactive (SXSWi) Festival may have proved disappointing. There was no single breakout company, platform or technology to be found, no matter how hard the hundreds of tech journalists and bloggers sized up the wares, apps and presentations of the startups and tech titans vying for attention. For those who went in search of connections to the tech community, it was a goldmine. After the dust from nearly 20,000 attendees rambling around Austin's streets settled, MG Siegler declared advertising and the iPad 2 the "winners" of SXSWi 2011.

Based upon my experiences there, it's hard to call him wrong, given how plastered with advertising the city had become. Big brands moved in for the week, from Apple's popup store to CNN, whose cafe sign was one of the iconic images of the event.

CNN Grill at SXSW

That said, there were technologies to be found that are clearly gathering steam. QR codes were everywhere, from Mashable's party to bar napkins to fundraising for Japan. "Gamification," where entrepreneurs, nonprofits or even governments try to add a game layer to commerce, learning or, really, anything, was also hot.

But those are just the technologies that caught my eye. Below I outline the larger trends that continue to resonate with me as the excitement of SXSWi naturally dissipates.

Offline is online

Angry Birds being played via a projector
Angry Birds being played via projector

While there were literally hundreds of companies vying to grab some of the hyperkinetic attendees' attention, the perspective that matters most is what their behavior can tell us about where our relationship to technology and one another is headed. The social behavior, omnipresent mobile devices and technologies embraced at SXSWi point to something more interesting: the dissolution of the boundaries between the offline and online world.

The Pew Internet and LIfe Project estimates Internet penetration in the United States at around 79%. A few countries are more wired. The majority are less so, although that's changing. At SXSWi, however, nearly everyone was connected to the Internet from waking to sleeping. The early adopters at the festival can offer some insight into what's coming down the pike for the rest of society, as more of us are constantly connected by a mobile device with a pervasive Internet connection.

At an impromptu panel on social media and the Middle East held at Twitter's SXSWi retreat, NPR's Andy Carvin reflected that for many of the young people whose struggles he's been chronicling in real-time, the question of "offline vs online" wouldn't register as meaningful: they've already merged in their lives.

The collective behavior of those networked masses was at the crux of Oliver Burkema's dispatch on SXSWi at the Guardian. He observed that the festival heralded:

... the final disappearance of the boundary between 'life online' and 'real life', between the physical and the virtual. It thus requires only a small (and hopefully permissible) amount of journalistic hyperbole to suggest that the days of 'the internet' as an identifiably separate thing may be behind us.

If that all sounds a bit familiar to Radar readers, it should: Clay Shirky, in discussing the role of the Internet as a platform for collective action heralded the "death of cyberspace" and the end of geek culture back in January. This isn't a new idea, but the research and observation are finally aligning with reality.

Shirky's SXSWi talk about social media, the so-called "dictator's dilemma" and Egypt, drew hundreds in the audience to think about what the impact of these trends means for millions of people in the Middle East and beyond. The insight that Shirky shared again is that connecting citizens to one another has been historically undervalued. As more connection technologies enter countries where information has been tightly controlled, expect the Internet to continue to act as a disruptor.

In watching the ebb and flow of the hordes experiencing idea overload of SXSWi, Edward Boches noticed the same melding of online and offline lives. If the behavior of these early adopters is a precursor to mainstream adoption, expect this trend to continue.

This also means that civil society will continue to need great teachers and thoughtful guides to information gathering and digital literacy. Based upon those demands, Phoebe Connolly's choice to call SXSW 2011 the "year of the librarian" looks spot on.

Mobile, location, and social

In his analysis of 2011 tech trends at the beginning of the year, my colleague Mike Loukides observed that "you don't get any points for predicting that 'Mobile is going to be big in 2011'." If you're looking for what was big at SXSWi, any reasonable observer had to acknowledge that mobile devices, maps, websites, services, experiences, marketing and data were profoundly relevant.

Stowe Boyd, who had written that he wasn't going to SXSWi, found something of great value in the "thriving petri dish for a social, mobile future." As Boyd observed in his post on SXSWi:

Over 50% of the world's population is now urban, and that is expected to rise to over 60% by 2030. The cities will not only be bigger, but increasingly dense, so what we learn from SxSW today could shape the social, mobile, urban landscape of the near future, since many of the architects of the future were there, taking notes.

With tablets in abundance, Caroline McCarthy's noted that SXSWi offered a peek at what a post-PC society might look like. By year's end, more Xooms, Galaxy Tabs and BlackBerry Playbooks will likely joins the millions of iPads already in consumers' hands. Augmented reality apps that bridge the gap between online and offline life, like Sensierge, may be on many of them.

Where 2.0: 2011, being held April 19-21 in Santa Clara, Calif., will explore the intersection of location technologies and trends in software development, business strategies, and marketing.

Save 25% on registration with the code WHR11RAD

You can put location and social media in the same category of trend: impossible to omit but obvious to report. Location-based applications like Foursquare and Gowalla were visible everywhere, with a host of other startups looking to pick up some screen real estate and new users.

There was one clear winner in the social space: POPVOX, which took home an award for the best social networking site at the SXSWi Accelerator competition. Making social media in politics meaningful has been a tough nut to crack, but trying to use the Internet to make Congress smarter may be an idea whose time has come. [Disclosure: Tim O'Reilly was an early investor in POPVOX.]

California-based Votizen, which powered a social media campaign for the Startup Visa bill during SXSWi, is focusing on this space as well. Look for an in-depth report on their efforts here on Radar soon.

Mastering social media saturation at SXSWi, however, as Daniel Terdiman pointed out at CNET, required developing better filters and tuners for signals. Jeremiah Owyang found SXSWi great for networking but only if people remembered to detach from their mobile devices.

Last year, Clive Thompson wrote about about the death of the phone call. In 2011, SXSWi may have "socially" written its obituary. Making a phone call was superseded by texts, checkins, email, tweets and instant messages. That's not to say that there weren't plenty of people on the phone there, just that the cornucopia of other communication options means synchronous voice communication wasn't always the first or even third option. Social media overload on Twitter, Facebook and location-based networks created an opportunity for group messaging apps like, Fast Society or Beluga to connect people without sharing the information with everyone.

Part of the impulse to check and recheck social media is deep-seated in biochemical pathways, as Caterina Fake described in her analysis of FOMO and social media. "FOMO," or the "Fear of Missing Out," has been used by savvy entrepreneurs to drive use of their apps to find out what's happening, where and with whom. As Katherine Rossman reported for the Wall Street Journal, at SXSWi 2011, "looking down was the new looking up."

For some geeks who have been in the industry for a long time, however, being more substantively human in person was a greater attraction. As Gina Trapani may have said it best:

The best kind of social networking: one face talking to another face within 3 feet of each other.

Big data, open data and your data

Reid Hoffman's keynote talk was one of the highlights, due in no small part to the founder of LinkedIn's focus on data. Hoffman's big idea is the importance of big data, which in many ways is leading us into the next stage of the Internet. In 5 years, he posited "a product designer may need to have the characteristics of a data scientist." As we look ahead, "the future is sooner and stranger than we think."

Hoffman provided 10 rules of entrepreneurship gleaned from his time as an entrepreneur and venture capitalist. "The way we make human progress is how we collaborate together," he said.

New digital platforms and analytics will offer unexpected opportunities. "Airbnb gives us the market of eBay for [physical] space," said Hoffman, enabling people to shift how things are priced or offered. They also bring new risks. "Trying to make data trails invisible to people is nearly a Sisyphean task," he said, highlighting a key flashpoint that lurks amidst the growing petabytes of data: privacy and security.

"I would like a data dashboard with the information that the government has about me," said Hoffman, perhaps akin to the Google dashboard that provides insight into data on that service. What concerned him is that we might end up in a state akin to Aldous Huxley's "Brave New World." Awash in data, how do we discern truth in vast amounts of data?

While some will balk at the versioning of calling this "Web 3.0," the coming age of data science looks too big and too fundamental a shift to ignore.

Privacy and digital rights

Nestled amidst the hype and potential of the technologies on display were concerns about what this explosion of location and mobile will mean for unwary consumers. "Tucked modestly into the corner, an American Civil Liberties Union table offers white papers and postcards warning of the privacy dangers of all this data mining," reported Jessica Clark for PBS Mediashift.

I moderated a panel on a "social networking bill of rights," which has continued to receive attention in the days since the festival from MSNBC,, Identity Blog, Liminal States, and PC World.

At, Alistair Fairweather highlighted key questions for the technology industry to consider in the months ahead:

Why is user data always vested within the networks themselves? Why don't we host our own data as independent "nodes," and then allow networks access to it?

On that count, a briefing with a startup emerging from stealth mode at SXSWi suggested how an emerging ecosystem of trust frameworks could offer a trust layer for new class of personal data stores like the Locker Project. More on trust frameworks and that startup in a future post.

SXSWi grows up

SXSW at night

The SXSWi festival itself may be the biggest winner — or loser, depending on the critic taking stock. McCarthy called it out: SXSWi has changed — "so deal with it."

SXSWi 2011 was less a conference than a much-needed test bed for new promotions like the partnership between Foursquare and American Express, edgy marketing initiatives, and fresh habits of mobile behavior like the much-hyped showdown between a handful of similar "group messaging" applications, which permitted attendees to communicate and travel in packs as Twitter has grown too overwhelmingly popular to use as a fine-tuned way to navigate the festival.

Fair warning: 2011 was my first SXSWi. I harbor no kind, warm memories of when the festival was smaller, gentler or allowed for different kinds of community building in the technology world. In many respects, the expansion of SXSWi into an immense tech trade show / geek spring break / sprawling conference reflects the vastly expanded role for interactive technologies in modern society. Whether we do something meaningful with them beyond finding a better party is in our hands.

Coming soon: A report on SXSWi for the Gov 2.0 crowd. As Patrick Ruffini observed, SXSWi 2011 "was the year that open government made its first huge splash" at the festival.

March 12 2011

SXSW 2011: Can Facebook photos be used commercially?

Social network grilled over whether businesses and advertisers could co-opt 'Flickr's worth of photos uploaded every month'

Much of the focus of this discussion was inevitably focused on Facebook's photos product manager, Sam Odio, who disappointingly played the "not my remit' card when asked the most interested and pertinent questions about Facebook's use of users' photos, including facial recognition and how images might be co-opted by advertisers.

• Facebook sees "a Flickr's worth of photos uploaded every month", said Odio. But it's worth considering the different values of those two services: Flickr includes some high-quality, well edited photography, while Facebook focuses on storytelling over quality. It doesn't matter, said Odio, if that first photo of your newborn nephew is blurry: it's the social context behind the photo.

• Odio fielded a question by one delegate about how businesses and advertisers might start appropriating photos for commercial use. "We're not in the business of selling ads through people's photos and we want to prevent businesses having free rein over users," he said. "But businesses are users," pushed the delegate. Odio said Facebook would want the people in the photos to be telling the story – which means advertising would be there but more subtly, and directed by users.

• As for ownership of photos, Odio said that comes down to the need to build the API in such a way that it can access your friends' photos. If each of those users retained ownership, that would become very complicated. "There are worries we are going to use photos in advertising but it doesn't really benefit us that much given how sensitive the subject is."

Yan-David Erlick, a serial entrepreneur who founded, predicted that social photos will become even more integrated with our lives through different sorts of tagging. "Timelines between items will mean that over time, these entities are not viewed as individual pieces of media but will have contextual attributes tying them to other pieces."

• Odio explained how after struggling to keep his startup photo site Divvyshot going in 2009, ploughing in all his own savings, he got a random email one Sunday night. It was from Blake Ross, who later turned out to be co-creator of Firefox, at an address at Facebook. "He said 'Sam – your site looks interesting. You should come here.' I was living with six developers at the time and they were all looking over my shoulder to figure out if the email was fake or not." It was, and Facebook acquired Divvyshot in April 2010.

• Feature requests aren't always the best way to develop a product. Odio said nobody asked for Instagram, which just raised $7m in funding, but now it is taking off. Facebook's engineers also have a monthly hackathon where they can work on whatever they like; that doesn't determine product direction but features such as drag-and-drop organisation have come out of that.

• On facial recognition, all Odio would say is that Facebook "hasn't been able to move quickly on it given how sensitive it is", which does seem to imply it would have liked to do plenty if it could have got away with it.

• Odio said a startup should make the product extremely simple; he had got distracted when trying to add too many features and functions. "Focus on one thing and do it extremely well. In early days the product needs to be explained to users in 10 seconds or less."

• One delegate said he was concerned that Facebook is becoming such an important repository for his life, and that photos are the most easily accessible part of that archive compared to status updates or messages. Erlich described the web being used as an external memory for us all, from photos to phone numbers; this ties in with Clay Shirky's idea of cognitive surplus – if machines can take over the mechanical parts of our brain function, what can we do with the space and energy that frees up? © Guardian News & Media Limited 2011 | Use of this content is subject to our Terms & Conditions | More Feeds

Older posts are this way If this message doesn't go away, click anywhere on the page to continue loading posts.
Could not load more posts
Maybe Soup is currently being updated? I'll try again automatically in a few seconds...
Just a second, loading more posts...
You've reached the end.
No Soup for you

Don't be the product, buy the product!

YES, I want to SOUP ●UP for ...