Newer posts are loading.
You are at the newest post.
Click here to check if anything new just came in.

September 24 2013

Four short links: 24 September 2013

  1. Measuring Heart Rate with a Smartphone Camera — not yet realtime, but promising sensor development.
  2. iBeaconslow-power, short-distance location monitoring beacons. Any iOS device that supports Bluetooth Low Energy can become an iBeacon, and can detect other iBeacons when they are nearby. Apps can be notified when iBeacons move in and out of range of the device, and can monitor the proximity of iBeacons as their proximity changes over time.
  3. Analysis: The Quantified Self (BBC) — radio show on QS. Good introduction for the novice.
  4. Tinke — heart rate, blood oxygen, respiration rate, and heart rate variability in a single small sensor that plugs into your iOS device.
  5. August 07 2013

    Four short links: 7 August 2013

    1. Toxic Behaviouronly 5% of toxic behavior comes from toxic people; 77% of it comes from people who are usually good.
    2. More Encryption Is Not The Solution (Poul-Henning Kamp) — To an intelligence agency, a well-thought-out weakness can easily be worth a cover identity and five years of salary to a top-notch programmer. Anybody who puts in five good years on an open source project can get away with inserting a patch that “on further inspection might not be optimal.”
    3. On Location With Foursquare (Anil Dash) — Foursquare switched from primarily being concerned with the game-based rewards around engagement and the recording of people’s whereabouts to a broader mission that builds on that base to be about location as a core capability of the Internet.
    4. The Flipped Flipped Classroomthe “exploration first” model is a better way to learn. You cannot have the answers before you think of the questions. (via Karl Fisch)
    Sponsored post
    Reposted byLegendaryy Legendaryy

    June 25 2013

    Four short links: 25 June 2013

    1. Cliodynamics: History as Sciencea systematic application of the scientific method to history: verbal theories should be translated into mathematical models, precise predictions derived, and then rigorously tested on empirical material. In short, history needs to become an analytical, predictive science.
    2. Cricket — indoor location system from MIT. In a nutshell, Cricket uses a combination of RF and ultrasound technologies to provide location information to attached host devices. Wall- and ceiling-mounted beacons placed through a building publish information on an RF channel. [...] The listener runs algorithms that correlate RF and ultrasound samples (the latter are simple pulses with no data encoded on them) and to pick the best correlation. Even in the presence of several competing beacon transmissions, Cricket achieves good precision and accuracy quickly.
    3. The World at Nightan international effort to present stunning nightscape photos and time-lapse videos of the world’s landmarks against celestial attractions.
    4. Paul Steinhardt on Impossible Crystals (YouTube) — quasi-crystals with five-fold symmetry previously believed impossible. And then he found one, and led an expedition in 2011 to Chukotka in Far Eastern Russia to find new information about its origin and search for more samples. As you do when you’re the Albert Einstein Professor of Science at Princeton, a job title that comes with no pressure at all to bring home the impossible.

    June 24 2013

    Four short links: 24 June 2013

    1. Reading Runes in Animal Movement (YouTube) — accessible TEDxRiverTawe 2013 talk by Professor Rory Wilson, on his work tracking movements of animals in time and space. The value comes from high-resolution time series data: many samples/second, very granular.
    2. Best Science Writing Online 2012 (Amazon) — edited collection of the best blog posts on science from 2012. Some very good science writing happening online.
    3. Designing Effective Multimedia for Physics Education (PDF) — Derek Muller’s PhD thesis, summarised as “mythbusting beats lectures, hands down”. See also his TED@Sydney talk.
    4. Melomics — royalty-free computer-generated music, all genres, for sale (genius business model). Academic spinoff from Dr. Francisco J. Vico’s work at UMA in Spain.

    November 08 2012

    Square Wallet, the Apple Store, and Uber: Software Above the Level of a Single Device

    Back in 2003, Dave Stutz, in his parting letter to Microsoft, wrote a prescient line about the future of technology: “Useful software written above the level of the single device will command high margins for a long time to come. Stop looking over your shoulder and invent something!” Software above the level of a single device! That line stuck with me, and has been a foundation of my thinking and writing ever since, helping to shape both The Open Source Paradigm Shift and What is Web 2.0?

    But this line has never seemed more prescient than today, in the new wave of software that blends mobile devices in the hands of more than one person, big data back ends, and a profound re-imagination of services, business processes, and interfaces. Yesterday’s announcement that 7,000 Starbucks locations now accept Square Wallet drives home just how much technology is changing the game for business. It isn’t just the web, big data, or even mobile, it’s the combination of them all into new systems of interaction between companies and their customers.

    If you’ve never experienced the magic of walking into a coffee shop, having the cashier glance down at their iPad-based Square Register to verify your face and payment credentials already provided by your phone’s automatic check-in, and buying your coffee simply by confirming your name, you haven’t yet tasted the future.

    Square Wallet and Square Register aren’t just mobile applications, they are a profound rethinking of the entire business process of buying something at a retail location. They combine not just one but two mobile applications, a cloud-based data backend with payment information, identity, and perhaps even your purchase preferences at a merchant you frequent, location-based check-in, and more, all woven into a seamless experience. Software above the level of a single device. Retail will never be the same again.

    The Apple Store has got a lot of the same magic. Gone is the cash register. Clerks instead wander the store, offering advice, and, when you’re ready to buy, they hand you your product, and offer to email you your receipt. Your name and credit card are already on file. You and the sales clerk are already part of the system. Software above the level of a single device.

    Or consider Uber. You look on your phone. The nearest car is three minutes away. You choose the car and driver you want – perhaps based on proximity, but perhaps on the basis of user ratings of the driver. When the driver is outside, you receive a text message. When you arrive at your destination, you simply thank the driver and step out. Payment information is already on file. Software above the level of a single device. Magic.

    This is only the beginning of a great rewiring of every aspect of business processes and interactions. The web was never just about content, but always about building the infrastructure for a kind of internet operating system. The first apps on that operating system were thinly upgraded versions of what went before, but the true native apps are starting to arrive. Software above the level of a single device. Magic.

    March 23 2012

    Top Stories: March 19-23, 2012

    Here's a look at the top stories published across O'Reilly sites this week.

    Why StreetEasy rolled its own maps
    Google's decision to start charging for its Maps API is leading some companies to mull other options. StreetEasy's Sebastian Delmont explains why and how his team made a change.

    What is Dart?
    Dart is a new structured web programming platform designed to enable complex, high-performance apps for the modern web. Kathy Walrath and Seth Ladd, members of Google's developer relations team, explain Dart's purpose and its applications.

    My Paleo Media Diet
    Jim Stogdill is tired of running on the info treadmill, so he's changing his media habits. His new approach: "Where I can, adapt to my surroundings; where I can't, adapt my surroundings to me."

    The unreasonable necessity of subject experts
    We can't forget that data is ultimately about insight, and insight is inextricably tied to the stories we build from the data. Subject experts are the ones who find the stories data wants to tell.

    Direct sales uncover hidden trends for publishers
    A recent O'Reilly customer survey revealed unusual results (e.g. laptops/desktops remain popular ereading devices). These sorts of insights are made possible by O'Reilly's direct sales channel.

    Where Conference 2012 is where the people working on and using location technologies explore emerging trends in software development, tools, business strategies and marketing. Save 20% on registration with the code RADAR20.

    March 16 2012

    Top Stories: March 12-16, 2012

    Here's a look at the top stories published across O'Reilly sites this week.

    Understanding place and space in a digital Babel
    Computational linguist Robert Munro says the subtleties of spatial distinctions are growing in importance as more of the world's digital information takes the form of non-English, unstructured text.

    When game development met Kickstarter
    Several game developers have decided that game funding and Kickstarter are two great tastes that taste great together.

    The state of ebook pricing
    Joe Wikert looks at the agency model, efficiencies, fixed pricing and other major trends that will drive ebook pricing in the months ahead.

    Foxconn and Ford, Emerson and Jobs
    Ralph Waldo Emerson's essay on "Compensation" was a source of inspiration for Henry Ford. It also affirms some of the cosmic truths Steve Jobs held dear.

    Three of our best data interviews from Strata CA 12
    Featuring: Hadoop creator Doug Cutting on the similarities between Linux and the big data world, Max Gadney from After the Flood explains the benefits of video data graphics, and Kaggle's Jeremy Howard looks at the difference between big data and analytics.

    Where Conference 2012 is where the people working on and using location technologies explore emerging trends in software development, tools, business strategies and marketing. Save 20% on registration with the code RADAR20.

    February 03 2012

    Makers and hackers: The Where Conference is looking for you

    Where Conference 2012The program for Where, our geolocation and mapping conference, is almost complete. Now we're looking for makers, hackers, developers, and DIYers to bring awesomeness to the 2012 Where Conference (April 2-4 in San Francisco).

    There are three ways to participate.

    1. Share an amazing geo/location/data visualization video or image

    Geodata is often best expressed visually. Inspired by projects like Cab Spotting, Dave Imus' The Essential Geography Of The United States Of America and Eric Fisher's Locals and Tourists, we want your data viz videos, imagery and cartography. (Be sure that you have rights to the underlying data and that you attribute it properly.)

    2. Create an interactive RFID installation

    Inspired by Mediamatic, each attendee will have an RFID tag that can be paired with our conference social network. If an attendee swipes his or her tag, you'll be able to:

    • Fetch info about the owner of a swiped badge.
    • Show the owner of a swiped badge where they are supposed to be next, according to their personal schedule.
    • Send the owner of a swiped badge a message via the attendee directory.
    • Make two owners of swiped badges contacts within the attendee directory.

    3. Mini Maker Faire: Hardware project

    The Where Mini Maker Faire will take place on Wednesday, April 4. We're interested in any hardware project that is in the geo/location/sensing space, particularly ones that feature:

    • Kinect/Computer Vision Arduino/Lilypad/ADK Processing for Android
    • Beagle Board/Panda Board
    • NFC/RFID
    • Gadgeteer Wearables
    • ROBOTS!!!!

    Mini Maker Faire setup includes a four-inch skirted, countertop-level table, Wi-Fi and power.

    Acceptances will be rolling. The deadline to get your proposal in is March 1, so apply soon. If your project is accepted for any of the above, you'll receive a pass to Where.

    Where Conference 2012 — O'Reilly's Where Conference, being held April 2-4 in San Francisco, is where the people working on and using location technologies explore emerging trends in software development, tools, business strategies and marketing.

    Save 20% on registration with the code RADAR20

    October 11 2011

    Why indoor navigation is so hard

    Map of the Air and Space museum in Washington, D.C. Remember the days before you could pull your smartphone out of your pocket and get instant directions from your current location to anywhere in the world? It's kind of foggy for me, too.

    In fact, I'm so used to relying on my smartphone that I feel increasingly flustered when wandering the aisles of Costco, locating the elephant house at the zoo, or searching for decent food at the airport. Shouldn't my magical pocket computer help me with this, too?

    The answer is "yes," of course. But there are challenges to implementing indoor navigation today.

    User interface

    The maps app on your smartphone has one primary concern: getting you from 106 Main Street to 301 Sunny Lane, or from work to home, or from home to Taco Bell. Why are you going to Taco Bell and what percentage of your taco beef will be meat filler? The app doesn't need to know. Thus, the typical interface for a smartphone maps app is a big map and a search box.

    You might assume that an indoor navigation app for, say, the American Museum of Natural History has the same primary concern: getting you from the main entrance to the T-Rex. But why go to the T-Rex? How do I know there's a T-Rex here anyway? And what if my kids have 20 things they want to see and we only have two hours to see everything? And what's going on this week — are there special exhibits?

    It turns out that creating a useful indoor navigation app requires more than navigation. So, an effective mobile UI should be more "smart guide" and less "paper maps" on your smartphone.

    It's a design challenge, like any other mobile app. Help visitors decide where they need to go first, then direct them there.


    Getting directions to the plumbing section of a store is certainly useful. But let's say you're looking for a particular Delta kitchen faucet. Wouldn't it be more useful to search in a retail app for "Delta faucet," check that it's in stock, then get directions right to that product? Who cares if it's in the plumbing section or the kitchen section?

    To be truly useful, an app needs to integrate with dynamic data.

    Similarly, a university campus app could offer to guide a student to "Kennedy Hall Room 203," but wouldn't it be better to search for "Econ 101" instead? Who cares where Econ 101 takes place today? Even better, just have students enter their name once, fetch their schedule, and automatically take them to whatever their next class is. Why make users do more work than they have to?

    Current location

    OK, so you decide you want directions to that Delta faucet I mentioned earlier. Ideally, the app will automatically start from your current location.

    Now comes the great sadness: GPS, as you may know, does not work indoors. The satellite signals are just too weak to penetrate anything much thicker than the metal roof of your car.

    However, all modern smartphones have Wi-Fi built in, and wireless networks are common enough in indoor spaces that an app could easily scan for known access points and calculate your position using trilateration.

    Here's the catch, however: Unlike the wide open world of Android, developers on the iPhone side aren't allowed to perform these Wi-Fi "signal scans."

    Fortunately, there are alternatives. One approach is to make the building do the work instead of the device. Some Wi-Fi installations, such as the Cisco MSE, can determine the location of any wireless device in the building. The access points themselves listen for the Wi-Fi signals created by your phone, then estimate its position via trilateration. This solution has been deployed successfully at a few locations, including at the American Museum of Natural History.

    Designing for inaccuracy

    One consequence of most indoor positioning systems is a lower degree of accuracy compared to GPS. For instance, indoor systems can usually guess which room you're in, and that's about it. Precision depends on signal fluctuations, which depend on factors like how many people are in the room, how you're holding your phone, and other vagaries.

    An effective mobile app must design for this reality from the very beginning. One technique that will help users greatly is to point out quickly recognizable features of the environment.

    The Meridian app, for example, uses a short text label to describe each direction step. (Disclosure: I'm the CTO and co-founder of Meridian.) Below, "Rose Room" is clearly marked in the "real world" space and easy to spot, as are the stairs headed down.

    Meridian app
    The Meridian app uses step-by-step text labels.

    The best way to combat inaccuracy, however, is by making it as easy as possible for users to self-correct. In the Meridian app, the map can easily be dragged, rotated, zoomed in and out, and the turn-by-turn steps can be flipped through with ease. If the starting location isn't perfect, the user will instinctively drag around and figure it out.

    Putting it all together

    Building amazing indoor app experiences is not only possible, it's already happening. This year alone, many places — from stadiums and retailers to museums and corporate campuses — have launched apps that are used by hundreds of people every day for navigation and to access location-based content.

    Indoor Wi-Fi positioning technology isn't a research project anymore; it's out there and works with the devices we all now carry. With the right user interfaces, it can be just as effective as GPS is outdoors.

    It's time to spread the incredible experience of wandering around a place as enormously complex as the History Museum without ever feeling lost.


    August 03 2011

    Developer Week in Review: Lion drops pre-installed MySQL

    A busy week at Casa Turner, as the infamous Home Renovations of Doom wrap up, I finish the final chapters of "Developing Enterprise iOS Applications" (buy a copy for all your friends, it's a real page turner!), pack for two weeks of vacation with the family in California (Palm Springs in August, 120 degrees, woohoo!), and celebrate both a birthday and an anniversary.

    But never fear, WIR fans, I'll continue to supply the news, even as my MacBook melts in the sun and the buzzards start to circle overhead.

    The law of unintended consequences

    Lion ServerIf you decide to install Lion Server, you may notice something missing from the included software: MySQL. Previous releases of OS X server offered pre-installed MySQL command line and GUI tools, but they are AWOL from Lion. Instead, the geek-loved but less widely used Postgres database is installed.

    It seems pretty obvious to the casual observer why Apple would make this move. With Oracle suing Google over Java, and Oracle's open source philosophy in doubt, I know I wouldn't want to stake my bottom line on an Oracle package bundled with my premiere operating system. Apple could have used one of the non-Oracle forks of MySQL, but it appears they decided to skirt the issue entirely by going with Postgres, which has a clear history of non-litigiousness.

    Meanwhile, Oracle had better be asking themselves if they can afford to play the games they've been playing without alienating their market base.

    South Korea fines Apple 3 million won, which works out to ...

    Apple has bee been hit with a penalty from the South Korean government that's a result of the iPhone location-tracking story that broke earlier this year. Now, Apple may have more money than the U.S. Treasury sitting in petty cash right now, but it will be difficult for them to recover from such a significant hit to their bottom line: a whopping 3 million won, which works out to a staggering ... um ... $2,830. Never mind.

    Strata Conference New York 2011, being held Sept. 22-23, covers the latest and best tools and technologies for data science -- from gathering, cleaning, analyzing, and storing data to communicating data intelligence effectively.

    Save 20% on registration with the code STN11RAD

    Java 7 and the risks of X.0 software

    Java 7 was recently released to the world with great fanfare and todo. This week, we got a reminder why using an X.0 version of software is a risky endeavor. It turns out that the optimized compiler is really a pessimized compiler, and that programs compiled with it stand a chance of crashing. Even better, there's a chance they'll just go off and do the wrong thing.

    Java 7 seems to be breaking new ground in non-deterministic programming, which will be very helpful for physics researchers working with the Heisenberg uncertainty principle. What could be more appropriate for simulating the random behavior of particles than a randomly behaving compiler?

    Got news?

    Please send tips and leads here.


    June 30 2011

    Strata Week: Google Plus focuses on data control

    Here are a few of the data stories that caught my attention this week.

    Your data and Google+

    Google PlusIt's hard to ignore the big story of the week: the launch of Google+, Google's latest stab at social. Google+ is comprised of several pieces, namely Circles, Sparks, Hangouts, Mobile, and Huddle — content and photo sharing, video chat, and mobile messaging. It's an ambitious project to be sure, particularly — as most pundits are quick to point out — with Google's less-than-stellar track record in the social space. The reviews a few days in seem mostly positive, with the observation all around that what Google needs to be successful here isn't simply a good user experience, but, well, users.

    The approach that Google has taken with Google+ purposefully differentiates it from other social networks, and the emphasis is on users' control of their own personal information. Google describes those other networks as "sloppy," "scary" and "insensitive." Rather than utilizing the blunt instrument of "friend" or "follower" to describe all relationships, Google Circles allows users to classify them on a more granular level: not simply "friend" or "family" or "acquaintance," but also self-created labels.

    Google says this is part of its larger effort to give users better control of their own data (see video below). What remains to be seen is if that's something most people are interested in, particularly if it means reassembling relationships and designing Circles on yet another social network.

    (Google's Joseph Smarr, a member of the Google+ team, will discuss the future of the social web at OSCON. Save 20% on registration with the code OS11RAD.)

    Yahoo spins off its Hadoop division

    Rumors have been circulating for some time that Yahoo was planning on spinning its Hadoop division into its own separate company, and this week Yahoo and Benchmark Capital announced the formation of Hortonworks to do just that. The news, first reported by GigaOm's Derrick Harris, means that a small team of engineers from Yahoo will create a separate company to provide support and services for Hadoop users.

    Harris writes:

    By incorporating next-generation features and capabilities, Hortonworks hopes to make Hadoop easier to consume and better suited for running production workloads. Its products, which likely will include higher-level management tools on top of the core MapReduce and file system layers, will be open source and Hortonworks will try to maintain a close working relationship with Apache. The goal is to make HortonWorks the go-to vendor for a production-ready Hadoop distribution and support, but also to advance Yahoo's repeated mission of making the official Apache Hadoop distribution the place to go for core software. Earlier this year, Yahoo discontinued its own Hadoop distribution, recommitting all that code and all its development efforts to Apache.

    Hortonworks, which takes its name from the elephant in Dr. Seuss's "Horton Hears a Who!", will compete with others in the commercial Hadoop space, including Cloudera.

    Strata Conference New York 2011, being held Sept. 22-23, covers the latest and best tools and technologies for data science — from gathering, cleaning, analyzing, and storing data to communicating data intelligence effectively.

    Save 20% on registration with the code STN11RAD

    iPhone tracking: The book

    iPhone trackThe buzz has died down substantially from the iOS location news that Pete Warden and Alasdair Allan broke here on Radar earlier this year. But it's not gone altogether. Such are the hopes of author James Bridle who has self-published his own personal mapping history in "Where the F**k Was I?"

    The hardcover book costs $160, so I'm not too sure it's destined to be a bestseller. But the idea is brilliant nonetheless. The book is 202 pages long, with a separate page for each day between June 2010 and April 2011. Each page is a map, with more than 35,000 of Bridle's locations mapped via OpenStreetMap, along with a note about what he did that day.

    As Nate Hoffelder from The Digital Reader points out:

    He's taken digital data that was created by spying on him and he's converted it to an analog form. He's also selling the data that Apple took for free — data that was recorded surreptitiously by one party, and now anyone can have it.

    Got data news?

    Feel free to email me.


    May 25 2011

    Want to know where to build a new store? Check your human density data

    SpotRankLogo.jpgSpotRank, a data service offered by Skyhook, taps hundreds of millions of location points to predict "the density of people in predefined urban square-block areas worldwide at any hour, any day of the week."

    In a recent interview, Ted Morgan (@tedmorgan), co-founder and CEO of Skyhook, discussed potential applications for density data.

    The SpotRank data is incredibly rich and it's also new. So, there's a lot of experimentation going on and pilot [programs] using it as ways to suggest hot places to go at the moment or even the best time to go to a restaurant before it gets busy.

    A lot of brands and advertising agencies are looking at ways they can target based on this kind of human density data. We're working with retailers as they figure out where to put their next store based on pedestrian foot traffic.

    Daytime movement and demographic data is something that has never existed before. It's a gold mine, except it takes a while to change how everyone's been using people data. It's still in the early days, but it's incredibly valuable, and it's frankly why all the big platforms are fighting to protect and control their own location data — they know this SpotRank-like information is going to drive most of the marketing and planning and behavioral analysis going forward.

    For more of Morgan's thoughts on location data and services, check out the full interview in the following video:

    Android Open, being held October 9-11 in San Francisco, is a big-tent meeting ground for app and game developers, carriers, chip manufacturers, content creators, OEMs, researchers, entrepreneurs, VCs, and business leaders.

    Save 20% on registration with the code AN11RAD


    May 05 2011

    Softly buzzing phones could yield better augmented reality

    Current augmented reality technology primarily relies on the cameras in smartphones to connect users to surrounding environments. But is this interaction actually disconnecting users from reality rather than plugging them in? In a recent interview, Foursquare co-founder Dennis Crowley said the camera sometimes gets in the way.

    The screen shouldn't be between you and me, or me and the place. If the phone buzzes and I look at it silently and know what's going on, I think that's the experience Foursquare is going after. It's less to do with the screens in front of people's faces ...

    Crowley said a more subtle interaction could actually result in a deeper connection to the surrounding environment without interfering with what's happening in real life.

    Augmented reality, like holding up your phone and looking around? I'm not very bullish on that. But, you basically get the same results if you put the phone in your pocket, and it buzzes when you're near things — that's like a poor man's augmented reality, which I'm very bullish on.

    For more of Crowley's thoughts on mobile and location, check out the full interview in the following video:

    Disclosure: O'Reilly AlphaTech Ventures is a Foursquare investor.


    April 27 2011

    The iPhone tracking story, one week later

    By Alasdair Allan and Pete Warden

    It's now been a week since we published the iPhone tracking story, so it seemed a good time to cover what we've learned.

    The fix

    iPhone trackApple has just released a Q&A covering this problem and they will be fixing the issues we spotted with a software update. "The reason the iPhone stores so much data is a bug we uncovered," Apple notes in the statement.

    Apple explains that nearby locations are pulled down from an Apple database and stored on the phone. These locations are from a "crowd-sourced database of Wi-Fi hotspot and cell tower data." This matches the picture that was emerging from research. It explains why there's lots of locations that don't match towers, and also why the accuracy is within a few-hundred meters, since we've learned that "micro-cells" in urban areas are clustered closely together.

    The Q&A explains the technical workings behind the log and reassures us that only anonymous data is sent back. Our conclusions still apply.

    Apple doesn't address our claim that this reveals sensitive information about your travels. At this point we're just relieved to get an explanation and a fix, but people can examine their own data and decide for themselves how happy they would be sharing it with strangers.


    What Does Your iPhone Know About You? More Than You Think — Alexis Madrigal has written a fascinating follow-up piece covering the data that professionals can read from your phone. Using forensics tools like the Lantern program that Alex Levinson helped build, anyone with physical access to the device can construct a picture of the user's life. It's eye-opening what the "law enforcement, government, and corporate examiners" who purchase the system can uncover about your behavior.

    The Tell-all telephone visualization also makes for thoughtful viewing. It's built from details that a German politician forced his cell phone provider to share after it was caught storing six months of location data on its subscribers. I think one of the reasons that the iPhone Tracker application has had so much use is that it shows people their own data in an understandable way. Unfortunately, that means that similar information that's harder to access behind a company's firewall may not get the same scrutiny, just because it's harder to show in a way that connects with people.

    Uses for good

    I've long been a fan of Geoloqi's opt-in service for recording and sharing your travels, but several other projects in the same area have appeared in my inbox over the last few days. Maria Scileppi has created the Living Brushstroke project (see video below) to capture people's movements at events, and turn the data into art. Intriguing and beautiful patterns emerge as people cross paths. It's a very fresh way to look at our lives.


    April 24 2011

    Additional iPhone tracking research

    By Alasdair Allan and Pete Warden

    Here's the latest developments on iPhone tracking.

    Android records a short log

    The Guardian has a good overview of Android's equivalent to consolidated.db. It records the last 50 cell locations, and the last 200 Wi-Fi networks, but older entries are overwritten. As we mentioned in our original video, this was what we expected on the iPhone when we found the file, and it was the sheer scale and duration of the recording that floored us, along with how easy it was to access on your computer. Android doesn't appear to copy the file over when you sync, so you'd need physical access to the phone to read it.

    Phoning home your location

    In the Wall Street Journal there's a good story covering how phones often send your location back to servers at both Apple and Google. We've known that cell companies are gathering this kind of data, because they need it for their basic operations, but the most interesting question for me is how it's actually stored by these software companies. If it's truly just for improving their location services, it could be anonymized so that it would be hard to figure out an individual's movements if you had the data. Even if it's not, the data is somewhat protected when it's on a company's internal network, since that keeps it further out of reach than a file that's held on your machine.

    Better for tracking travel than home or office locations

    Sean Gorman and my friend Peter Batty have done some impressive work digging into the details of the location data. Their conclusion is that it's hard to spot locations where you spend a lot of time in the same place, like your house or place of work. It's almost as if re-visiting the same spot overwrites a lot of the older data for that place, which would fit with a lot of what we've seen. They also try to quantify the accuracy of the location, pointing out how many outliers appear.

    Even just showing where you've been traveling to is pretty concerning, but it's good to rule out some malicious uses. The work they've done gives us a lot more about the characteristics of the data, I'm looking forward to seeing more of this kind of analysis.

    Intriguingly, their work also has some support for Will Clarke's idea that the locations are associated with cell towers. Peter's data shows a cluster around Mile High Stadium, which he hasn't visited recently but which does have a lot of cell infrastructure. Sean has another map that overlays actual tower locations with his points, and it's clear they don't coincide, but could well be triangulated from multiple towers. Sean's observation fits with our initial hypothesis that the locations are the result of sometimes-inaccurate triangulation from towers, but Peter's is evidence that there's a bias in the data to clustering around tower positions.

    Peter is investigating the WiFiLocation table. This typically contains a lot more points than the cell version, with 219,000 entries in Alasdair's data versus only 29,000 cell points. We didn't visualize this in the application because the derived lat/long points are a lot noisier, but that may be an issue with the quality of the location-lookup tables Apple are using since they switched away from SkyHook. It appears to record the ID of many of the WiFi networks you've come into range of, so I'll be interested to see what Peter and others discover about this data.


    April 22 2011

    iPhone tracking: The day after

    By Alasdair Allan and Pete Warden

    iPhone trackI don't think either of us were expecting to see this story strike such a nerve. There's been some amazing detective work from researchers across the web, and so here's a selection of the most interesting immediate reactions.

    Alex Levinson — Right from launch, we had an FAQ pointing to articles by people like Ryan Neal and Paul Courbis who had found this file (consolidated.db) before, but hadn't understood or been able to communicate its significance. The main reason we went public with this was exactly because it already seemed to be an open secret among people who make their living doing forensic phone analysis, but not among the general public — even pretty geeky people like Alasdair and me. We were freaked out by the implications of this data and how unprotected it was, but most of the forensics community seemed to miss quite how creepy ordinary people would find it.

    I do appreciate how frustrating this must be for Alex though, and would like to apologize personally to him that we didn't include his article among the prior research we cited. Unlike the others, it didn't show up in web searches or the books we referenced. It also didn't help that most of the follow-up articles by other people left out the details that we'd tried to make clear about who found it first. We obviously didn't communicate it as well as we thought we had, which is completely our fault.

    My Life According to the iPhone's Secret Tracking Log — Alexis Madrigal has a far more interesting life than me, judging by his map. I especially like the points from a flight with Jim Fallows somewhere over West Virginia. As he says, this data can be incredibly interesting, and as data geeks we were just as fascinated as he is. I actually look forward to a future where we can use this sort of information, but with the user's permission.

    Apple is not “recording your moves” — Both of us have been following Will Clarke's blog for a while and we liked this article. It's good to look skeptically at the accuracy of the data both in space and time. We do disagree about one of the conclusions though: that the points are just the locations of cell towers. That was one of our first thoughts when we saw the data. But the fact that there's thousands of different points scattered across small areas, all in slightly different places, seems like pretty strong evidence that they're not just the locations of cell towers. Another way of putting that is that there's a lot more points than there are towers. There's also lots of points with the same tower ID code that are in different locations. That all led to our conclusion that it was trying to figure out the device's position, even if it wasn't very good at it.

    Until we get a deeper analysis, that's just a provisional conclusion of course. But getting smart folks like Will to dig into this and correct anything we've got wrong is exactly why we open-sourced it. He also picks up on the Las Vegas Anomaly. Multiple people have reported seeing a phantom trip to the city show up, and one theory (other than a lot of lost weekends) is that Apple has an unpacking or testing facility there. Alasdair's phone that was shipped with iOS4 shows this, whereas my older device that originally had iOS3 doesn't, which was suggestive. I wonder if Will's device is a newer one, too?

    OpenStreetMap — The application we released relies on this volunteer-run site to render the background map tiles. We ended up tripling their usual load, according to a team member. They actually fired up extra servers to cope, so I made sure to add a link to their donation page from our main site. If you got something out of the application, please do consider giving something to them, or even getting involved. It's a fantastic team and community. How many other organizations would have responded to heavy usage by a free client by paying for more servers themselves? I even messed up their credit text on the initial version of the application, but they were very understanding about that too.


    April 21 2011

    Data News: Week in Review

    The Where 2.0 Conference was held April 19 - 21 in Santa Clara, Calif., so it's no surprise there were plenty of location-based developments to talk about this week in the data space. Here are a few of the data stories — place-based and otherwise — that caught my eye.

    Your iPhone tracks your location

    iPhone trackOn Wednesday, Pete Warden and Alasdair Allan made headlines with the story of their discovery of an iPhone file that tracks its owner's location. The iPhone appears to use cell-tower triangulation to periodically record user's latitude and longitude, storing the data in a file that lives on the iPhone and is transferred to a user's computer when the device is synced.

    According to their research, the file appears to be part of iOS 4 update, as that's the point from which the recordings start. While the existence of the file raises some questions — what are Apple's plans for this data — more disconcerting may be that the file is unencrypted, leaving this trove of location data stored locally but unprotected. Apple doesn't transmit the data, it appears, but no other device seems to have a comparable file, according to Warden and Allan.

    While there are questions about privacy and security here, the data is quite compelling, thanks in no small part to the iPhone Tracker tool Warden and Allan have built that will read this file on a user's computer and visualize their movements. Your phone has surreptitiously been tracking you, but the maps replay a fascinating and fairly accurate record of where you've travelled since June 2010.

    Crowdsourced data versus "real statistics"

    Ushahidi co-founder Eric Hersman wrote a strong defense of crowdsourced data this week in his post, "The Immediacy of the Crowd." His blog post served as a response to one that appeared last month on the social enterprise organization Benetech's blog. The title of the latter post -- "Crowdsourced data is not a substitute for real statistics" — probably demonstrates immediately why Ushahidi would object.

    The Benetech post (along with a subsequent Fast Company article) suggests that crowdsourced data from mobile phones and SMS can "lead rescue teams in the wrong direction" and that that data might not be good for statistical analysis or modeling.

    On one hand, this is an interesting and important academic debate here. Which is better, crowdsourced data or statistical patterns? Are there patterns in crowdsourced data that we can use, in aggregate or as predictions in real time?

    But the back and forth between the blogs, as Hersman observes in his post, overlooks an important element: Crisis response is messy and hardly a "clinical environment where we all get to sit back, sift data and take our time to make a decision."

    U.S. Senate finally releases its financial data ... in PDF

    It's been almost two years since the U.S. Senate agreed to make the official record of its expenditures publicly available online. This week the Senate finally revealed its plan to release the information. According to the Sunlight Foundation, the Senate will begin to release records in November. This will cover the period from April to September.

    But the data will be in PDF format. As the Sunlight Foundation notes with dismay:,

    The legislation was rather clearly intended to create the release of actual data, not data in the difficult-to-reuse form of a paper document. Unfortunately, PDF documents can meet the standard of searchable (as long as the text is exposed), and itemized (if the items are listed), so the Senate is getting by on a technicality, and reaching for the lowest common denominator.

    How do we demand more accessible, structured datasets? Or, how do we challenge the PDF?

    Got data news?

    Feel free to email me.


    April 20 2011

    Got an iPhone or 3G iPad? Apple is recording your moves

    By Alasdair Allan and Pete Warden

    Today at Where 2.0 Pete Warden and I will announce the discovery that your iPhone, and your 3G iPad, is regularly recording the position of your device into a hidden file. Ever since iOS 4 arrived, your device has been storing a long list of locations and time stamps. We're not sure why Apple is gathering this data, but it's clearly intentional, as the database is being restored across backups, and even device migrations.

    iPhoneTracker screen
    A visualization of iPhone location data. Click to enlarge.

    The presence of this data on your iPhone, your iPad, and your backups has security and privacy implications. We've contacted Apple's Product Security team, but we haven't heard back.

    What makes this issue worse is that the file is unencrypted and unprotected, and it's on any machine you've synched with your iOS device. It can also be easily accessed on the device itself if it falls into the wrong hands. Anybody with access to this file knows where you've been over the last year, since iOS 4 was released.

    In the following video, we discuss how the file was discovered and take a look at the data contained in the file. Further details are posted below.

    What information is being recorded?

    All iPhones appear to log your location to a file called "consolidated.db." This contains latitude-longitude coordinates along with a timestamp. The coordinates aren't always exact, but they are pretty detailed. There can be tens of thousands of data points in this file, and it appears the collection started with iOS 4, so there's typically around a year's worth of information at this point. Our best guess is that the location is determined by cell-tower triangulation, and the timing of the recording is erratic, with a widely varying frequency of updates that may be triggered by traveling between cells or activity on the phone itself.

    What are the implications of this location data?

    The cell phone companies have always had this data, but it takes a court order to access it. Now this information is sitting in plain view, unprotected from the world. Beyond this, there is even more data that we have yet to look at in depth.

    For example, in my own case I (Alasdair) discovered a list of hundreds of thousands of wireless access points that my iPhone has been in range of during the last year.

    How can you look at your own data?

    We have built an application that helps you look at your own data. It's available at along with the source code and deeper technical information.

    What can you do about this?

    As we note around the 20-minute mark in our video discussion, an immediate step you can take is to encrypt your backups through iTunes (click on your device within iTunes and then check "Encrypt iPhone Backup" under the "Options" area).


    April 14 2011

    ePayments Week: Android's predicted ascendance

    Here's what caught my attention in the payment space this week.

    In the future, half the world will be Android

    Well, half the smart phone world, anyway. Gartner Research is predicting that Google's Android operating system will continue its drive to become the most popular mobile OS in the world and, by the end of 2012, could own about half the market. Windows' mobile will pick up share, too, more than doubling to about 11% over the same time. Where will all that market share come from? Gartner sees Symbian dropping like a rock as Nokia ports over to Windows. Perhaps more surprising, Mashable's Ben Parr noted with a tad of skepticism, is Gartner's prediction that Apple's iOS will peak this year and begin to dip by next.

    Predictions are always a little sketchy, but there's no doubt Android is gaining share and significantly reshaping the smartphone and mobile markets by packaging its app-friendly platform onto inexpensive hardware. This week AT&T announced a prepaid smart phone, the LG Thrive, following the lead of Verizon, which began offering prepaid smart phones last autumn on RIM, WebOS and Android handsets. Google has also been working with telecom carriers to make buying apps from its marketplace as seamless as it is over at Apple's App Store. AT&T and Sprint will now add the app cost to your monthly bill, so no need for a credit card or even Google Checkout.

    Where is it all heading? Gigaom's Om Malik talked with Juniper Networks chairman Pradeep Sindhu to get his thoughts. In a brief conversation that ranges from the IBM 360 to the cloud, Sindhu made the point that smartphones as platforms will not only place greater demands on the network but will also change the way we think about it, blurring the lines between what is mobile and what is web. For a more hands-on and tactical view of where it's going, developers might want to look at the sessions in the Android track at next month's Google Developer Conference, May 10-11, in San Francisco.

    NFC in China

    Bus and subway riders in China have already been using prepaid cards equipped with near-field communications (NFC) technology to pay for their rides. With the infrastructure to receive their payments already in place, it should be a relatively easy switch to put the NFC card inside their phones instead of their wallets. NFC World's Christopher Brown said that Watchdata, which produces both types of cards, has produced more than 3 million NFC SIMM cards for mobiles, complete with a little antenna that sticks out of the card slot. The third million shipped in the past four months, Brown noted, so the pace is quickening. China Telecom has distributed most of the cards, but competitor China Unicom is also trying out the phone cards, launching a program in Beijing that goes beyond transit into theaters, markets and restaurants.

    No word on whether the external antenna is an annoyance. But we're pretty sure it's a temporary inconvenience since handsets in the not-too-distant future will probably put the NFC circuitry and its antenna inside the phone.

    Where 2.0: 2011, being held April 19-21 in Santa Clara, Calif., will explore the intersection of location technologies and trends in software development, business strategies, and marketing.

    Save 25% on registration with the code WHR11RAD

    Quova's Location Developer Challenge wants to know: Where U At?

    Quova, which offers ways to help websites know the locations of their visitors, is holding a Location Developer Challenge to promote its nascent developer program. If you have an idea for an interesting location-aware web app, or a fresh take on visualizing location data, take a look at the details. They'll be spurring on promising contestants with monthly spot bonuses of $500 and then sending the winner to the Future of Web Apps conference in Las Vegas June 27-29, to show off the winning app.

    While much of the buzz around geolocation centers around mobile apps, Quova's business focuses on fraud detection and geographically targeted ads for website visitors from desktop and laptop browsers. "We're not a mobile play," said company spokeswoman LaurieAnne Lassek in a phone interview, "but the web is not going away." Quova draws on a database of 3 billion publicly available IP addresses, mostly in North America and Europe. It opened its API last November and is just beginning to build a developer program. Registering for the program gives you access to the API and up to 10,000 queries on their data. "When you start to monetize the product, we work out a revenue-sharing deal," Lassek said.

    Note: Quova will be at next week's Where 2.0 Conference in Santa Clara, Calif.

    Got news?

    News tips and suggestions are always welcome, so please send them along.

    If you're interested in learning more about the payment development space, check out PayPal X DevZone, a collaboration between O'Reilly and PayPal.


    April 08 2011

    The convergence of biometrics, location and surveillance

    Applying biometric matching to location-based surveillance technologies produces both fascinating possibilities and scary scenarios.

    I recently spoke with Tactical Information Systems CTO Alex Kilpatrick and Mary Haskett, co-founder and president, about the state of biometrics and what we need to be concerned about as surveillance becomes more prevalent in our society. They'll expand on many of these ideas during a session at the upcoming Where 2.0 Conference.

    Our interview follows.

    What is biometric matching and how might it be used in future consumer applications?

    Alex KilpatrickAlex Kilpatrick: Biometrics is the science that studies things that make an individual unique. Interestingly enough, there are lots of things that make an individual unique, especially when examined closely enough. The most common is fingerprints, but the pattern of ridges on your palms and feet are just as unique. Your iris, the colored part of your eye, is extremely easy to read and unique, even among twins. The shape of your ear, the way you walk, the size of your hands, your smell, the way you talk, and of course your face are all unique.

    Biometric matching comes in two forms: verification and identification. Verification is when I come to a sensor and say "I am Alex Kilpatrick" and the sensor verifies that one of my biometrics, perhaps my face, matches the biometric on record. This type of matching is relatively easy. Identification is when I present a biometric to a system and ask it "Who is this?" That is a much harder problem, especially when there are hundreds of millions of records. The FBI deals with this problem every day, taking unknown fingerprints from crime scenes and looking for a match in their database containing millions of records.

    Like most technologies, biometrics can be used for good or evil. In consumer applications, it can be used to help prevent identify theft by providing strong authentication for financial transactions. It can be used for convenience — to allow you to login to a computer, or gain entry to your health club without having to remember a password or carry an ID. Disney World uses biometrics to prevent season pass holders from sharing their passes with other people. It can be used to identify people who can't identify themselves, such as Alzheimer's patients, autistic children, very young children, people in accidents, etc.

    Mary HaskettMary Haskett: I think the entire industry is holding its breath and waiting to see what's going to happen in the consumer space.

    The potential is enormous and current methods, like a card or code that you have to show or recite, are actually quite crude. But everyone is wondering if the consumer will accept it. Are they willing to use their fingerprints, face image, or iris image in these ways?

    Where 2.0: 2011, being held April 19-21 in Santa Clara, Calif., will explore the intersection of location technologies and trends in software development, business strategies, and marketing.

    Save 25% on registration with the code WHR11RAD

    How are biometrics being used in surveillance technologies?

    Alex Kilpatrick: In terms of "surveillance society" activities, there are only two players: face and iris. There are technologies such as gait — the way you walk — that may theoretically be used to monitor people, but these are not discriminating enough to uniquely identify individuals. For a biometric technology to be effective for covert surveillance, it has to be able to detect you from a distance without your knowledge. Currently, face matching is the only biometric technology that supports covert surveillance, meaning that you can be tracked without your consent. However, it is not robust in situations involving poor lighting, poor cameras, or poor angles. Face matching is very effective for "passport"-style controlled poses, but its accuracy drops off rapidly with shadows, or with the traditional angles used by current surveillance cameras.

    Outside of face, there are iris cameras available now that can detect an individual at distances up to 2-4 feet, but these require the cooperation of the individual — they have to look at the sensor.

    The bottom line, though, is that there are not any broadly effective biometric means of covert surveillance available right now. That will change in the near future, though.

    The most "promising" technology for covert surveillance is 3D face matching. This uses a three-dimensional model of an individual's face, so it is much more robust when matching partial faces, or faces at low angles from surveillance cameras. These systems are not currently effective for large populations, but research is progressing rapidly in this area. I would guess that this technology will be mainstream in the next 5-10 years.

    Iris technology may also advance to become effective without user participation as well. Your iris is effectively a barcode to a computer, and it provides a much better system for matching than a face. For iris to be effective for covert surveillance, better cameras with highly adaptive focusing systems will need to be developed.

    That said, the advancement of biometric technology may ultimately be a moot point because many people carry around a GPS tracker with them all the time in the form of a smartphone. These devices can track your location down to 10 meters if they use GPS, and 300 meters if they use triangulation. That may not be accurate enough to place you at a particular location, but it is enough to build a pretty accurate profile of your activities. The devices can be activated (including GPS) without your consent, if required by law enforcement or other authorities.

    What are some of the pros and cons of location-based surveillance tech?

    Mary Haskett: If you consider apps like Foursquare and Gowalla to be location-based surveillance technologies, and I do, then you've already experienced some of the benefits. I've run into people who were at the same cafe where I was eating and I was very happy to get a chance to connect with them. I've tried restaurants based on seeing people I know check-in and comment on their experiences. But I will also sometimes choose not to check-in at a restaurant because I don't want to hear my kids complaining "You had sushi for lunch? Why don't you ever take me when you get sushi?"

    Alex Kilpatrick: Like so many technologies, the advantages are a double-edged sword. When I travel, I can meet up with people in the cities I am visiting, but I also have a social obligation to meet up with those people, even when I may not feel like being social. I can track my kids on their way home from school, but they can never have the covert thrill of going somewhere they probably shouldn't go. A restaurant can offer me a free dessert for checking in, but I can also be targeted by a deluge of ads based on where I'm walking.

    The biggest disadvantage of these technologies is the continual erosion of the expectation of privacy. As people share more, there is less of a societal need to keep things private. This results in people sharing more than they should, as well as a troubling lack of concern about what companies are doing with their data.

    Scott Adams, of Dilbert fame, wrote a great blog post about living in "noprivacyville." In this thought experiment, he took privacy to an extreme — would you be willing to have every aspect of your life tracked in exchange for 30% lower cost of living? Insurance companies are already offering discounts if you are willing to have them install a GPS tracker in your car, and accept continual surveillance. Those companies can save a lot of money if they can see how you really drive, and adjust your costs accordingly. Many people would welcome 30% savings, and they will probably feel like they aren't changing their behavior at all. But it's a slippery slope. Once you accept surveillance in one aspect of your life, it's easy to let it slip into another. Before you know it you're paying a 30% premium to have a browser that doesn't track all of your web activity.

    The biggest danger to society is not the technologies themselves. Technology can be controlled. The biggest danger is that over time our society will just accept surveillance as part of the cultural landscape.

    How can people protect themselves from unwanted surveillance?

    Alex Kilpatrick: There are "dazzle" techniques that can be used to degrade face matching — contacts can block iris matching, a rock in your shoe will fool gait sensors, etc. However, all of these approaches are stop-gap measures. They don't address the fundamental root of the problem: Ultimately, as a society we have to decide that we will accept some risk in exchange for living in a truly free society.

    The biggest barrier to unwanted surveillance is knowledge and vigilance. Knowledge about where it is occurring, knowledge about what is being done with the information, knowledge about who is behind the surveillance and what their motivations are. Learn to ask "Why do you need this?" and "What if I leave this blank?"

    Vigilance is about protecting your privacy; don't share personal information unless there is some real value to sharing it, and only share information with the people who absolutely need the information in order to do their job, or with people you trust. Once you share the information, it's out of your hands. Trusting a company to protect it is not a viable long-term strategy. The only long-term strategy is building a society where people value privacy and are willing to fight for it.

    Associated photo (on home and category pages): Points by Vince Alongi, on Flickr


    Older posts are this way If this message doesn't go away, click anywhere on the page to continue loading posts.
    Could not load more posts
    Maybe Soup is currently being updated? I'll try again automatically in a few seconds...
    Just a second, loading more posts...
    You've reached the end.
    No Soup for you

    Don't be the product, buy the product!

    YES, I want to SOUP ●UP for ...