Newer posts are loading.
You are at the newest post.
Click here to check if anything new just came in.

June 20 2012

Four short links: 20 June 2012

  1. Researcher Chats To Hacker Who Created The Virus He's Researching -- Chicken: I didn’t know you can see my screen. Hacker: I would like to see your face, but what a pity you don’t have a camera.
  2. Economist on QR Codes -- Three-quarters of American online retailers surveyed by Forrester, a research firm, use them. In April nearly 20% of smartphone users in America scanned one, up from 14% in May last year.
  3. Reconstructing the Ruins of Warsaw -- what an amazing accomplishment!
  4. The Great German Energy Experiment (Technology Review) -- political will: the risk and the successes. Certainly a huge gulf between Germany and America in where they are, and political will to be more renewable.

June 19 2012

Four short links: 19 June 2012

  1. Mobile Maps (Luke Wroblewski) -- In the US, Google gets about 31 million users a month on its Maps app on iOS. On average those users spend more than 75 minutes apiece in the app each month.
  2. The Importance of Public Traffic Data (Anil Dash) -- Bill Gates and Paul Allen's first collaboration was a startup called Traf-O-Data, which recorded and analyzed traffic at intersections in their hometown using custom-built devices along with some smart software. Jack Dorsey's first successful application was a platform for dispatch routing, designed to optimize the flow of cars by optimizing the flow of information. It's easy to see these debates as being about esoteric "open data" battles with governments and big corporations. But it matters because the work we do to build our cities directly drives the work we do to build our communities online.
  3. Mozilla Thimble -- Write and edit HTML and CSS right in your browser. Instantly preview your work. Then host and share your finished pages with a single click.
  4. Design of the Guardian iPad App (Mark Porter) -- thoughtful analysis of the options and ideas behind the new Guardian iPad app. Unlike the iPhone and Android apps, which are built on feeds from the website, this one actually recycles the already-formatted newspaper pages. A script analyses the InDesign files from the printed paper and uses various parameters (page number, physical area and position that a story occupies, headline size, image size etc) to assign a value to the story. The content is then automatically rebuilt according to those values in a new InDesign template for the app. (via Josh Porter)

May 18 2012

Visualization of the Week: Urban metabolism

This week's visualization comes from PhD candidates David Quinn and Daniel Wiesmann, who've built an interactive web-mapping tool that lets you explore the "urban metabolism" of major U.S. cities. The map includes data about cities' and neighborhoods' energy usage (kilowatt per hour per person) and material intensity (kilo per person) patterns. You can also view population density.

resource_intensity.jpg
Click to see the full interactive version.

Quinn writes that "one of the objectives of this work is to share the results of our analysis. We would like to help provide better urban data to researchers." The map allows users to analyze information on the screen, draw out an area to analyze, compare multiple areas, and generate a report (downloadable as a PDF) with more details, including information about the specific data sources.

Quinn is a graduate student at MIT; Wiesmann is a PhD candidate at the Instituto Superior Técnico in Lisbon, Portugal.

Found a great visualization? Tell us about it

This post is part of an ongoing series exploring visualizations. We're always looking for leads, so please drop a line if there's a visualization you think we should know about.

Fluent Conference: JavaScript & Beyond — Explore the changing worlds of JavaScript & HTML5 at the O'Reilly Fluent Conference (May 29 - 31 in San Francisco, Calif.).

Save 20% on registration with the code RADAR20

More Visualizations:

April 18 2012

Four short links: 18 April 2012

  1. CartoDB (GitHub) -- open source geospatial database, API, map tiler, and UI. For feature comparison, see Comparing Open Source CartoDB to Fusion Tables (via Nelson Minar).
  2. Future Telescope Array Drives Exabyte Processing (Ars Technica) -- Astronomical data is massive, and requires intense computation to analyze. If it works as planned, Square Kilometer Array will produce over one exabyte (260 bytes, or approximately 1 billion gigabytes) every day. This is roughly twice the global daily traffic of the entire Internet, and will require storage capacity at least 10 times that needed for the Large Hadron Collider. (via Greg Linden)
  3. Faster Touch Screens More Usable (Toms Hardware) -- check out that video! (via Greg Linden)
  4. Why Microsoft's New Open Source Division (Simon Phipps) -- The new "Microsoft Open Technologies, Inc." provides an ideal firewall to protect Microsoft from the risks it has been alleging exist in open source and open standards. As such, it will make it "easier and faster" for them to respond to the inevitability of open source in their market without constant push-back from cautious and reactionary corporate process.

March 22 2012

Four short links: 22 March 2012

  1. Stamen Watercolour Maps -- I saw a preview of this a week or two ago and was in awe. It is truly the most beautiful thing I've seen a computer do. It's not just a clever hack, it's art. Genius. And they're CC-licensed.
  2. Screens Up Close -- gorgeous microscope pictures of screens, showing how great the iPad's retina display is.
  3. Numbers API -- CUTE! Visit it, even if you're not a math head, it's fun.
  4. China Now Leads the World in New iOS and Android Device Activations (Flurry) -- interesting claim, but the graphs make me question their data. Why have device activations in the US plummeted in January and February even as Chinese activations grew? Is this an artifact of collection or is it real?

March 09 2012

Four short links: 9 March 2012

  1. Why The Symphony Needs A Progress Bar (Elaine Wherry) -- an excellent interaction designer tackles the real world.
  2. Biologic -- view your social network as though looking at cells through a microscope. Gorgeous and different.
  3. The Cost of Cracking -- analysis of used phone listings to see what improves and decreases price yields some really interesting results. Phones described as “decent” are typically priced 23% below the median. Who would describe something they’re selling as "decent" and price it below market value unless something fishy was going on? [...] On average, cracking your phone destroys 30-50% of its value instantly. Particularly interesting to me since Ms 10 just brought home her phone with *cough* a new starburst screensaver.
  4. OpenStreetMap Welcomes Apple -- this is the classy way to deal with the world's richest company quietly and badly using your work without acknowledgement.

February 16 2012

Four short links: 16 February 2012

  1. The Undue Weight of Truth (Chronicle of Higher Education) -- Wikipedia has become fossilized fiction because the mechanism of self-improvement is broken.
  2. Playfic -- Andy Baio's new site that lets you write text adventures in the browser. Great introduction to programming for language-loving kids and adults.
  3. Review of Alone Together (Chris McDowall) -- I loved this review, its sentiments, and its presentation. Work on stuff that matters.
  4. Why ESRI As-Is Can't Be Part of the Open Government Movement -- data formats without broad support in open source tools are an unnecessary barrier to entry. You're effectively letting the vendor charge for your data, which is just stupid.

January 06 2012

Visualization of the Week: AntiMap

A new mobile phone app, AntiMap Log, allows users to record their own data as they move around. The app uses the phone's GPS and compass sensors to capture the following data: latitude, longitude, compass direction, speed, distance, and time.

While the AntiMap Log — available for both Android and iPhone — is the data-gathering component, it's just one part of a trio of open source tools. AntiMap Simple and AntiMap Video provide the visualization and analysis components.

AntiMap Video was originally designed to help snowboarders visualize their data in real-time, synced with footage of their rides. Here's a demo video:

That same snowboarder data is also used in the following visualization:

AntiMap snowboard visualization

AntiMap describes the visualization:

Circles are used to visualise the plotted data. The color of each circle is mapped to the compass data (0˚ = black, 360˚ = white), and the size of each circle is mapped to the speed data (bigger circles = faster) ... You can see from the visualisation, during heelside turns (left) the colours are a lot whiter/brighter than toeside turns (right). The sharper/more obvious colour changes indicate either sudden turns or spins (eg. the few black rings right in the centre).


Found a great visualization? Tell us about it

This post is part of an ongoing series exploring visualizations. We're always looking for leads, so please drop a line if there's a visualization you think we should know about.

Strata 2012 — The 2012 Strata Conference, being held Feb. 28-March 1 in Santa Clara, Calif., will offer three full days of hands-on data training and information-rich sessions. Strata brings together the people, tools, and technologies you need to make data work.

Save 20% on registration with the code RADAR20

More Visualizations:

September 15 2011

Global Adaptation Index enables better data-driven decisions

The launch of the Global Adaptation Index (GaIn) literally puts a powerful open data browser into the hands of anyone with a connected mobile device. The index rates a given country's vulnerability to environmental shifts precipitated by climate change, its readiness to adapt to such changes, and its ability to utilize investment capital that would address the state of those vulnerabilities.

Global Adaptation Index

The Global Adaptation Index combines development indicators from 161 countries into a map that provides quick access to thousands of open data records. All of the data visualizations at gain.globalai.org are powered by indicators that are openly available and downloadable under a Creative Commons license.

"All of the technology that we're using is a way to bring this information close to society," said Bruno Sanchez-Andrade Nuño, the director of science and technology at the Global Adaptation Institute (GAI), the organization that launched the index.

Open data, open methodology

The project was helped by the World Bank's move to open data, including the release of its full development database. "All data is from sources that are already open," said Ian Noble, chief scientist at GAI. "We would not use any data that had restrictions. We can point people through to the data source and encourage them to download the data."

Being open in this manner is "the most effective way of testing and improving the index," said Noble. "We have to be certain that data is from a quality, authoritative source and be able to give you an immediate source for it, like the FAO, WHO or disaster database."

"It's not only the data that's open, but also our methodology," said Nuño. "Data.WorldBank.org is a really good base, with something like 70% of our data going through that portal. With some of the rest of the data, we see lots of gaps. We're trying to make all values consistent.

Strata Conference New York 2011, being held Sept. 22-23, covers the latest and best tools and technologies for data science — from gathering, cleaning, analyzing, and storing data to communicating data intelligence effectively.

Save 30% on registration with the code ORM30

Node.js powers the data browser

"This initiative is a big deal in the open data space as it shows a maturing from doing open data hacking competitions to powering a portal that will help channel billions of investment dollars over the next several years," said Development Seed associate Bonnie Bugle in a prepared statement. Development Seed built the site with open source tools, including Node.js and CouchDB.

The choice of Node is a useful indicator, in terms of where the cutting edge of open source technology is moving. "The most important breakthrough is moving beyond PHP and Drupal — our initial thought — to Node.js," said Nuño. "Drupal and PHP are robust and well known, but this seems like the next big thing. We really wanted to push the limits of what's possible. Node.js is faster and allows for more connections. If you navigate countries using the data browser, you're just two clicks away from the source data. It doesn't feel like a web page. It feels native."

Speed of access and interoperability were important considerations, said Nuño. "It works on an iOS device or on a slow connection, like GPRS." Noble said he had even accessed it from rural Australia using an iPad.

Highlights from the GAI press conference are available in the following video:



Global Adaptation Index Press Conference: Data Browser Launched from Development Seed on Vimeo.

Related:

September 01 2011

Four short links: 1 September 2011

  1. A Chart Engine -- Android charting engine.
  2. The Illusion of Asymmetric Insight -- we are driven to create and form groups and then believe others are wrong just because they are others.
  3. Urban Mapping API -- add rich geographic data to web and non-web applications.
  4. Tell Us A Story, Victoria -- a university science story-telling contest.

August 31 2011

Four short links: 31 August 2011

  1. OSMdroid -- The OpenStreetMapView is a (almost) full/free replacement for Android's MapView class. Also see this tutorial. (via Simon Gianoutsos)
  2. 10 Immutable Laws of Security (Microsoft) -- an oldie but a goodie. Law #1: If a bad guy can persuade you to run his program on your computer, it's not your computer anymore.
  3. What's in The Trough? (BERG London) -- as a predictor or similar tool for action, the Gartner Hype Cycle is comically useless. As a tool for brainstorming, as BERG point out, it's fantastic.
  4. JP Rangaswami's Enterprise Gamification (Livestream) -- video of JP's "Enterprise Gamification" talk. As Kevin Slavin points out, the introduction is cheesily bad but the talk is pantswettingly good.

July 01 2011

Four short links: 1 July 2011

  1. paper.js -- The Swiss Army Knife of Vector Graphics Scripting. MIT-licensed Javascript library that gives great demo.
  2. TileMill for Processing -- gorgeous custom maps in Processing. (via FlowingData)
  3. Research Assistant Wanted -- working with one of the authors of Mind Hacks on augmenting our existing senses with a form of "remote touch" generated by using artificial distance sensors, such as ultrasound, to stimulate tactile stimulators (vibrating pads) placed against the surface of the head.. (via Vaughn Bell)
  4. GoldenORB -- a cloud-based open source project for massive-scale graph analysis, built upon best-of-breed software from the Apache Hadoop project modeled after Google’s Pregel architecture. (via BigData)

April 08 2011

Four short links: 8 April 2011

  1. A Practical Guide to Varnish -- Varnish is the http accelerator used by the discerning devops.
  2. Ferrofluid Sculptures (New Scientist) -- hypnotic video of an iron-based fluid that is moulded by magnetic fields, which I include for no good reason than it is pretty pretty science. (via Courtney Johnston)
  3. Twisted Highscores List -- clever leaderboard for tickets, reviews, commits, and fixes. A fun retro presentation of the information, rather than a determined effort to jolly up the grim task of software development by spraying on a thin coat of gamejuice. (via Jacob Kaplan-Moss)
  4. Beauty of Maps (YouTube) -- BBC's "Beauty of Maps" tv show is available in full on YouTube. Aspects of visualization and design here, as well as practical cartography. (via Flowing Data)

April 05 2011

Four short links: 5 April 2011

  1. The Big Map Blog -- awesome old maps, for the afficionado. (via Sacha Judd)
  2. sshuttle -- poor man's VPN built over ssh. (via Hacker News)
  3. Remembering LineDrive -- I, too, am bummed that LineDrive never became standard. And Maneesh, one of its cocreators. Check out his publications list!
  4. Websockets Pacman -- multiplayer Pacman, where players take the role of ghosts. All implemented with WebSockets in HTML5. (via Pete Warden)

March 07 2011

Four short links: 7 March 2011

  1. DigitalKoot -- Playing games in Digitalkoot fixes mistakes in our index of old Finnish newspapers. This greatly increases the accuracy of text-based searches of the newspaper archives. (via Springwise and Imran Ali on Twitter)
  2. Some Things That Need To Be Said (Amanda Hocking) -- A.H. is selling a lot of copies of her ebooks, and she cautions against thinking hers is an easily reproduced model. First, I am continuously overwhelmed by the amount of work I have to do that isn't writing a book. Middlemen give you time in exchange for money. Second, By all accounts, he has done the same things I did, even writing in the same genre and pricing the books low. And he's even a better writer than I am. So why am I selling more books than he is? I don't know. I'm reminded of Duncan Watts's work MusicLab which showed that "hits" aren't predictable. It's entirely possible to duplicate Amanda's efforts and not replicate her success.
  3. A Literary Appreciation of the Olson Timezone Database -- timezones are fickle political creations, and this is a wonderful tribute to the one database which ruled them all for 25 years.
  4. TileMill -- a tool for cartographers to quickly and easily design maps for the web using custom data. Open source, built on Mapnik.

March 03 2011

Four short links: 3 March 2011

  1. Guangzhou City Map -- Chinese city maps: they use orthographic projection (think SimCity) and not satellite images. A nice compromise for usability, information content, and invisible censorship. (via Hacker News)
  2. Broken Windows, Broken Code, Broken Systems -- So, given that most of us live in the real world where some things are just left undone, where do we draw the line? What do we consider a bit of acceptable street litter, and what do we consider a broken window? When is it ok to just reboot the system, and when do you really need to figure out exactly what went wrong?
  3. Android Malware -- black hat copied apps, added trojans, uploaded to Android Marketplace. Google were slow to respond to original developer's claims of copying, quick to react to security guy's report of malware. AppStores are not magic moneypumps in software form, no more than tagging, communities, or portals were. User contributions need editorial oversight.
  4. The League of Movable Type -- a collection of open source fonts, ready for embedding in your web pages.

December 08 2010

Four short links: 8 December 2010

  1. Send Us Your Thoughts (YouTube) -- from the excellent British comedians Mitchell and Webb comes this take on viewer comments in the news. (via Steve Buttry's News Foo writeup)
  2. Amazon proves that REST doesn’t matter for Cloud APIs -- with the death of WS-* and their prolix overbearing complexity, the difference between REST and basic XML RPC is almost imperceptible. As this essay points out, the biggest cloud API is Amazon's and it's built on RPC instead of REST.
  3. Kinect Piano (YouTube) -- turn any surface into a piano. (via David Ascher on Twitter)
  4. Google Maps Label Readability -- detailed analysis of the design decisions that make Google's labels so much more readable than the competition's. Fascinating to see the decisions that go into programmatically building a map: leaving white space around cities, carefully avoiding clustering, even how adding an extra level of information can make things simpler.

November 25 2010

Four short links: 25 November 2010

  1. A Day in the Life of Twitter (Chris McDowall) -- all geo-tagged tweets from 24h of the Twitter firehose, displayed. Interesting things can be seen, such as Jakarta glowing as brightly as San Francisco. (via Chris's sciblogs post)
  2. British Library Release 3M Open Bibliographic Records) (OKFN) -- This dataset consists of the entire British National Bibliography, describing new books published in the UK since 1950; this represents about 20% of the total BL catalogue, and we are working to add further releases.
  3. Gadgets for Babies (NY Times) -- cry decoders, algorithmically enhanced rocking chairs, and (my favourite) "voice-activated crib light with womb sounds". I can't wait until babies can make womb sound playlists and share them on Twitter.
  4. GP2X Caanoo MAME/Console Emulator (ThinkGeek) -- perfect Christmas present for, well, me. Emulates classic arcade machines and microcomputers, including my nostalgia fetish object, the Commodore 64. (via BoingBoing's Gift Guide)

November 23 2010

Four short links: 23 November 2010

  1. Goodbye App Engine -- clear explanation of the reasons why Google AppEngine isn't the right thing to build your startup on. Don't read the comments unless you want to lose faith in humanity. (via Michael Koziarski on Twitter)
  2. Neato Robotics XV-11 Tear-down -- the start of hackable LIDAR, which would enable cheap and easy 3D mapping, via a Roomba-like robovacuum with a LIDAR module in it. (via Chris Anderson on Twitter)
  3. Boilerpipe -- code to remove boilerplate wrappers from a webpage, returning just the text you care about. (via Andy Baio)
  4. Visual Eyes -- web-based authoring tool developed at the University of Virginia to weave images, maps, charts, video and data into highly interactive and compelling dynamic visualizations. (via Courtney Johnston's Instapaper feed)

November 11 2010

Strata Week: Life, by the numbers


This week we follow the path of data through cities, streets, and border conflicts.
We conclude our journey with a little brain work, as a programming challenge is announced
to automatically identify topics and trends in Twitter and Facebook updates. And don't forget registration
is now open for Strata 2011, our conference
about making data work.

Mining urban life

If you're interested in massive sources of data, you don't have to look further than daily life in a major urban center.

New York's Open311 Complaints
Screen from Pitch Interactive's illustration of Open311's call volumes.
November's Wired magazine features an in-depth look from Steven Johnson at what can be gleaned from 100 million calls to New York's 311 service. With up to 200 representatives manning the phones, the 311 call center fields around 50,000 calls every day. Calls range from inquiries about services to complaints about street lights, road conditions, and nuisance noise.

Every call to the 311 service is logged, and the data used to help city officials plan. While some trends are obvious, such as an uptick in call volume on holidays, Johnson reports that other more subtle and helpful patterns have emerged:

For example, officials now know that the first warm day of spring will bring a surge in use of the
city's chlorofluorocarbon recycling programs. The connection is logical once you think about it: The hot weather
inspires people to upgrade their air conditioners, and they don't want to just leave the old, Freon-filled units
out on the street.


It's not just the presence of patterns in the data, but also their disruption that can provide insights, such as clusters of complaints about unsanitary conditions in restaurants enabling city health officials to take rapid action.

Though highly successful, the New York City effort is just the start, as Johnson observes:

But even a city government like [Michael] Bloomberg's, which prides itself on entrepreneurial flair, needs to recognize the limits of its capacity to innovate. For every promising Scout map, there are hundreds of ideas for interesting civic apps lurking in the minds of citizens ...

In order for others to innovate, the 311 data should be openly available to build on, perhaps more open than the city authorities can manage. As Johnson writes, one candidate solution to this is Open311, a project from OpenPlans that aims to be a national 311 system by coordinating a standardized, open-access, read/write model for citizens to report non-emergency issues.

Uncovering the social effect of traffic

Strata RegistrationAnother project from OpenPlans is Streetfilms, whose mission is to encourage "livable streets" by producing films supporting community advocacy to help make roads work better for pedestrians and cyclists.

Streetfilms' phrase "livable streets" refers to the work of David Appleyard, whose research into how people experience streets with different traffic volumes was published in 1981. Appleyard's work advanced thought on traffic, and showed that heavy traffic has a strongly negative effect on social cohesion.

Revisiting Appleyard's 1981 work, Streetfilms hase created animated 3-D visualizations of the data collected from the 30-year0old urban planning study.

When errors go big

Whether streets or countries, one of the first uses of data in civilized history was in mapping: a prototypical story of data, its visualization, distribution and use in matters of state and the empowerment of citizens. A recent incident reminds us that errors in such valued data can have great consequences.

One of the most influential sources of mapping data at the moment is the ubiquitous Google Maps. As reported by Search Engine Land, an error in Google Maps recently exacerbated a territorial fracas between Nicaragua and Costa Rica.

A Nicaraguan commander, Eden Pastora, camped in an area on the border between the two countries, removed
the Costa Rican flag and planted that of his own country. The commander cited Google Maps as the justification
for his actions, and the mapping service was found to be erroneous in its placement of the border.

Addressing the error, Google Maps revealed that the problem arose in data provided by the U.S. Department of State. The Google response helpfully provides more than 100 years of historical explanation of the territorial dispute between Nicaragua and Costa Rica.

It's likely that the troops moved in before the error was used to justify the action, but this is a grim reminder that there is the potential for significant human cost from data errors.





Registration for Strata 2011 is now open. Save 20% with the code "STR11RAD"







Natural language processing bake-off


Joseph Turian of MetaOptimize has announced a competition in the field of natural language processing (NLP). The challenge is to construct a method for finding the top semantically-related terms over a vocabulary of several million words. The idea is to extract from a corpus a topic, which might be referred to in different ways.

What are the possible applications of such a method? Turian explains:

Increased insight into emerging topics, trends, and new products. Run this on social media updates (Facebook posts, Tweets) after collecting sufficient mentions of a topic, trend or product, and have more insight into what is being discussed.

The challenge is scheduled for the next two weeks, and those interested should head over to Turian's blog post for more details. Those who just want to learn the results of the challenge can sign up for the mailing list.

What's the prize for the fastest solution, aside from peer acclamation? Paying work for the winner, according to Turian.

Send us news

Email us news, tips and interesting tidbits at strataweek@oreilly.com.


Older posts are this way If this message doesn't go away, click anywhere on the page to continue loading posts.
Could not load more posts
Maybe Soup is currently being updated? I'll try again automatically in a few seconds...
Just a second, loading more posts...
You've reached the end.

Don't be the product, buy the product!

Schweinderl