Newer posts are loading.
You are at the newest post.
Click here to check if anything new just came in.

April 10 2012

Four short links: 9 April 2012

  1. E-Reading/E-Books Data (Luke Wroblewski) -- This past January, paperbacks outsold e-books by less than 6 million units; if e-book market growth continues, it will have far outpaced paperbacks to become the number-one category for U.S. publishers. Combine that with only 21% of American adults having read a ebook, the signs are there that readers of ebooks buy many more books.
  2. Web 2.0 Ends with Data Monopolies (Bryce Roberts) -- in the context of Google Googles: So you’re able to track every website someone sees, every conversation they have, every Ukulele book they purchase and you’re not thinking about business models, eh? Bryce is looking at online businesses as increasingly about exclusive access to data. This is all to feed the advertising behemoth.
  3. Building and Implementing Single Sign On -- nice run-through of the system changes and APIs they built for single-sign on.
  4. How Big are Porn Site (ExtremeTech) -- porn sites cope with astronomical amounts of data. The only sites that really come close in term of raw bandwidth are YouTube or Hulu, but even then YouPorn is something like six times larger than Hulu.

November 10 2011

Four short links: 10 November 2011

  1. Steve Case and His Companies (The Atlantic) -- Maybe you see three random ideas. Case and his team saw three bets that paid off thanks to a new Web economy that promotes power in numbers and access over ownership. "Access over ownership" is a phrase that resonated. (via Walt Mossberg)
  2. Back to the Future -- teaching kids to program by giving them microcomputers from the 80s. I sat my kids down with a C64 emulator and an Usborne book to work through some BASIC examples. It's not a panacea, but it solves a lot of bootstrapping problems with teaching kids to program.
  3. Replaying Writing an Essay -- Paul Graham wrote an essay using one of his funded startups, Stypi, and then had them hack it so you could replay the development with the feature that everything that was later deleted is highlighted yellow as it's written. The result is fascinating to watch. I would like my text editor to show me what I need to delete ;)
  4. Jawbone Live Up -- wristband that sync with iPhone. Interesting wearable product, tied into ability to gather data on ourselves. The product looks physically nice, but the quantified self user experience needs the same experience and smoothness. Intrusive ("and now I'm quantifying myself!") limits the audience to nerds or the VERY motivated.

Sponsored post

October 25 2011

What to watch for in mobile web apps

Mobile web apps are mostly defined by their "if onlys" — if only they had the feel of native apps, if only they could fully interact with devices, etc.

James Pearce, (@jamespearce), senior director of developer relations at Sencha, has a more optimistic take on mobile web apps. In the following interview, Pearce discusses promising web app technologies, and he explains why device APIs could make the web a lot more interesting. Pearce also weighs in on developers' current concerns, such as backward web app compatibility and testing across devices.

Our interview follows.

What are the most promising mobile web app technologies?

James_Pearce_Mug.pngJames Pearce: There are two significant technologies that I think we will see mature in the coming years. The first is WebGL, which provides a way for web developers to access the low-level graphic capabilities of the computer or device that the browser is running on. WebGL creates a huge opportunity to use web technologies to build games, to create high-performance simulations, and to develop other types of graphically heavy user interfaces — at least, when the browser supports it well. Currently, no default mobile smartphone browser offers good support, but I expect this to change soon.

Device APIs are also important because they allow web applications within a browser to query and interact with the device they're running on. These are not being implemented quickly, for various reasons, but the mobile web will take a huge step forward when web applications can interface with a device in much the same way native applications can. Imagine the possibilities that would come about through granting web apps access to a device's camera, contacts, calendar, messaging, and so on. The web will suddenly become a much more interesting place.

What are the best examples of HTML5 mobile apps? What lessons can and should be learned from them?

James Pearce: We have a directory at Sencha of some of the best examples, and this is just the tip of the iceberg. Of course there are some very well-known brands doing cool mobile things with web technology already — the Financial Times is a good example. We're seeing a whole spectrum of different types of apps and companies deploying services and applications with these tools and techniques.

Financial Times web app
Screenshot of the Financial Times web app

That said, it's still very early days, and I'm not sure we're ready to say we fully understand the best practices for building, deploying and promoting HTML5 web apps. But, certainly, we're seeing a lot of creativity around user experiences, and developers are pushing the bounds of what mobile devices can do. I guess if there's one thing we can learn, it is that the future is still very much up for grabs.

How much backward compatibility should mobile developers incorporate into their web apps?

James Pearce: This depends a lot on the type of app. If you are creating a relatively simple web application or site, then it's a good idea to employ techniques that allow it to function on down-scale devices. Normally, this means starting with simple content or markup and progressively enhancing the application in response to the additional capabilities the browser contains.

For full-fledged web applications, this is not at all easy to do, and at some point you need to draw the line with regard to the functionality the device must absolutely have in order to support your application's purpose. For example, a photo-sharing application is almost no use on a device with no camera API. A location-based application is pretty useless if the device can't determine its location and convey it to the browser, and so on.

What is the best way to test across devices without owning those devices?

James Pearce: That's an eternal challenge in mobile. The Apple and BlackBerry emulator tools are great. There's also one in the Android SDK, though sadly, of lower fidelity right now. Services like DeviceAnywhere allow you to remotely connect to devices — and Nokia runs a similar service — but ultimately, there's no alternative to the real thing to truly judge how your app's user experience shapes up.

TOC NY 2012 — O'Reilly's TOC Conference, being held Feb 13-15, 2012, in New York City, is where the publishing and tech industries converge. Practitioners and executives from both camps will share what they've learned and join together to navigate publishing's ongoing transformation.

Register to attend TOC 2012

This interview was edited and condensed.


July 14 2011

There are bigger issues surrounding the .gov review

The United States federal government is reforming its Internet strategy. In the context of looming concerns about the debt ceiling, high unemployment, wars abroad, rising healthcare costs, and the host of other issues that the White House and Congress should be addressing, that might seem like a side issue.

It's not. The federal government spends some $80 billion dollars every year on information technology. If you're paying any attention at all to government IT, you know that Uncle Sam is not getting his money's worth from that investment. Furthermore, the federal government has not been getting the kinds of returns in productivity or efficiency that the private sector has enjoyed over the past decade or so. Former White House OBM director Peter Orzag called that disparity the "IT gap" last year. So when the Obama administration launched a new initiative on Web reform this summer, it might have seemed a little overdue.

Better late than never, and better now than later.

Citizens are turning to the Internet for government data and services in unprecedented numbers, and they're expecting to find answers, applications and, increasingly, people. While public servants can join the conversations on social networks to address the latter demand, delivering improved information and e-services confronts the federal government with some tough choices, given budget constraints. That challenge is one reason that they're looking to the general public and the private sector for some ideas on how they can improve their strategy.

This week, in service of that goal, the White House hosted a livechat on improving federal websites with Macon Phillips, White House director of digital strategy, Vivek Kundra, the federal chief information officer, and Sheila Campbell, director of the GSA’s Center for Excellence in Digital Government. The chat, which has become a standard tool in the White House's online communications toolkit over the last year, included a livestream from, a Facebook chat and an active Twitter backchannel at the #dotgov hashtag. The White House also took questions through a form on and its Facebook wall.

These issues aren't new, of course, even if the tools for discussion have improved. And if you've been following the Gov 2.0 movement over the years this issue of how the government can use the Internet and associated technologies to work better has been at the core of the discussion throughout. Success online used to be measured by having a website, said federal chief information Vivek Kundra. As he observed immediately afterwards, "those days are long gone."

If the federal government is going to reform how it uses the Internet, it will need to learn and apply the lessons that Web 2.0 offers to Gov 2.0, whether it's standing up open government platforms, leveraging the cloud, crowdsourcing, or making data-driven policy.

Government is also going to need to stop creating a new .gov website for every new initiative, particularly if they're not optimized for search engines. There's some good news here: "Every month, historically, federal agencies would register 50 new domain names, said Kundra on Tuesday. "That's been halted."

This proliferation of federal .gov websites has been an issue for some time — call it ".gov sprawl" — and that what's driven the .gov reform effort in the context of the Obama administration's campaign to cut government waste. This week, for the first time, a dataset of federal executive branch Internet domains has been published as open government data online. The dataset of federal gov domains is hosted on and has been embedded below:

Federal Executive Branch Internet Domains

"This dataset lists all of the executive branch second-level domains within the top-level .gov domain, and which agencies own them," commented General Services Agency new media specialist Dan Munz in the Community page for the dataset. "As White House Director of Digital Strategy Macon Philips has pointed out (see ""), while many of these domain names point to sites that are valuable, some are duplicative or unnecessary. That makes it harder to manage the .gov domain, impairs agencies' ability to distribute information, and creates a user experience for citizens that just isn't as good as it could or should be. How can we fix that? Over the coming months, we'll have a plan for streamlining the federal executive branch webspace, and we want to invite you into the conversation. We're releasing this dataset as a first step, so that you can explore, comment, remix, and maybe even use the data to map the .gov domain in ways we haven't seen before."

OSCON Data 2011, being held July 25-27 in Portland, Ore., is a gathering for developers who are hands-on, doing the systems work and evolving architectures and tools to manage data. (This event is co-located with OSCON.)

Save 20% on registration with the code OS11RAD

Why reforming .gov matters

This effort is not impressing all observers. Micah Sifry, the co-founder of the Personal Democracy Forum, has called the move to delete redundant websites "cheap, dumb and cynical" at techPresident. "Redundant government websites probably cost the taxpayer a fraction of what we spend on military bands, let alone what we spend on duplicative and unnecessary government websites promoting the Army's, Navy's, Air Force's, Merchant Marine's, Naval Academy's, and Coast Guard's bands' websites! (According to NPR, the Marines spend $50 million a year on their bands, and the Army $198 million." In a larger sense, Sifry argued, "if you are really serious about eliminating stupid and pointless spending, then you'd be pushing for laws to strengthen protections for government whistleblowers (instead of going an stupid and pointless rampage to prosecute them!), since insiders know where the real waste is hidden."

Sifry is absolutely right on one count: the amount of money up for savings through reducing federal .gov websites is dwarfed by what is saved by, say, reducing Medicare fraud using new data analytics tools, or in finding cost savings in defense spending. Reducing the number of federal .gov websites by 90% would not significantly address the federal deficit. The biggest federal cost savings from this week's .gov livechat were likely cited by Kundra, when he said that 137 federal data centers were closed by the end of this calendar year, each of which consumes immense amounts of energy.

Where Sifry may have been overly harsh in his critique is in not acknowledging how progressive a perspective the White House appears to have embraced here. (Progressive meaning "forward-thinking," not political ideology, in this case.) Democratizing healthcare data so that it showed up in search engine results or is integrated into applications makes it more useful, argues Kundra, citing the improvements to Moving from a static website to a universe of applications and services provisioned by open government data is shifting from a Web 1.0 vision to 2.0 reality. In a country where 35% of citizens have a smartphone, delivering services and providing information to a mobile audience has to be factored into any online strategy, whether in the public or private sector. And, in most cases, it's the private sector that will be able to create the best applications that use that data, if government acts as a platform to empower civic coders. Phillips acknowledged that explicitly. "The best mobile apps," he said, "are going to be driven by the private sector making use of public data."

If e-government is going to move toward "We-government" — as Sifry has described the growing ecosystem of civic media, technology-fueled transparency advocates and empowered citizens — government data and services will need to be discoverable where and when people are looking for them. That is ultimately, in part, what getting .gov reform right needs to be about, versus straightforward cost-savings.

Kundra asked the broader community to "help us think through how we're going to provide services over the mobile Internet." If, as he said, search is the default way that people search for information now, then releasing high quality open data about government spending, the financial industry, healthcare, energy, education, transportation, legislation and campaign finance would be a reasonable next step. Tim O'Reilly has been sharing a simple piece of advice to the architects of platforms for years: "Don't make people find data. Make data find the people."

The .gov reform, in that context, isn't just about reducing the number of websites and saving associated design or maintenance costs. It's about reducing the need to ever visit a website to retrieve the information or access a citizen requires. In the years ahead, it will be up to Congress and Kundra's successor as federal CIO — along with whomever he or she reports to in the Oval Office — to get that part of "web reform" done.

May 04 2011

Four short links: 4 May 2011

  1. Maqetta -- open source (modified BSD) WYSIWYG HTML5 user interface editor from the Dojo project. (via Hacker News)
  2. Hacker News Analysis -- interesting to see relationship between number of posts, median score, and quality over time. Most interesting, though, was the relative popularity of different companies. (via Hacker News)
  3. Real Time All The Time (Emily Bell) -- Every news room will have to remake itself around the principle of being reactive in real time. Every page or story that every news organisation distributes will eventually show some way of flagging if the page is active or archived, if the conversation is alive and well or over and done with. Every reporter and editor will develop a real time presence in some form, which makes them available to the social web. When I say "will" I of course don't mean that literally . I think many of them won't, but eventually they will be replaced by ones who do. (via Chris Saad)
  4. Changes in Home Broadband (Pew Internet) -- Jeff Atwood linked to this, simply saying "Why Web 1.0 didn't work and Web 2.0 does, in a single graph." Ajax and web services and the growing value of data were all important, but nothing's made the web so awesome as all the people who can now access it. (via Jeff Atwood)

December 31 2010

What lies ahead: Gov 2.0

Tim O'Reilly recently offered his thoughts and predictions for a variety of topics we cover regularly on Radar. I'll be posting highlights from our conversation throughout the week. -- Mac

Is open government moving from theory to practice?

Tim O'ReillyTim O'Reilly: The initial rush of interest in open government and transparency is wearing off and people are getting to work. Gov 2.0 startup founders are figuring out business models -- such as advertising and providing back-end services for cities -- and the first crop of startups are being funded. Early entrants, like SeeClickFix and CrimeReports, are growing. I think we'll see a number of new startups in this space. [Disclosure: O'Reilly AlphaTech Ventures is an investor in SeeClickFix.]

Open government's transition is also leading to practical applications. The Blue Button initiative from Veterans Affairs, which allows veterans to easily download their medical records, is an idea that's bound to spread. Blue Button is a direct outcome of the Open Government Initiative, but that connection probably won't be recognized. As is so often the case, the things that really make a difference get put into a different category than those that fail.

Along those lines, people might say that open government failed because many of the items on the punch list didn't happen the way they were originally envisioned. When we look back, we'll realize that open government is not just about transparency and participation in government decision making, but the many ways that open data can be put to practical use.

There are profound open government projects taking shape. For example, the Department of Health and Human Services (HHS) could transform our healthcare system through open data and medical records. HHS Connect and The Direct Project are all about creating the standards for interoperability between medical records. We'll eventually see and benefit from larger efforts like these.

Another open data project that I'm fond of that started very early in the open government process is GTFS, the General Transit Feed Specification. That's the data standard that lets transit districts feed their bus and train arrival times to applications like Google Transit, or any of the many smartphone apps that help you plan your trip on public transit. This standard started as a collaboration between Google and the city of Portland, but is now available from many cities. It's a great example of how governments can think like platform providers. They have to equip their buses and trains with GPS, and report out the data. They could report it just to their own bus stops and train stations, or they could make it available to third parties to deliver in a hundred ways. Which is better for citizens? It's pretty obvious.

And of course, this is government data following in the footsteps of great open data projects of the past, such as the satellite weather data released by NOAA to power the world's weather forecasters, or even the GPS signals that were originally designed only for military use but then released for civilian use.

Much of what you're describing sounds like the Web 1.0-to-2.0 trajectory. Do you see similarities?

Tim O'Reilly: At the end of the Web 1.0 era, some people claimed the web had failed because banner advertising, pop-overs, pop-unders and all the increasingly intrusive forms of advertising didn't work. Then Google came along with a better idea. I think something similar will happen with Gov 2.0. Many of the things people label as "Gov 2.0" are really the early signals and efforts of a "Gov 1.0" period. Important shifts will eventually occur, and they won't have anything to do with government agencies putting up wikis or using Twitter. We will see many unexpected outcomes over time.

A collection of posts that look ahead to 2011 can be found here.


December 17 2010

Four short links: 17 December 2010

  1. Down the ls(1) Rabbit Hole -- exactly how ls(1) does what it does, from logic to system calls to kernel. This is the kind of deep understanding of systems that lets great programmers cut great code. (via Hacker News)
  2. Towards a scientific concept of free will as a biological trait: spontaneous actions and decision-making in invertebrates (Royal Society) -- peer-reviewed published paper that was initially reviewed and improved in Google Docs and got comments there, in FriendFeed, and on his blog. The bitter irony: Royal Society charged him €2000 to make it available for free download. (via Fabiana Kubke)
  3. Bixo -- an open source web mining toolkit. (via Matt Biddulph on Delicious)
  4. How Facebook Does Design -- podcast (with transcript) with stories about how tweaking design improved the user activity on Facebook. One of the designers thought closing your account should be more like leaving summer camp (you know a place which has all your friends, and you don’t want to leave.) So he created this page above for deactivation which has all your friends waving good-bye to you as you deactivate. Give you that final tug of the heart before you leave. This reduced the deactivation rate by 7%.

November 15 2010

Hiring trends among the major platform players

After re-reading Tim's post on the major internet platform players, I looked at recent hiring trends* among the companies he highlighted. First I examined year-over-year changes in number of job postings (from Aug to Oct 2009 vs. Aug to Oct 2010). Consistent with the recent flurry of articles about hiring wars, all the companies (except for Yahoo) increased** their number of job postings. Winning the battle for the Internet's points of control requires amassing talent:


Below is the breakdown by most popular occupations over the last three months:


[For a similar breakdown by location (most popular metro areas), click HERE.]

(*) Using data from a partnership with SimplyHired, we maintain a data warehouse that includes most U.S. online job postings dating back to late 2005. Since there are no standard data formats for job postings across employment sites, algorithms are used to detect duplicate job postings, companies, occupations, and metro areas. The algorithms are far from perfect, so the above results are at best extremely rough estimates.

(**) As a benchmark, the total number of job postings in our entire data warehouse of jobs grew 68% from Aug/Oct 2009 to Aug/Oct 2010.

October 28 2010

Getting closer to the Web 2.0 address book

Tim O'Reilly has often asked a good question: seeing as my phone and email know who I communicate with, my company directory knows who I work with, and Amazon knows who has written books for my company, why can't these various things somehow combine to automatically deal with friend requests coming from social networks?

This specific problem is just one example of a wider issue. Given that so much diverse and overlapping information about each of us is spread between many applications, why are seemingly simple things based on combinations of information -- like automatically reacting to known friend requests -- still not possible?

Tim summarizes this by asking "Where is the web 2.0 address book?"

The traditional reaction to Tim's need would be to begin work on the design of a new application to provide the suggested features. This might give rise to an online social address book incorporating his specific suggestions, along with anything else the application designers might dream up to provide additional functionality.

While such an application might be cool and work well in the short term, I believe that approach is doomed to failure. The main reason is that we can't anticipate the future. Baking decisions about what to support or not into applications is almost certain to leave us wanting and needing more. At that point we're beholden to application designers' willingness, priorities, and ability to further adapt.

What happens when Tim comes up with another idea, or when you or I do? While an application might be designed to be open -- for example, by providing an API or a plugin system -- at the end of the day it's just another application sitting in front of its collection of data. This problem is particularly severe in our specific case: Tim would ideally like the incorporation not just of his phone's data and his email, but specific information from the O'Reilly org chart and from Amazon. That level of specificity -- or personalization, if you prefer -- is well beyond the interests of anyone writing a general phone book application.

For these reasons, among others, I believe the traditional "let's build a cool new app" reaction to Tim's dilemma is the wrong approach.

The other answer

An alternate answer begins with a sharp departure from this thinking. I suggest that the Web 2.0 address book need not take the familiar shape of an application. Instead, like a physical address book, it might be mainly about the data. That is, centered in a common data store through which a variety of loosely coupled applications interact with users and with one another.

In such a world, a friend request to Tim would be added to a shared online data store. There it would be noticed by an application running on a mobile phone, a periodic script that scans Tim's inbox, or an internal O'Reilly script with access to the company's staff list. Any application recognizing the requesting user's details could add information to the request object to indicate recognition. The initiating application could detect this and act on it.

Several aspects of this idealized scenario are worth close consideration:

  1. Such a system would require an underlying storage -- a "data fabric" as Tim has called it -- that was shared not just for reading, but also for writing. The shared writable data fabric would be open to future applications. They could contribute or consume information as they saw fit, without requiring the permission of existing applications, possibly without knowing or caring about each other's existence. Future applications would interact seamlessly with existing ones by simply following established data conventions.
  2. While any application should able to join the fun and contribute, a permissions system would be needed to protect existing information and to selectively share it among authorized applications. For example, Tim might authorize an application on his phone to add information about numbers he has dialed or email addresses he has been in contact
    with. He could authorize another application -- in our case an automatic friend request resolver -- to read that information and to create more information on his behalf to indicate existing friendships.
  3. This data-centric answer to the "Where is the web 2.0 address book?" question points to a wide class of future applications which cooperate and interact not through synchronous pre-ordained API calls, but via asynchronous data protocols which follow open-ended conventions. This is in strong contrast to applications which hold information in databases with relatively inflexible structure behind APIs that strictly delimit possible interactions. Applications using a shared writable data storage can adopt conventions and data protocols by which they cooperate to achieve joint actions. Applications that operate in this way leave the door open for new conventions, for additional unanticipated data, and for future applications that adopt the existing conventions, or introduce new ones.

At Fluidinfo we're building a shared online "cloud" data system, called FluidDB, with the characteristics outlined above. FluidDB aims to usher in a new class of applications, as described above, in which data can be thought of as being social. The data allows diverse applications to interoperate and communicate asynchronously using shared conventions. [Disclosure: Tim O'Reilly is an investor in FluidInfo.]

In FluidDB, all objects may have information added to them by anyone or any application at any time -- with no questions asked. Whereas a more traditional approach locks down entire database objects, FluidDB has a simple and flexible permissions system that operates at the level of the tags that comprise objects. Objects are always writable, including by future applications and by idiosyncratic applications that know, for example, how to access the O'Reilly staff list.

To put things in a more general light, Tim's "How ridiculous is this?" complaint (see image, below) is symptomatic of a wider problem. It highlights the awkwardness of a computational world in which applications keep their data behind UIs and APIs that are designed for specific purposes. These prevent augmentation, sharing and search, and they ultimately prove inflexible. It is ridiculous that even simple operations combining information in obvious ways are still beyond our reach.

How ridiculous is this slide
Slide from Tim O'Reilly's "What is Web 2.0?" presentation at Web 2.0 Expo Berlin 2007.

Relief does not lie in the direction of more applications holding more data behind more APIs. It lies instead in allowing related data to coexist in the same place. Both Google and Wikipedia have demonstrated, in very different ways, the massive value that accrues from co-locating related data. There's a real need for a Wikipedia for data. Writable and with an object for everything, like a wiki, but also with permissions, typed data, and a query language to make it more suitable for applications. Such a store can hold the data, the metadata, support search across these, and provide a locus for applications to communicate.

The Web 2.0 address book then becomes a ball of data, not an application. Something we access as we please, using a collection of tools that individually suit us, and without any application having the final word on what's possible.


October 26 2010

Dancing out of time: Thoughts on asynchronous communication

I find it useful to try to think clearly about communication systems. This includes the ways in which they are synchronous or asynchronous (or a mixture), and the disruption that occurs when new technology allows us to replace synchronous systems with asynchronous ones, or to deliver new forms of asynchronous communication.

This post is the first of a two-part series. Below, I take a look at the core differences between synchronous and asynchronous techniques, and I suggest an alternative to traditional synchronous API-based communication between applications. Part two, coming later this week, will address Tim O'Reilly's question: "Where is the Web 2.0 address book?"

Synchronous and asynchronous communication

Synchronous communication requires that the participating parties -- information producers and consumers -- are simultaneously involved. Examples abound: telephone calls, in-person meetings, a parent teaching a child a song, an audience listening to a public lecture or a live concert, buyers and sellers negotiating prices at a market, the broadcasting and simultaneous consumption of live events, Morse code flashed between ships at sea, and the Gotham City bat signal.

Communication can also be asynchronous. In this case, information is somehow -- broadly speaking -- published and is only later consumed.

The history of technology has many examples of the tremendous impact that can result when a form of synchronous communication is replaced with something asynchronous. The most obvious example is the invention of writing, which allowed information to be written and consumed later. There are many familiar examples, such as CDs and DVDs, the VCR (and its modern derivatives like the TiVo and Boxee), voicemail, etc.

There are many other less obvious examples, too. Post-it Notes are asynchronous: they let us publish information in physical locations that we choose carefully so the intended recipient will likely encounter them later. Blog posts, including this one, are asynchronous. There are also plenty of non-human examples, such as dogs urinating on lamp posts.

Communication doesn't always divide cleanly into synchronous or asynchronous. Many forms of communication have some elements of both. How would you classify the Pony Express, smoke signals, sky writing, and telegrams? Susan Boyle's performance on "Britain's Got Talent" was experienced synchronously by a few thousand people, and asynchronously by 50 million. And when Charlie bit his brother's finger (again) in the back seat of the car, the event was experienced synchronously by just three people, but later, asynchronously, in 233 million online viewings.

The requirements of asynchronous communication

The main requirement for asynchronous communication is a mechanism for interim information storage. This can take many forms: wet concrete, the soft bark of a tree, a wall, or one's own skin. Along with storage, one must also have a means by which to deposit information. In our previous examples that would be a finger, a pen-knife, spray paint, and a tattoo needle.

It's difficult to overstate the impact of technology on our ability to communicate asynchronously. Modern digital systems provide us with virtually unlimited amounts of cheap storage, along with the means to efficiently deposit/retrieve information into/from this storage. Consider Slideshare as an example: You can post presentation slides and have them read asynchronously by thousands or even millions, rather than just seen by the dozens who might attend a presentation in person.

The other requirement is a set of conventions about the communication. These may be high-level and taken for granted. For example, the language of communication. Conventions are often more explicit, and can become specialized over time, like apartment ads: lg 3br apt, d/w & a/c, hdwd flrs. Conventions can also take the form of protocols, in which parties exchange information using the shared storage. Examples include quoting conventions in email threads, and people using @name to address others and check for mentions in Twitter.

Asynchronous communication lends itself to scale

Asynchronous communication almost always scales better than synchronous. Synchronization is usually centralized, complicated, and expensive because participants are often forced to waste time waiting.

The main requirements for asynchronous communication are things that modern technology is providing in abundance. Experiences that were formerly limited to a small set of synchronous participants are now regularly experienced asynchronously by a significant fraction of the people on the planet.

At smaller scales, there are many familiar examples -- including non-digital ones -- of how asynchronous communications scale so easily and cheaply. Putting up a billboard beside a busy highway is a cheap and easy way of delivering a message to many. Using "Wanted" posters is better than having the sheriff stand on the corner describing a fugitive to passers by. Having a mailbox in front of your house provides temporary mail storage, which ironically can be faster and more convenient than a synchronous meeting with a courier service who will put your package back in their truck if you're not home to sign for it. Higher tech examples are abundant, as mentioned above.

Because asynchronous systems scale so much better than synchronous ones, coming up with ways to replace the latter with the former can result in the creation of extraordinary value. For example, consider Twitter and YouTube.

APIs, data protocols, and FluidDB

My main interest in thinking about the possibilities of asynchronous communications is in the area of how applications communicate (and might come to communicate). This is often done via APIs that applications provide, and in much of the computational world communication between applications takes the form of synchronous calls to these API methods. I believe this presents an opportunity for an asynchronous alternative, and that the alternative can have an impact similar to those we've seen historically when synchronous systems are replaced by asynchronous ones.

In particular, I'm currently involved in building FluidDB, a new kind of database under development at Fluidinfo. Among other things, FluidDB aims to provide shared always-writable storage that applications can use to exchange information. [Disclosure: Tim O'Reilly is an investor in FluidInfo.] I believe that predefined API calls between applications are generally less flexible and powerful than asynchronous communications.

In the second part of this series, coming later this week, I'll give an example of asynchronous flexibility by offering an answer to Tim O'Reilly's question "Where is the Web 2.0 address book?" His question arises from a frustration with the lack of communication between applications. Application-independent shared writable storage and simple conventions for communication look to me like the best way forward.


October 07 2010

Four short links: 7 October 2010

  1. How to Manage Employees When They Make Mistakes -- sound advice on how to deal with employees who failed to meet expectations. Yet again, good parenting can make you a good adult. It’s strange to me that in the technology sector we have such a reputation for yellers. Maybe it’s business in general and not just tech. [...] People stay at companies with leaders who rule like Mussolini because they want to be part of something super successful. But it does tend to breed organizations of people who walk around like beaten dogs with their heads down waiting to be kicked. It produces sycophants and group think. And if your company ever “slips” people head STRAIGHT for the door as they did at Siebel. I’d love to see a new generation of tech companies that don’t rule through fear. (via Hacker News)
  2. Information Wants to be Paid (Pete Warden) -- I want to know where I stand relative to the business model of any company I depend on. If API access and the third-party ecosystem makes them money, then I feel a lot more comfortable that I'll retain access over the long term. So true. It's not that platform companies are evil, it's just that they're a business too. They're interested in their survival first and yours second. To expect anything else is to be naive and to set yourself up for failure. As Pete says, it makes sense to have them financially invested in continuing to provide for you. It's not a cure-all, but it's a damn sight better than "build on this so we can gain traction and some idea of a business model". Yet again, Warden reads my mind and saves me the trouble of finding the right words to write.
  3. 0Boxer -- Chrome and Safari extensions to turn gmail into a game. (via waxy)
  4. Twitter's New Search Architecture (Twitter Engineering Blog) -- notable for two things: they're contributing patches back to the open source text search library Lucene, and they name the individual engineers who worked on the project. Very classy, human, and canny. (via straup on Delicious)

October 06 2010

10 Lessons for Gov 2.0 from Web 2.0

What is Web 2.0? In 2005, it meant geeks embracing a set of principles and practices: using the web as a platform, harnessing collective intelligence, data is the new "Intel inside," and others. By 2010, many of the dominant companies and services that embody or fuel Web 2.0 have become global brands: Google, Craigslist, Amazon, Facebook, Twitter, and a host of new mobile communities or platforms. These companies are often defined by what they allow users to do: upload pictures or video, stay connected to friends, track and discover news, save bookmarks, create communities, etc.

For non-geeks, Web 2.0 meant the online world became a place you could publish and participate. It became about everyone with an Internet connection exploring an interactive web. Instead of static brochureware, the web became a read/write medium, with a social layer that accelerated quickly. As with most technical innovations, the evolution of an Internet operating system has been incremental and cumulative.

The same is true of government 2.0, or "Gov 2.0," which Tim O'Reilly defined as thinking like a platform provider that can bring services to citizens using government data and the creative power of the private sector.

Gov 2.0 has often been defined by its utility to help citizens or agencies solve problems, either for individuals or the commons. That's what the giants of the Web 2.0 era have been able to do successfully outside of the government world, and that's the paradigm that many Gov 2.0 events have been exploring.

In that vein, Gov 2.0 is not defined by social media any more than Web 2.0 is. Collaborative software -- including blogs, wikis, RSS, interactive video and social networks -- is an elemental feature of Gov 2.0, but it does not encompass all of it. For example, a congressional hearing this summer defined Government 2.0 in the context of Web 2.0 technologies, balancing potential security and privacy issues against innovation and cost savings.

So what does Web 2.0 mean to Gov 2.0? Many aspects cannot be discerned at this point, but one thing is certainly clear: It's about all of us. Creating a smarter, more innovative government matters to every citizen.

In their analysis of "Web 2.0 five years on, John Battelle and Tim O'Reilly wrote that "if we are going to solve the world's most pressing problems, we must put the power of the web to work -- its technologies, its business models, and perhaps most importantly, its philosophies of openness, collective intelligence, and transparency. And to do that, we must take the web to another level. We can't afford incremental evolution anymore."

In his advice on the direction of the first Government 2.0 Summit, federal CTO Aneesh Chopra urged the technology community gathering for the Gov 2.0 Summit not to focus on the successes of Web 2.0 in government, but rather on the unsolved problems that confront the country.

That community that Chopra has looked to for ideas came together at the Web 2.0 Expo last week in New York City. In no particular order, following are 10 lessons from Web 2.0 that could be applied to government. If you also attended the conference or have been thinking about the topic, please share your thoughts in the comments.

Work on stuff that matters

Thinking about the future is an obvious place to start when looking at the lessons of Web 2.0 for Gov 2.0. Tim O'Reilly recently spoke about the big issues that we all confront and some of the long-term trends that confront the tech industry, citizens and government alike: financial crises, income inequality, soaring healthcare costs, to name a few. Humanity as a whole has even greater challenges.

What does this mean to government? Can citizens collaborate with officials, workers and one another to apply a civic surplus to open government? During the 2008 election, then Senator Barack Obama said that "the challenges we face today -- from saving our planet to ending poverty -- are simply too big for government to solve alone. We need all hands on deck." As President, finding solutions to grand challenges means that Obama is looking again to the technology community for answers. Whether he finds them will be a defining element in judging whether a young Senator from Illinois that leveraged Web 2.0 to become President can tap into that collective intelligence to govern in the Oval Office.

Explore the power of platforms

This year, the overarching theme for the Web 2.0 Expo was platforms. In an increasingly mobile, social, and real-time world, Facebook and Twitter have continued to grow in popularity as the globe's No. 1 and No. 3 social networking websites.

This year Anil Dash, director of Expert Labs, talked with Ryan Sarver, Twitter's director of platform, and Bret Taylor, the chief technology officer of Facebook. Both have been entrusted with maintaining and improving the application programming interfaces (API) that enable millions of developers to build apps upon their platforms. Twitter's success has been driven by acting as a platform for thousands of such apps, allowing third parties to add functionality to Twitter that the relatively tiny company could not. While that strategy has evolved as Twitter itself has matured, Facebook has gone even further toward allowing value to be created on its platform, in particular social gaming. Zynga, the makers of the wildly popular Farmville game, has grown dramatically since 2007, with a potential valuation of some $5 billion. Facebook's platform for adding a social layer is well-liked: Five months after launching social plugins, Taylor said that about 2 million websites have added them.

What could embracing platforms mean for government? Making community health data as useful as weather data has potential. Open source may improve healthcare through NHIN Direct. Standardizing APIs, empowering users, and working with developers can be a crucial complement to publishing open data online, as may be true in rebooting As Nat Torkington has pointed out, however, "government-as-platform doesn't absolve us from asking what services should be provided by a government."

Make better data-driven decisions

How will the e-commerce platforms of tomorrow compare to the engines of today? Given Charlie Kim's analysis on data-driven e-commerce, we can expect them to be smarter, leaner and more efficient. Next Jump, which Kim founded during the .com boom, uses data to turn browsing into buying. There's no question that real-time monitoring is important to cybersecurity and fighting insider threats. Just as sentiment analysis at the point of sale and understanding of customer behavior improves Web 2.0 businesses, real-time monitoring of log management is crucial to improved cybersecurity.

What does data-driven e-commerce mean for government? If the government is to provide citizens with better e-services, save energy through data center consolidation or unlock innovation through healthcare information, the lessons of data-driven e-commerce are relevant.

Crowdsourcing can empower citizens

Yes, the term "crowdsourcing" has become a buzzword. That does't mean the underlying phenomenon doesn't have immense power. After Lukas Biewald of CrowdFlower and Leila Chirayath Janah of Samasource stepped off stage to talk about "the future of work," they'd offered a reminder of how channeling crowdsourcing into distributed work effects government, research and the workforce.

What does crowdsourcing mean to government? The power of empowered citizens to improve communities and collaborate with government showed how this Web 2.0 idea is an elemental component of the future of Gov 2.0 and participatory platforms.

Location, location, location

Location isn't just significant in the real estate game. No one was surprised to see Dennis Crowley focusing on the power of location-based social networking, APIs and location data at the Web 2.0 Expo. After all, he founded Foursquare, which just passed 3 million users. The bigger question that the success of Foursquare inspires might be whether location-based services could increase civic engagement in millennials. In September, Foursquare announced a partnership with CNN that would give a "healthy eater" badge to anyone who checks in at one of 10,000 farmers' markets.

What does this mean for government? Don't expect Michelle Obama to check in after she "gets up and moves" to the White House farmers' market quite yet. But citizens may be another matter. "We've seen time and time again how Foursquare can be used to drive people to action, and CNN's campaign is a perfect example of how brands can use the platform to promote good behavior, such as healthy eating," said Crowley. Foursquare has embraced a platform approach in allowing other services to build on top of its API. If it goes ahead and creates civic badges for volunteering or registering to vote, Foursquare could become a platform for civic involvement. The 1,600-pound gorilla, however, is Facebook. If the social networking giant puts more resources behind its Places product, government could have both new opportunities and privacy risks to consider.

Find better filters for information overload

Katie Couric at Web 2.0 ExpoWhen CBS Evening News anchor Katie Couric said that we need better filters for a tsunami of news at Web 2.0 Expo, she was was expressing a frustration common to government officials, media and citizens alike. Will hanging out with the geeks improve the quality network news? Judging from Couric's comments, she's using online platforms to share what she's reading and interact with the people formerly known as her audience. Now that she's joined the online conversation, however, she's beset with the same challenges of finding relevant news, sourcing information and attributing material. Given her old school journalism chops, much of that will come easily. But identifying and mastering digital tools that make the real-time web relevant while retaining a healthy flow of news won't be a walk in the park.

Will hanging out with the geeks change government as well as broadcast media? Maybe. The consequences of successfully applying the lessons of Web 2.0 to Gov 2.0 have the potential be even more far reaching in government. It's in the self interest of traditional media to support a more educated citizenry with greater digital literacy. Living in an information bubble with like-minded people is both "limiting and dangerous to a democracy," said Couric during her talk. That's one reason that the Knight Commission was created, and why the information needs of this democracy must be considered as technology continues to evolve.

Design for how people live and work

"Design is really part of life. In particular, it's a fundamental ingredient for progress," said Paola Antonelli. senior curator in the Department of Architecture and Design at the Museum of Modern Art (MoMA). "When technology people and when scientists create revolutions or create something new, designers are the ones who make these revolutions into objects that people can use."

What's the Web 2.0 lesson on design for Gov 2.0? It's simple: Design matters. Well designed websites help citizens find key services. Better designed signs make for safer traffic patterns, evacuations, safety precautions or access to key utilities. Infographics can explain crucial public health messages.

Live in the future

What can pornography, neuroscience and maps teach people about the future of media and business? Plenty. Each has driven the web's development. Today is no different. Congressional representatives may be unhappy with people accessing pornography from government computer equipment, but the technology that's driven the industry deserves further study on its own merits. As New York Times technology writer Nick Bilton observed in his talk, we're now living in the future that science fiction authors imagined in decades past.

What do changes in media consumption mean for government? Citizens are searching more for information online, and in more places, than ever before. They will interact with legislators and consume e-services outside of normal business hours. Government officials building the infrastructure for a more participatory digital republic should consider how to reach those digital natives while retaining real world facilities for those left behind by the digital divide.

Look to HTML5 and mobile access

It's not hard to see that the future is mobile, as cellphone ownership skyrockets around the world and wireless broadband data usage soars. These days, it seems like there's an app for nearly anything and everything. When Daring Fireball's John Gruber talked about Apple and the Open web, the importance of learning HTML5 as a means of offering the best experience to all mobile platforms shone through. "Web 1.0 is http in your browser. Web 2.0 is http everywhere," said Gruber. If Web 2.0 means people will be accessing the web everywhere, delivering rich media experiences that are platform-neutral is crucial.

What does HTML5 mean for government? As government entities create new .gov sites and invest in new apps, it's important to think of how the open web has evolved. Sure, Flash and HTML may be converging, but some actions speak volumes: "One of the major technology decisions we're dealing with is mobile platforms," said Brett Taylor, the CTO of Facebook. "We're doubling down on HTML 5." Why? It's about giving the most people the best mobile experience possible, without requiring people to buy an expensive smartphone to use an app.

Learn how to Mayor

Web 2.0 has also been driven by laughter, from the LOLcats to Rickrolling to the steady march of shared funny news. That's why it makes sense to end with Baratunde Thurston's talk on "how to Mayor," where he takes "Foursquare politics to the next level." The Onion's web editor understands the confluence of technology and humor better than most, and at Web 2.0 Expo he demonstrated that mastery to a live audience.

What does humor mean to government? Business, politics and government, after all, are often deadly serious. If Defense Secretary Robert M. Gates manages to "leave them laughing," however, you can see how humor can be a useful tool. When Thurston added humor to a healthy Foursquare competition between friends, the result was a valuable lesson in how new media can be put to good use for campaigns. In the heated election season to come, a little laughter to the partisan wrangling will be a welcome respite. While the political videos that go viral are often the misfires, using technology to discover and speak to the needs, desires and sentiments of constituents where they live and work will still matter. As former Speaker of the House Tip O'Neill famously said, "all politics is local." In a world where Web 2.0 includes location data, mapping mashups and social media, we can expect that aphorism to continue to ring true.

Credit: Katie Couric photo by James Duncan Davidson.


September 16 2010

Four short links: 16 September 2010

  1. jsTerm -- ANSI-capable telnet terminal built in HTML5 with Javascript, Websocket, and Node.js. (via waxpancake on Twitter)
  2. MySQL EXPLAINer -- visualize the output of the MySQL EXPLAIN command. (via eonarts on Twitter)
  3. Google Code University -- updated with new classes, including C++ and Android app development.
  4. Cloudtop Applications (Anil Dash) -- Anil calling "trend" on multiplatform native apps with cloud storage. Another layer in the Web 2.0 story Tim's been telling for years, with some interesting observations from Anil, such as: Cloudtop apps seem to use completely proprietary APIs, and nobody seems overly troubled by the fact they have purpose-built interfaces.

August 31 2010

Points of Control: The Web 2.0 Summit Map

In my blog post State of the Internet Operating System a few months ago (and the followup Handicapping the Internet Platform Wars), I used the analogy of "the Great Game" played out between England and Russia in the late Victorian era for control of access to India through what is now Afghanistan. In our planning for this year's Web 2.0 Summit, John Battelle and I have expanded on this metaphor, exploring the many ways that Internet companies at all levels of the stack are looking for points of control that will give them competitive advantage in the years to come.

Now, John has developed that idea even further, with a super-cool interactive map that shows the Internet platform wars in a kind of fantasy landscape, highlighting each of the players and some of the moves they might make against each other. Click on the link at the top of the image below to get to the full interactive version. You might also want to read John Battelle's description of the points of control map and how to use it.

Some of the battlegrounds are already clear, as Google has entered the phone hardware market to match Apple's early lead, while Apple is ramping up its presence in advertising and location-based services to try to catch up to Google. Meanwhile, Facebook adds features to compete with Twitter and Foursquare, Google (and everyone else) keeps trying to find the magic bullet for social networking, and tries to eat Yelp's lunch with Place Pages, Microsoft gains share in search and tries again to become a player in the phone market. Areas like social gaming, payment, speech and image recognition, location services, advertising, tablets and other new form factors for connected devices, are all rife with new startups, potential acquisition plays, and straight-up competition.

In the map, we've tried to highlight some of the possible competitive vectors. I'm sure you'll have other ideas about companies, possible lines of attack, and possible alliances. We hope to hear your feedback, and we hope to see you at the Web 2.0 Summit, where we'll be talking with many of the key players, and handicapping the next stage of the Great Game.


August 10 2010

"Knowledge is a mashup"

These days, we hear a lot about open data, open government, and Gov 2.0. President Obama's Open Government directive has given us access to huge data sets through avenues such as But we have a lot more assets as a country than just digital 0s and 1s in CSV files. We also have artifacts and science and history and experts. Can open government apply to those assets as well?


If the Smithsonian Commons project is any indication, the answer is yes. I talked to Michael Edson, director of Web and New Media Strategy for the Smithsonian about the project.

The Commons, which is currently being prototyped, is one of the best examples I've seen of "Gov 2.0 in action." Highlights include:

  • Open access to data and knowledge, applied in a way that matters to people's lives.
  • It's transparent, participatory, and collaborative in a way that harnesses what Clay Shirky has called "the value of combinability" -- an end result greater than the sum of its parts.
  • It is made significantly more useful by our contributions.

If these things are important to you -- if you see the power of freeing real objects and history and culture -- then go check out the prototype and let the Smithsonian staff know what you think. They need to hear from you before they can go on to the next phase in the project.

What is the Smithsonian Commons project?

Gov 2.0 Summit, 2010The Smithsonian Commons project is an effort to provide online access to Smithsonian research, collections, and communities. After all, not everyone can pop into one of the Smithsonian museums anytime they want. Even if they could, the buildings hold less than 2 percent of the collection. And anyway, if you're a teacher and want to borrow some American history for a class lesson, I hear the people that work at the museums don't like it much when you collect a bunch of that history in a shopping bag and fly back to Seattle with it.

But that description makes it sound like the project is about making a web site and putting pictures of stuff online, and that's not really it at all.The project goes well beyond just access. This is key, as making information available should be merely the first step. It also has to be applicable and useful to people's lives and it has to be the foundation for collaboration that makes the whole greater than the sum of the parts.

How can people use the information? In the case of the Smithsonian Commons, Edson says the idea is for it to be a catalyst and a platform to empower "innovators, explorers, and sense-makers."

In addition, Edson says the Smithsonian Commons "isn't just a commons of stuff the Smithsonian owns and controls ... Rather, the commons is about seeking powerful network effects from the sum total of the stuff, and the data, and the expertise, and the activities and programs, and the communities that form where stuff needs doing and figuring out in the real world."

He notes that, "in the 19th and 20th centuries the best way to do this was to build big bricks-and-mortar institutions and throw all the stuff and experts in there together so organized work could happen." But now, "we have important and audacious new goals that won't yield easily to the old ways of getting stuff done."

The prototype home page shows how this could work. A teacher can search through the vast collection for material for her class because the public has collaborated on tagging, recommending, discussing physical objects (from across all Smithsonian museums), and assembling videos, articles, and community-uploaded material. The teacher can filter by grade level and can download or share what she gathers, as well as store it in her personal collection. The sample video associated with the prototype shows the teacher downloading the information to a PowerPoint slide, but she could just as easily share the information to a Facebook page. (OK, maybe students don't want teachers to know about their Facebook accounts. But you get the idea.)


Why is the Smithsonian Commons project important?

We tend to think of information as data on a screen, but as Edson points out, the physical items that museums house represent ideas, science, culture, and history:

"I think museums, libraries, archives, and research organizations have a critical role to play in building the preconditions for sustained rational thought and discourse in society," he said. "And we can and should be an engine for creativity and innovation. I think we provide the building blocks for this by publishing our collections and research data in as open and free a way as possible. We provide scaffolding through scholarly research, exhibitions, publications, and public programs. But the mortar — the connective tissue that holds it all together — comes from the curiosity and activity and participation of millions of users, makers, and participants."

How does the Smithsonian Commons project impact developers, makers, and innovators?

When I see the work of the open government movement, I am impressed by how much has been accomplished, but also see there is so much more to be done. Developers need to take that raw data and make it applicable to the every day lives of citizens. Makers, hobbyists, and experts can take what the Smithsonian Commons project hopes to provide as a foundation for collaboration, innovation, and relevance to our every day lives.

And that ability is one of the core attributes of the Smithsonian Commons project. Edson explains:

"I often describe the Smithsonian Commons in Maker terms — that a commons is a kind of organized workshop where the raw materials of knowledge creation can be found and freely assembled into new things, by us, by you, by anybody. Cory Doctorow or Mister Jalopy might say that the Smithsonian Commons is a museum and research complex as it would exist if reconstructed around the Owner's Manifesto. Knowledge is a mashup!"

Edson says this collaboration is important in achieving the Smithsonian's five-year plan of "unlocking the mysteries of the universe, understanding and sustaining a biodiverse planet, valuing world cultures, and understanding the American experience" which Edson notes is doing what "Tim O'Reilly would call "stuff that matters."

Why the Smithsonian project is "crazy good"

"We say that the Smithsonian Commons will be vast, findable shareable, and free," Edson says. "These four things together give us something powerful and unique. Take away one and you get something good, but not crazy good."

What does this mean in practice?

  • Vast: Anyone can have access to the entire Smithsonian collection, staff, vistors, and partners.
  • Findable: Search, navigation, and user experience design, recommendations, comments, and social networks come together to help users find exactly what they need
  • Sharable: The project encourages use and reuse for work, pleasure, education, online and offline
  • Free: "The Smithsonian Commons will be built on the premise that free, high-quality resources will spread farther and create more opportunities for discovery and creation than those that are restricted by unnecessary fees and licenses," Edson says.

Who is the Smithsonian Commons project for?

Smithsonian Commons is for everyone, of course. But in the beginning, the makers and innovators are key. The Smithsonian wants to operate the Commons project a bit like a Web 2.0 startup: launch early and often. Iterate based on how people use it and what they really need. Who can make the best use of what the project has to offer and what is most useful to them?

The Smithsonian Commons prototype is the first step in that process. Publish some ideas and get feedback. Iterate. Repeat. Then ramp things up once the best direction becomes clear. Edson notes that getting permission, or at least forgiveness, for working this way is perhaps one of the greatest challenges in the Gov 2.0 movement: "In Gov 1.0 and in most large organizations, we like to design things in toto, pour the concrete, and be done with it. Varying that process requires a lot of stamina."

I think this is an awesome approach. Now that we can put things online easily and let people use things the way they want to rather than force our audiences into a particular model, why not take the best advantage of that? Edson says that it's easy to make generalizations about the Smithsonian audience, but in reality the Smithsonian is "the consummate long-tail business".

This project will be a great experiment to see how a large government organization can operate like a Web 2.0 startup and learn the needs of the audience as the project evolves.

The power of what the web can be

This project is an amazing example of the true capabilities of the web. It merges offline and online information, makes experts available in any topic we want, provides global collaboration, and gives all of us access to valuable knowledge as building blocks for something even greater. In "Cognitive Surplus" -- and noted above -- Clay Shirky talks about "the value of combinability." This project is a perfect example of what he describes. As I wrote about this concept on my blog:

"Shirky writes “if you have a stick, and someone gives you another one, you have two sticks. If you have a piece of knowledge — that rubbing two sticks together in a certain way can make fire — you can do something of value you couldn’t do before.” And here too is another new surplus the culture of the web gives us. By sharing knowledge, tools, failures, successes, ideas, we can better combine them for sums much greater than the parts. He notes that the community size has to be big enough, sharing has to be easy, there should be a common format or way of understanding the information, and then, there’s the last component, the one that technology can’t solve — people. Can we work well together? Do we understand each other, trust each other, want others to make what we do better?"

Edson says:

"The thing that makes the Smithsonian Commons different than a commons developed by a commercial entity is that the Smithsonian is in the forever business. By putting something in the Smithsonian Commons we're asking people to trust us. We're not going to scam you. We're not going to violate your privacy. We're not going to get bought by a competitor or just decide to go out of business one day. We're going to be honest about what we do and don't know, we're going to be open to new ideas and points of view, we're going to help each other figure out the world, and these promises are good, forever. Museums and libraries and archives are some of the few organizations in our culture that enter into those kinds of promises, and we take that responsibility very seriously."

So what's next?

Edson says that the Smithsonian has never done a project like this before, so they've got no real process for it. Right now, they are soliciting feedback and comments. You can head over to the prototype right now and tell them what you think, what you would like the project to be, and how you'd best be able to use it. The reaction so far has been overwhelmingly positive. But the Smithsonian wants to hear from as many people as possible before going forward so, ultimately, they build what people really want rather than what they think people might want. That's a true Web 2.0 approach to Gov 2.0.


July 23 2010

Web 2.0 risks and rewards for federal agencies

usgs-earthquake-tweets.jpgThe nature of record keeping and government transparency in the information age is rapidly changing. Officials can text, tweet, direct message, send "Facemail," IM or Skype, all from a personal smartphone. That's why yesterday's testimony of David Ferriero, Archivist of the United States, at a hearing on "Government 2.0: Federal Agency Use Of Web 2.0 Technologies" was both critically relevant and useful. (It's embedded below, after the jump.)Officials are "free to use external accounts as long as emails are captured into records management systems," he said. "Every new technology provides new challenges to what is a record." Ferriero said that new guidance on government use of social media will be released this fall, updating the 2009 guidance issued by the National Archives and Records Administration (NARA).

The biggest challenge, said Ferriero, is whether the record is the whole site or just a portion. "Web 2.0 offers opportunities unimagined a decade ago," he said.

David McClure, associate administrator for citizen services and innovative technologies at the General Services Administration, echoed that sentiment in his testimony."Web 2.0 isn't fundamentally about the technology itself but how people are coming together to achieve extraordinary results," he said, pointing to uses for idea management, ranking or ranking ideas, communication and more. "From an efficiency perspective, a lot of software meets those needs without the need for the agency to build tools, when the market is as robust as it is today."

More on the House subcommittee hearing on Government 2.0 after the jump, including a United States General Accountability Office (GAO) report on Web 2.0 and security in government and videos.

The potential of Web 2.0 technology, as illustrated by the many examples McClure provided in his testimony, is balanced by both risk and privacy concerns. "The expectation is that any tool for government use adheres to the Privacy Act and a Privacy Impact Assessment," said McClure. Applications must be compliant with relevant regulations to be on, said McClure.

Gregory Wilshuen, director of information security issues for the GAO, testified at the hearing about government security challenges posed by the use of Web 2.0 platforms by federal agencies. He delivered a new report, embedded below, and said that the GAO will be looking at the preparation of agencies to retain records from social media platforms. "We've found a number of agencies using technology to interact well," he said. "Several are using technology in an effective manner using videos and blogs." Wilshuen said that the GAO will examine whether information maintained by third-party providers is subject to Freedom of Information requests, which is, as he put it, as "rather strenuous."

In reviewing federal activity, Wilshuen said the GAO found most agencies are using social media platforms. He highlighted three effective examples:

The challenge throughout all of these applications lies in privacy, security and records management, said Wilshuen. "Are these federal records?"

Testimony of United States Archivist

"If we're going to be advising other agencies on how to use these tools, we need to use them ourselves," said Ferriero. For instance, Ferriero said that given the severe budget conditions expected for the year ahead, they're using IdeaScale to crowdsource ideas. Similarly, the National Archives crowdsourced the redesign of its website.

"We need to rethink the definition of a record," said Ferriero. "What part of technology is permanent that we need to keep in perpetuity?" When asked about the areas the committee should focus oversight upon, Ferriero said that it's clear agencies have identified a "moderate to high risk" regarding archiving electronic agencies and that NARA needs to provide more guidance." His written testimony is embedded below:

Testimony of David McClure on Web 2.0 in government

McClure offered one direction for archiving: distinguishing between official government business versus personal use. Some officials have chosen to use multiple accounts for that reason, though gray areas are a real risk given the closeness of Washington society.

McClure was crystal clear on one point: the rapid changes to technology have changed American society. "Web 2.0 tools are essential for responding to shifting expectations of government," he said, citing the hundreds of billions of pieces of content shared on social networks and viewership of YouTube. McClure said that government is "expected to engage" on these new platforms. Using them, however, "should be aligned with core government principles," much as social media is used for business purposes in the private sector. McClure pointed to the Library of Congress, State Department and the TSA's IdeaFactory as examples of agencies using social media to deliver on their missions.

Consumer watchdog: More scrutiny of Web 2.0

Simpson urged the subcommittee not only to look at abstract technologies but also to compare those providing services, their approaches to privacy. In cloud computing and security, there's a "tendency of tech companies to overpraise." Simpson said Google missed the deadline for its Los Angeles cloud implementation, as the city had to come up with more money after Google did not meet security requirements set by the Los Angeles police department. (Google and partner Computer Sciences Corp. agreed to reimburse the city for the cost of the delay, which, according to MarketWatch, should only reach about $135,000).

Simpson reflected that his personal experience of Web 2.0 on the Obama campaign showed him that they can be powerful tools to "improve government transparency, responsiveness and citizen involvement." He balanced that potential with challenges to consumer privacy, including Google's "Wi-Spy" and Buzz missteps and Facebook's changing privacy policy. Simpson also raised concerns about federal agencies implicitly endorsing individual Web 2.0 technology companies, advocating for the enactment of robust privacy laws by Congress with meaningful warnings.

His testimony is embedded below:

Government 2.0, meet politics

Due to continued friction between Republicans and the Obama White House, the oversight hearing got off to a bit of a bumpy start. Politics overshadowed the technology. At issue was the Google Buzz imbroglio that involved White House deputy CTO Andrew McLaughlin and the absence of White House deputy CTO Beth Noveck, who had originally been slated to testify in front of the subcommittee on June 24th. Contrary to earlier reports, Consumer Watchdog's John Simpson did testify.

The substance of the Republican concern goes to whether communications by members of the executive branch are subject to the Presidential Records Act and thus must be disclosed. The position taken by Representative Lacy Clay (D-IL) was that the White House Office of Science and Technology Policy (OSTP), where McLaughlin works, is subject to the Federal Records Act, wherein an individual using a third-party electronic messaging system, such as Gmail, needs to ensure that a record gets into a proper records management system. Given that McLaughlin produced the email records due to a FOIA request and was reprimanded by the White House, Clay considered the matter closed.

Rep. Patrick McHenry (R-SC) did not, nor did Rep. Darrell Issa (R-CA). Issa, the ranking member on the subcommittee, has continued to voice concerns about the administration's Google ties. McHenry also raised concerns about reports that White House staff met lobbyists at Caribou Coffee or used personal email accounts to communication, thereby avoiding visitor logs or records management systems.

The issue of officials and staff in the executive branch using non-federal email systems is far from new, however, as Rep. Eleanor Holmes Norton noted in her response to McHenry. Among the thousands of emails from the Bush Administration were those sent by Karl Rove, which now appear lost to history.

It is unfortunate that, due to the politics surrounding the hearing, Noveck did not testify, a choice perhaps driven by media reports of a "showdown," as her knowledge of the use of social software by the federal government would have offered insight to both the subcommittee and the American people, to whom she and the Representatives are both ultimately responsible. After votes to subpoena Noveck and adjourn the hearing were denied on party lines, 5-4, the subcommittee heard from the four witnesses. Hillicon Valley reported further on the political wrangling yesterday.

On video, transparency and Government 2.0

The Oversight Committee posted videos of the testimony and the back and forth between legislators. Additionally, House IT staff said the government hearing on Government 2.0 would be streamed at It was impossible to miss, however, that the committee Twitter account, @OversightDems, didn't issue a single tweet about the hearing. An ironic committee policy came to light as well: only credentialed press were allowed to use laptops at the hearing, hampering the ability of government staff, bloggers or citizens to participate in documenting, discussing or posting status updates about the event online. Emily Long, a reporter from NextGov, didn't find the hearing to be social at all. Given the focus on the elements of open government in 2010, participation, transparency and collaboration, or Government 2.0 as described by McClure, wherein lightweight tools are used to share information about government activities, that policy deserves to be revisited.

Part 1

Part 2

Part 3

Part 4


July 08 2010

June 03 2010

Velocity Culture: Web Operations, DevOps, etc...

Velocity 2010 is happening on June 22-24 (right around the corner!).  This year we've added third track, Velocity Culture, dedicated to exploring what we've learned about how great teams & organizations work together to succeed at scale. 

Web Operations, or WebOps, is what many of us have been calling these ideas for years.  Recently the term "DevOps" has become a kind of rallying cry that is resonating with many, along with variations on Agile Operations. No matter what you call it, our experiences over the past decade taught us that Culture matters more than any tool or technology in building, adapting, and scaling the web.

Here is a small sample of the upcoming Velocity Culture sessions:

Ops Meta-Metrics: The Currency You Use to Pay For Change
John Allspaw (
Change to production environments can cause a good deal of stress and strain amongst development and operations teams. More and more organizations are seeing benefits from deploying small code changes more frequently, for stability and productivity reasons. But how can you figure out how much change is appropriate for your application or your culture?

A Day in the Life of Facebook Operations
Tom Cook (Facebook)
Facebook’s Technical Operations team has to balance this need for constant availability with a fast-moving and experimental engineering culture. We release code every day. Additionally, we are supporting exponential user growth while still managing an exceptionally high radio of users per employee within engineering and operations.

This talk will go into how Facebook is “run” day-to-day with particular focus on actual tools in use (configuration management systems, monitoring, automation, etc), how we detect anomalies and respond to them, and the processes we use internally for rapidly pushing out changes while still keeping a handle on site stability.

Change Management: A Scientific Classification
Andrew Shafer (Cloudscaling)
Change management is the combination of process and tools by which changes are made to production systems. Approaches range from cowboy style, making changes to the live site, to complex rituals with secret incantations, coming full circle to continuous deployment. This presentation will highlight milestone practices along this spectrum, establishing a matrix for evaluating deployment process.

There is a tremendous amount happing in our space in the coming weeks in addition to the conference itself.  First, the "Web Operations" book which John Allspaw & I edited goes to print on June 15th.  We're really excited about how it came together.  Then, immediately after Velocity is DevOpsDays, which is a great community event that continues the conversation after Velocity (and is free).  Hope to see you all there!

May 21 2010

Four short links: 21 May 2010

  1. Infrastructures (xkcd) -- absolutely spot-on.
  2. The Michel Thomas App: Behind the Scenes (BERG) -- not interesting to me because it's iPhone, but for the insight into the design process. The main goal here was for me to do just enough to describe the idea, so that Nick could take it and iterate it in code. He’d then show me what he’d built; I’d do drawings or further animations on top of it, and so on and so on. It’s a fantastic way of working. Before long, you start finishing each others’ sentences. Both of us were able to forget about distinguishing between design and code, and just get on with thinking through making together. It’s brilliant when that happens.
  3. Open Government and the World Wide Web -- Tim Berners-Lee offered his "Five-Star" plan for open data. He said public information should be awarded a star rating based on the following criteria: one star for making the information public; a second is awarded if the information is machine-readable; a third star if the data is offered in a non-proprietary format; a fourth is given if it is in Linked Data format; a fifth if it has actually been linked. Not only a good rating system, but a clear example of the significantly better communication by semantic web advocates. Three years ago we'd have had a wiki specifying a ratings ontology with a union of evaluation universes reconciled through distributed trust metrics and URI-linked identity delivered through a web-services accessible RDF store, a prototype of one component of which was running on a devotee's desktop machine at a university in Bristol, written in an old version of Python. (via scilib on Twitter)
  4. Data Access, Data Ownership, and Sharecropping -- With Flickr you can get out, via the API, every single piece of information you put into the system. Every photo, in every size, plus the completely untouched original. (which we store for you indefinitely, whether or not you pay us) Every tag, every comment, every note, every people tag, every fave. Also your stats, view counts, and referers. Not the most recent N, not a subset of the data. All of it. It’s your data, and you’ve granted us a limited license to use it. Additionally we provide a moderately competently built API that allows you to access your data at rates roughly 500x faster then the rate that will get you banned from Twitter. Asking people to accept anything else is sharecropping. It’s a bad deal. (via Marc Hedlund)

April 30 2010

State of the Internet Operating System Part Two: Handicapping the Internet Platform Wars

This post is Part Two of my State of the Internet Operating System. If you haven't read Part One, you should do so before reading this piece.

As I wrote last month, it is becoming increasingly clear that the internet is becoming not just a platform, but an operating system, an operating system that manages access by devices such as personal computers, phones, and other personal electronics to cloud subsystems ranging from computation, storage, and communications to location, identity, social graph, search, and payment. The question is whether a single company will put together a single, vertically-integrated platform that is sufficiently compelling to developers to enable the kind of lock-in we saw during the personal computer era, or whether, Internet-style, we will instead see services from multiple providers horizontally integrated via open standards.

There are many competing contenders to the Internet Operating System throne. Amazon, Apple, Facebook, Google, Microsoft, and VMware all have credible platforms with strong developer ecosystems. Then there is a collection of players with strong point solutions but no complete operating system offering. Let's take them in alphabetical order.


With the introduction in 2006 of S3, the Simple Storage Service, and EC2, the Elastic Compute Cloud, Amazon electrified the computing world by, for the first time, offering a general-purpose cloud computing platform with a business model that made it attractive to developers large and small. An ecosystem quickly grew up of companies providing developer and system-management tools. Companies like RightScale provide higher level management frameworks; EngineYard and Heroku provide Ruby-on-Rails based stacks that make it easy to use familiar web tools to deploy applications against an Amazon back-end; the Ubuntu Enterprise Cloud and Eucalyptus offer Amazon-compatible solutions.

A number of competitors, including Rackspace, Terremark, Joyent, GoGrid, and AppNexus are going head to head with Amazon in providing cloud infrastructure services. Many analysts are simply handicapping these providers and comparing them to offerings from Microsoft and Google. But to compare only cloud infrastructure providers is to miss the point. It's a bit like leaving out Microsoft while comparing IBM, Compaq, and Dell when handicapping the PC operating system wars. Hardware was no longer king; the competition had moved up the stack.

The key subsystems of the Internet Operating System are not storage and computation. Those are merely table stakes to get into the game. What will distinguish players are data subsystems.

Data is hard to acquire and expensive to maintain. Delivering it algorithmically at the speeds applications required for reasonable real-time performance is the province of very few companies.

In this regard, Amazon has three major subsystems that give it an edge: its access to media (notably books, music, and video); its massive database of user contributed reviews, ratings, and purchase data, and its One-Click database of hundreds of millions of payment accounts. As yet, only one of these, payment, has been turned into a web service, Amazon Flexible Payment Service.

Despite having an early lead in internet payment, Amazon reserved its use for too long for competitive advantage for its own e-commerce site, and didn't deploy it as an internet-wide service usable by developers until recently. And even then, Amazon lacks significant payment presence on mobile devices. Amazon has its own Kindle device for ebook sales, and its iPhone and Android apps for e-commerce, but powerful as these apps may be for driving sales to Amazon, they give the company no leverage in supporting third party developers. If anything, they will hinder the development of a mobile e-commerce ecosystem based on Amazon because Amazon is the largest competitor for many potential e-commerce developers.

Amazon's use of its media database as a back-end for its own proprietary e-reader device, the Kindle, highlights one of the fronts in what I've elsewhere called the War for the Web, namely the use of a dedicated front-end device giving preferential access to a player's back-end services. Apple and Google are in a much stronger position in this regard, with general-purpose smartphones as the device front-ends for their platforms. But Amazon has moved quickly to deploy its Kindle software on iPhone and Android; their compelling library of content may make the use of a proprietary device less important.

Two other Amazon service worthy of note are the Mechanical Turk service and the Fulfillment Web Service.

The Mechanical Turk service allows developers to farm out simple tasks to human participants. This turns out to be a remarkably powerful capability, with applications as divergent as data cleansing, metadata management, and even crowdsourcing disaster relief. There are many tasks that computers can't do alone, but that humans can help with. I've often made the case that all Web 2.0 applications are in fact systems for harnessing the collective intelligence of human users. But most of these applications do it in a single field of endeavor; Mechanical Turk is the leading general-purpose platform for putting people to work on small tasks that are easy for humans but hard for computers to do on their own.

Amazon's Fulfillment Web Service is another sleeper, whose full significance hasn't yet been realized. I foresee a future in which phone-based e-commerce makes the leap from virtual to physical goods. Right now, there's a huge business in selling songs, applications, ebooks, movies, and games on phones. There's an even bigger explosion coming in buying physical goods on the phone. And Amazon is the only platform player who actually can offer programmatically-driven fulfillment services. This is hugely important.

Amazon's weaknesses: Search (they have search capabilities with A9 and Alexa, but don't have a business model to support or extend those capabilities to developers at a cost (free) that is going to be required); advertising; location services; speech recognition; social graph. They have a very strong hand, very deep in some areas, but almost completely lacking in others.

Amazon also is weaker financially than its big three competitors: Apple, Google, and Microsoft. Jeff Bezos argues that this is actually a strength. He has noted more than once that Amazon's core business is retail, a notably low margin business. Cloud computing is a better business for Amazon than the school of hard knocks where it's learned to make a profit. "Commodity businesses don't scare us," he says. "We're experts at them. We've never had 35 or 40 percent margins like most tech companies."

This idea, of course, applies only to the commodity layers of cloud computing. And that's one more reminder that the outsized profits actually reside in the data subsystems where lock-in is achievable.


A few years ago, everyone thought that the big industry showdown was between Microsoft and Google. Now, Apple is the company to beat. With over 185,000 applications, the iPhone app store is creating a new information and services marketplace to rival the web itself. While Apple doesn't provide Amazon-like cloud hosting services, they don't have to. iPhone apps don't live on the web per se, though most of them, apart from local games, do rely on internet-based services.

Apple's strongest Internet OS subsystems are media (the iTunes store), application hosting (the App Store), and payment. Apple has over a hundred million people who are used to buying content with one click. They've given Apple their payment credentials, and use them to buy a wide variety of digital goods: first music, then applications, including games, then books.

What's next? As physical goods e-commerce takes off on the phone, I expect Apple to try to insert itself into the great money river flowing through its platform. Apple takes a 30% cut from application sales. While this percentage is too high for physical goods, it's not hard to imagine Apple interposing itself as the payment processor for applications ranging from ebay to Chipotle, taking a little bit of a much larger revenue stream.

Apple's weaknesses are legion. They have no cloud computation platform, they are latecomers to location and advertising, with interesting acquisitions but no clear strategy and nothing like a critical mass of data. (They do, however, have piles of cash, and strategic acquisitions could quickly change those dynamics.) They have great social graph assets in the form of user address books, email stores, and instant messaging friend networks, but they show little sign of understanding how to turn those assets into next generation applications or services. But most strikingly, they don't really seem to understand some key aspects of the game that is afoot.

If they did, MobileMe would be free to every user, not a $99 add-on. Web 2.0 companies know that systems that get better the more people use them are the key to marketplace dominance in the network era. The social graph is one such system, for which Facebook is currently the market leader. Companies that want to dominate the Internet Operating System either need to make a deal with Facebook to integrate their platforms, or have a compelling strategy for building out their own social graph assets. Unless Apple is planning a deal with Facebook, their current MobileMe strategy seems only to indicate that they don't understand the stakes.

Apple's other weaknesses might well be addressed by an alliance with Microsoft, which has strengths everywhere that Apple is weak. Given Apple's feud with Google, this is an increasingly likely scenario. In fact, you can imagine a 3-way alliance between Apple, Facebook, and Microsoft that would make for a very powerful platform. That being said, alliances are relatively weak at coordinated execution, so this opportunity may be stronger in theory than it turns out in fact.

But all of these weaknesses may be outweighed by the amazing job that Apple has done in creating a new computing paradigm to rival the web itself. As Jim Stogdill wrote in a recent post, the iPad isn't a computer, it's a distribution channel:

One interesting twist is how the iPad combines network effects and constrained distribution. The bright shiny object design of the iPad leads to network effects at the app store which in turn drives more consumers back to the device itself. Then to the degree that those two forces hold consumers in thrall of the device, Apple can use the device as the point of sale for content worth more than the device itself. The leverage is linked - the first leads to market presence, and then the market presence makes for stronger monetization opportunities in the device-hosted channel.

The other interesting thing is that so many of those "apps" are really just web pages without a URL. Or books packaged as an app. In short, this is content that is abandoning the web to become a monetizable app.

History is never completely new and we've seen things like this happen before. Prior to the 1980's essentially all television was broadcast in the clear. An unconstrained distribution channel like broadcast TV could only be monetized through ad sales, but along came cable with its point-to-point wave guides and surprised consumers were suddenly faced with paying for access.

This is Apple's trump card.

There are those who point to the lessons of VHS vs Betamax and the commodity PC vs Apple, and see Android as an inevitable winner. However, as Mark Sigal points out in Five reasons iPhone vs Android isn't Mac vs Windows:

It is a truism that in platform plays he who wins the hearts and minds of developers, wins the war. In the PC era, Apple forgot this, bungling badly by launching and abandoning technology initiatives, co-opting and competing with their developers and routinely missed promised milestones. By contrast, Microsoft provided clear delineation points for developers, integrated core technologies across all products, and made sure developer tools readily supported these core initiatives. No less, Microsoft excelled at ensuring that the ecosystem made money.

Lesson learned, Apple is moving on to the 4.0 stage of its mobile platform, has consistently hit promised milestones, has done yeomen's work on evangelizing key technologies within the platform (and third-party developer creations - "There's an app for that"), and developed multiple ways for developers to monetize their products. No less, they have offered 100 percent distribution to 85 million iPhones, iPod Touches and iPads, and one-click monetization via same. Nested in every one of these devices is a giant vending machine that is bottomless and never closes. By contrast, Google has taught consumers to expect free, the Android Market is hobbled by poor discovery and clunky, inconsistent monetization workflows. Most damning, despite touted high-volume third-party applications, there are (seemingly) no breakout third-party developer successes, despite Android being around two-thirds as long as the iPhone platform.

And that's only one of the five compelling reasons that Mark puts forward for Apple to win in mobile. (However, see Chris Lynch's rebuttal for the corresponding arguments why Android will win.)

Nonetheless, even if Apple has the dominant mobile platform, they won't have the full recipe for the operating system of the future. A network connection has two ends, and until Apple can offer a complete suite of cloud data services, they can't deliver the kind of lock-in that Microsoft enjoyed in the PC era. This is actually a good thing, and a harbinger of the best outcome for the Internet OS, namely that no one controls enough of it, everyone has to compromise, and interoperability (the internet as "network of networks") continues to play its generative role.


Archilochus, the Greek fabulist, once said, "The fox knows many things, but the hedgehog knows one big thing." He might well have been talking about Facebook. Their one big thing, the social graph, might look like an incomplete offering, but they have made a great deal of progress based on it.

Facebook is more than a website. For many people, it is a replacement for the web, the entire platform, the world in which they receive news, communicate with friends, play games, store and share photographs and videos, and use any one of hundreds of thousands of applications. The Facebook Application ecosystem exceeds even the Apple App Store in the number of applications (500,000 to Apple's 187,000); third party developers like Zynga are amassing fortunes using entirely new social selling dynamics.

Facebook Connect is well on its way to becoming the universal single-signon for the web - one of the first Internet Operating System subsystems (after Google Maps) to get wide adoption across a range of websites not belonging to the platform provider. Even more importantly, Facebook Connect allows you to Facebook-enable mobile apps. Clearly, Facebook understands what it means to be a platform provider.

Their latest announcements, of Facebook's Graph API and Social Plugins, are taking Facebook beyond its original "walled garden" approach, instead turning Facebook into a social utility for the entire web (including mobile devices.)

Facebook is testing a payment platform, but perhaps more interestingly, they appear to be partnering with Paypal to increase their capabilities in this area. They are getting better at monetization via advertising, but they haven't yet found the golden path for social advertising that Google found for search.

Facebook's weaknesses: Location, control over mobile devices, general purpose computing and storage platforms. But these are weaknesses only in the context of the desire to have a vertically-integrated platform from a single vendor. It may instead be that the lack of these capabilities is Facebook's greatest strength, as it will force them into a strategy of horizontal integration.


There's no question in my mind that Google's Internet Operating System is the furthest along. Many observers will look first to Google AppEngine as Google's echo of the Win32 promise. How can anyone who lived through the transition from DOS to Windows not read the following paragraph, and not be struck by the similarity of intent?
"For the the first time your applications can take advantage of the same scalable technologies that Google applications are built on, things like BigTable and GFS. Automatic scaling is built in with App Engine, all you have to do is write your application code and we'll do the rest. No matter how many users you have or how much data your application stores, App Engine can scale to meet your needs."
As Mark Twain once said, "History does not repeat itself, but it does rhyme."

But to focus too much on AppEngine is to miss the point. If all the Internet Operating System does is provide storage and computation, then Amazon, Microsoft, and VMware are all contenders - and Amazon has the lead.

Remember, though, that in the future, the subsystems that applications depend on to differentiate themselves will largely be data subsystems. And data at scale is Google's sweet spot.

Consider a few of the following applications, and ask yourself how many companies could put together all the data and computation assets to deliver on the following promises:

  • Provide free turn-by-turn directions on the phone, with destinations set by natural language search rather than by address, with optional speech recognition to set the destination or to search along the route for items of interest, with real-time traffic information used to calculate your arrival time, and actual Streetview images of turns as well as of your final destination. (The ability to deliver these services directly is a critical advantage. Anyone who has to license one or more of the components from another company has a much harder time giving the service away for free.)
  • Point your cell phone camera at many common objects - a book cover, a wine label, a work of art in a museum, a famous building or other landmark, a company logo, a business card, a bar code, and even, in an unreleased version, a human face - and return information about that object (Google Goggles). (Amazon's e-commerce app for the iPhone and Android does show off some similar capabilities. Bar-code scanning and image recognition allow you to see something in the real world, quickly find it at Amazon, and either order it, or simply remember it on your Amazon wishlist. But the range of objects recognized is less complete, and the use case is Amazon e-commerce, versus Google's more general platform.)
  • Automatically dial all of your phone numbers until you answer one of them, and if you aren't found, automatically transcribe (even badly) the message that was left for you.
  • Translate your speech or document (even badly) into any one of fifty languages.
The list goes on. Google has the most impressive set of data assets of any company on the planet, together with the boldness to put them together into a vision of how computing will work in the future. They aren't afraid, any more than Bill Gates was with Windows 1.0, to put out something that is ahead of their current reach, with the persistence to stick with it till it works.

And that's leaving out Google's stronghold in search and advertising, its dominance via YouTube of internet video, its cloud office suite, its strong email offering, the fact that Google Maps is becoming the lingua franca of mapping across the web, and more. With Android, Google also has the front-end component of the full mobile-to-cloud stack, with an industry adoption strategy (open hardware from multiple manufacturers) that has worked before, for both the VCR and the personal computer. They have a robust application ecosystem, both for the phone and for the enterprise. A week after its launch, the Google Application Marketplace had nearly 1500 apps, a faster uptake than even the iPhone. Today, there are over 50,000 Android Apps. (But as Marc Sigal notes in the analysis linked earlier, Apple has demonstrated much more consistent monetization for developers. According to O'Reilly Research, 24% of iPhone apps are free apps, while 59% of Android apps are free.)

That being said, Google does have a payment platform. While Google Checkout was an also-ran in the web payment wars, it has renewed significance and opportunity in the mobile era, as every Android Market customer is, by default, now a Google Checkout customer. In this one story you see how having all of the elements together makes each of them stronger than they would be alone. If you have an Android phone, Google Checkout is suddenly the default payment option. It doesn't have to be the best. It's the incumbent.

Google's weaknesses: they are the one to beat, the new Microsoft that everyone is afraid of. In addition, they lack Apple's sure touch on user experience; even the slickest Android phone lags Apple's fit and finish. They also have yet to come up with a convincing social media subsystem, although they are clearly focused on this opportunity. Their strongest assets are not actually overtly social systems like Google Wave or Google Buzz, but the phone itself, and its connection to the Gmail-based cloud address book.

I continue to believe that the tools we actually use to communicate with each other - our phones, our email, our instant messaging, and our shared documents - are the most powerful measures of our real social network. The company that first cracks the code of reflecting that social network throughout its applications, and giving the user the power to harness that network, will ultimately win. Google has many of the data assets necessary to develop those next-generation applications, but they haven't yet found the way to put them together.


Microsoft, like Google, has a strong suite of capabilities across the board: the Azure hosting and computation platform, the Bing search engine and advertising platform, a full-featured mapping platform, speech recognition (via the 2007 acquisition of Tellme). They have made a promising restart in their mobile platform with Windows Mobile 7. They have enormous untapped business-oriented social media assets in Microsoft Exchange, Outlook, and Sharepoint. And of course, they have boatloads of cash, and the willingness to spend it to achieve strategic objectives. They understand the game they are playing, and how important it is to their survival.

That being said, Microsoft's biggest asset right now might just be that they aren't Google, making them the favored white knight of everyone from Apple to Facebook. With rumors flying that Apple is in talks with Microsoft to make Bing the default search engine on iPhone, you can see the shape of a possible future in which an alliance between Apple and Microsoft acts as a counter to Google's outsized ambitions. Add in a partnership with Facebook, in which Microsoft is an investor, and you have a powerful combination.

Microsoft's biggest weaknesses (apart from the fact that the original Windows Mobile platform was a failure, and they are now facing a restart) are the "strategy tax" of continuing to support Windows and Microsoft Office, the very same problem that kept Microsoft from seizing the internet opportunity in the late 1990s.

Another point of distinction between Microsoft and Google is Microsoft's "software plus services" vision - namely the idea that rich, device-specific client apps will be the front end to web services, versus Google's web-only vision. Microsoft argues that, faced with the success of native apps on smartphones, even Google is embracing a software-plus-services approach. However, the rich clients that are driving the equation are not PC-based; they are native smartphone apps. And barring a successful restart of Microsoft's phone strategy, Google has the advantage there.

This isn't to say that Microsoft won't continue to be a phenomenally successful company - just as IBM managed to do despite the death of its mainframe monopoly - but the cutting edge of the future is in data-backed mobile services, where they are playing serious catch up.

Microsoft's greatest opportunity, paradoxically, is to embrace open data services in the same way that IBM embraced Open Source Software, to integrate their offerings with those from Facebook, Nuance, Paypal, and other best of breed data services. The question is whether that's in their DNA or their business model. My bet is that it's Facebook that emerges as the integration point for selected Microsoft services (location and search in particular) rather than the other way around.


While Nokia is left out of many of the overheated discussions about the future, let's not forget that they are still the dominant phone supplier in the world, that they own significant location and mapping assets via their purchase of Navteq, and that they too have a platform vision in the form of Ovi, providing access to music, maps, applications, games, and more on Nokia phones. That being said, it's hard to conceive of Nokia as a first-tier player in the Great Game.


There is no doubt in my mind that payment will be one of the most important of the Internet OS subsystems. E-commerce, not advertising, is the killer business model of the mobile world.

And payment is hard. Knowing how much credit to extend is one of the problems that is best left to people with algorithmic expertise and massive amounts of data.

Apple and Google have their own built-in payment solutions (as does Microsoft for some of their platforms, such as Xbox), but what about everyone else? Visa and MasterCard remain sleeping giants; mobile phone carriers too have payment capabilities, but their business culture and systems make it difficult for them to deploy them for cutting edge applications. PayPal is web-native; making the transition to mobile has got to be their highest priority. Startups like Square (and others yet to be announced) are also taking aim at this area. Expect innovation. Expect competition. Expect acquisitions.

Salesforce also has a strong platform play, with thousands of business-oriented applications built on the platform. Salesforce in fact was the first to promulgate the idea of "platform as a service" (as distinguished from simply "software as a service" (individual applications) or "infrastructure as a service" (the kind of platform that Amazon pioneered.)


While Twitter is hardly, as yet, in the same league as Apple, Google, Microsoft, or Facebook, their dominance of the real time web has been game changing. They have a large and growing developer ecosystem, and a minimalist mindset that leads to rapid evolution. Twitter is increasingly used as transport for other kinds of data, and Twitter analytics are pointing the way towards other kinds of real-time intelligence.


At first glance, VMware might look like a niche player. Yes, they are the leader in application virtualization, and by virtue of that, a leader in corporate cloud computing. VMware's strategy appears to be based on making it easy for applications to migrate between cloud providers, and creating an easy interface between private and public clouds.

But is that enough?

But anyone who knows Paul Maritz knows that this is a man who understands the dynamics of internet data. While at PiCorp, the startup he founded after leaving Microsoft in 2000, he was focused on a vision of shared data in the cloud. "Why would you want to have your data in the cloud?" he told me in a private conversation some years ago. "For the same reason you keep your money in the bank rather than under your mattress. It becomes more valuable when it's kept with other people's data."

And as Scott Yara, founder and chairman of Greenplum, the massively multiprocessor Postgres database (disclosure: I am an advisor) pointed out to me when describing Greenplum's Chorus offering, it is not in fact Google that has the world's largest data repository. The New York Stock Exchange, T-mobile, Skype, Fox Interactive Media (MySpace), and many others, host their data in Greenplum. The sum of corporate data (sometimes referred to as "the dark web") is far greater than that in any consumer web company. Hence Chorus, Greenplum's platform to enable data sharing between its corporate customers.

Put in this light, VMware's management of private clouds may turn out to be an unexpected advantage. As companies far from the consumer web become fuller participants in the cloud data operating system, they will rely on facilities that VMware is already building: facilities that allow them to manage the boundaries between private and public data. This data and service segmentation may turn out to be one of the fundamental Internet OS capabilities.

In addition, VMware's acquisition of Zimbra might be seen as the first step towards acquiring internet data assets of the kind described in this article. Zimbra is an Exchange-compatible email platform; in the right hands it might be used to unlock Microsoft's business-oriented social graph.

VMware's acquisition of SpringSource is even more significant. Roman Stanek, CEO of cloud business-intelligence provider GoodData (in which I am an investor and a board member) remarked to me that for corporate cloud developers accustomed to programming in Java, Springsource is a kind of "Goldilocks" solution. Amazon's cloud APIs are too low-level; Google's and Microsoft's too high, with too much buy-in to Google's or Microsoft's systems; VMware's are "just right."

Maritz' long experience at Microsoft drove home to him the importance of developer tools. He understands how platform advantage is built, brick by brick. You can have the best platform in the world, but developer tools are what makes it stick.

VMware's weaknesses, of course, are many. They lack assets in media, in search, in advertising, in location based services, in speech recognition, and in many other areas that are going to be the currency of developers in future.

But that may not matter. Because there's another competitor in the mix.

Small Pieces Loosely Joined

In talking about the Internet Operating System, I've long used Tolkien's "one ring to rule them all" as a metaphor for platforms that seek, like Windows before them, to take control of the entire developer ecosystem, to be a platform on which all applications exclusively depend, and which gives the platform developer power over them. But there is another alternative. Both Linux and the World Wide Web are examples of what I call "small pieces loosely joined" (after David Weinberger's book of the same name). That is, these platforms have a simple set of rules that allow applications to interoperate, enabling developers to build complex systems that work together without central control.


Apple, Google, and Microsoft all seem to be plausible contenders to a one-ring strategy. Facebook too may want to play that game, though they lack the crucial mobile platform that each of the others hopes to control.

But it seems to me that one of the alternative futures we can choose is a future of cooperating internet subsystems that aren't owned by any one provider, a system in which an application might use Facebook Connect and Open Graph Protocol for user authentication, user photos, and status updates, but Google or Bing maps for location services, Google or Nuance for speech recognition, Paypal or Amazon for payment services, Amazon or Google or Microsoft or VMware or Rackspace for server hosting and computation, and any one of a thousand other developers for features not yet conceived.

This is a future of horizontal integration, not vertical integration. This integration is already happening at many levels. Consider music. Virtually every device that reads a music CD relies on Gracenote's CDDB to look up the track names; this is one of the net's oldest and most universally deployed data services. SonicLiving provides sites from Facebook to Pandora and Loopt with the ability for their users to find upcoming live concerts for artists they like - and to add them to their own calendars via "Universal RSVP."

Now it's certainly possible that Gracenote and SonicLiving (both private companies) might be acquired by someone looking to consolidate their hold on the infrastructure of online music, but evidence is strong that there will be countless "point" solutions like these that will be consumed by developers.

The Internet Operating System may end up looking more like a Linux distribution than a Microsoft or Apple PC or phone operating system. VMware might perhaps provide a cloud computing "kernel" while Facebook provides a social UI layer, Google or Bing provide alternate search subsystems, Android and iPhone and Nokia and the next generation of Windows Mobile provide mobile phone front-ends, and so on.

The VMForce announcement, which combines elements of VMware and Salesforce's respective offerings into a single developer platform, is a good sign of things to come, as companies who aren't holding on to the vision of a single vertically-integrated platform find it in their interest to work together.

As Benjamin Franklin so memorably said, just before signing the American Declaration of Independence: ""We must, indeed, all hang together, or most assuredly we shall all hang separately." Developers will need to make a choice between adopting any single platform, or pushing for open systems that allow interoperability and choice.

In the short term, I believe we'll see heightened competition, shifting alliances, and a wave of innovation, as companies fight for advantage in delivering next generation applications, and then use those applications to drive adoption of their respective platforms.

The key question, to my mind, is which of the "big four" (Apple, Google, Microsoft, and Facebook) will most strongly adopt the horizontal, open strategy.

Apple is the least likely. They have made a compelling case for vertical integration; what's more, they have made it work.

Microsoft seems like an unlikely ally of an open internet strategy given their history and heritage, but necessity is a good teacher.

Facebook has made selective moves towards openness. As their partnership with SonicLiving demonstrates, they consume web services from others as well as produce them. And while critics have argued that Facebook's recent open announcements don't go far enough, it's clear to me that Facebook gains more than it loses by cooperating with everyone from PayPal and VMware to Microsoft to strengthen their hand.

Google has made strong, public commitments to the values of the open web. Critics have pointed out, quite rightly, that For Google, The Meaning Of Open Is When It's Convenient For Them.

But frankly, this is true of any company balancing open and proprietary strategy. Does anyone doubt that IBM's commitment to open source software was gated on corporate advantage? Or that companies from Red Hat to MySQL have added proprietary elements to a core open source strategy?

In the end, companies make decisions about open versus closed and proprietary on competitive grounds. The art of promoting openness is not to make it a moral crusade, but rather to highlight the competitive advantages of openness, and to knit together the strategies of companies who might otherwise find themselves left out of the game.

This, by the way, is the backdrop for the discussion at this year's Web 2.0 Expo and especially Web 2.0 Summit. While the term "Web 2.0" has come to mean many things to many people, for me it's always been the story of what happens when you treat the internet, not any individual computer, as the platform. The Expo focuses on the technical infrastructure of the platform; the Summit focuses on the business models and the business strategy.

As John Battelle notes in his post Points of Control, "Fifteen years and two recessions into the commercial Internet, it’s clear that our industry has moved into a new competitive phase - a “middlegame” in the battle to dominate the Internet economy. To understand this shift, we’ll use the Summit’s program to map strategic inflection points across the Internet landscape, identifying key players who are battling to control the services and infrastructure of a websquared world."

Handicapping the Players

This post provides a conceptual framework for thinking about the strategic and tactical landscape ahead. Once you understand that we're building an Internet Operating System, that some players have most of the pieces assembled, while others are just getting started, that some have a plausible shot at a "go it alone" strategy while others are going to have to partner, you can begin to see the possibilities for future alliances, mergers and acquisitions, and the technologies that each player has to acquire in order to strengthen their hand.

I'll hope in future to provide a more thorough drill-down into the strengths and weaknesses of each player. But for now, here's a summary chart that highlights some of the key components, and where I believe each of the major players is strongest.

Chart Illustration 1-4.png

(Note that this chart is influenced by one that Nick Bilton of the New York Times put together last January.)

The most significant takeaway is that the column marked "other" represents the richest set of capabilities. And that gives me hope.

Older posts are this way If this message doesn't go away, click anywhere on the page to continue loading posts.
Could not load more posts
Maybe Soup is currently being updated? I'll try again automatically in a few seconds...
Just a second, loading more posts...
You've reached the end.
No Soup for you

Don't be the product, buy the product!

YES, I want to SOUP ●UP for ...