Newer posts are loading.
You are at the newest post.
Click here to check if anything new just came in.

April 10 2012

Four short links: 9 April 2012

  1. E-Reading/E-Books Data (Luke Wroblewski) -- This past January, paperbacks outsold e-books by less than 6 million units; if e-book market growth continues, it will have far outpaced paperbacks to become the number-one category for U.S. publishers. Combine that with only 21% of American adults having read a ebook, the signs are there that readers of ebooks buy many more books.
  2. Web 2.0 Ends with Data Monopolies (Bryce Roberts) -- in the context of Google Googles: So you’re able to track every website someone sees, every conversation they have, every Ukulele book they purchase and you’re not thinking about business models, eh? Bryce is looking at online businesses as increasingly about exclusive access to data. This is all to feed the advertising behemoth.
  3. Building and Implementing Single Sign On -- nice run-through of the system changes and APIs they built for single-sign on.
  4. How Big are Porn Site (ExtremeTech) -- porn sites cope with astronomical amounts of data. The only sites that really come close in term of raw bandwidth are YouTube or Hulu, but even then YouPorn is something like six times larger than Hulu.

November 10 2011

Four short links: 10 November 2011

  1. Steve Case and His Companies (The Atlantic) -- Maybe you see three random ideas. Case and his team saw three bets that paid off thanks to a new Web economy that promotes power in numbers and access over ownership. "Access over ownership" is a phrase that resonated. (via Walt Mossberg)
  2. Back to the Future -- teaching kids to program by giving them microcomputers from the 80s. I sat my kids down with a C64 emulator and an Usborne book to work through some BASIC examples. It's not a panacea, but it solves a lot of bootstrapping problems with teaching kids to program.
  3. Replaying Writing an Essay -- Paul Graham wrote an essay using one of his funded startups, Stypi, and then had them hack it so you could replay the development with the feature that everything that was later deleted is highlighted yellow as it's written. The result is fascinating to watch. I would like my text editor to show me what I need to delete ;)
  4. Jawbone Live Up -- wristband that sync with iPhone. Interesting wearable product, tied into ability to gather data on ourselves. The product looks physically nice, but the quantified self user experience needs the same experience and smoothness. Intrusive ("and now I'm quantifying myself!") limits the audience to nerds or the VERY motivated.

October 25 2011

What to watch for in mobile web apps

Mobile web apps are mostly defined by their "if onlys" — if only they had the feel of native apps, if only they could fully interact with devices, etc.

James Pearce, (@jamespearce), senior director of developer relations at Sencha, has a more optimistic take on mobile web apps. In the following interview, Pearce discusses promising web app technologies, and he explains why device APIs could make the web a lot more interesting. Pearce also weighs in on developers' current concerns, such as backward web app compatibility and testing across devices.

Our interview follows.

What are the most promising mobile web app technologies?

James_Pearce_Mug.pngJames Pearce: There are two significant technologies that I think we will see mature in the coming years. The first is WebGL, which provides a way for web developers to access the low-level graphic capabilities of the computer or device that the browser is running on. WebGL creates a huge opportunity to use web technologies to build games, to create high-performance simulations, and to develop other types of graphically heavy user interfaces — at least, when the browser supports it well. Currently, no default mobile smartphone browser offers good support, but I expect this to change soon.

Device APIs are also important because they allow web applications within a browser to query and interact with the device they're running on. These are not being implemented quickly, for various reasons, but the mobile web will take a huge step forward when web applications can interface with a device in much the same way native applications can. Imagine the possibilities that would come about through granting web apps access to a device's camera, contacts, calendar, messaging, and so on. The web will suddenly become a much more interesting place.

What are the best examples of HTML5 mobile apps? What lessons can and should be learned from them?

James Pearce: We have a directory at Sencha of some of the best examples, and this is just the tip of the iceberg. Of course there are some very well-known brands doing cool mobile things with web technology already — the Financial Times is a good example. We're seeing a whole spectrum of different types of apps and companies deploying services and applications with these tools and techniques.

Financial Times web app
Screenshot of the Financial Times web app

That said, it's still very early days, and I'm not sure we're ready to say we fully understand the best practices for building, deploying and promoting HTML5 web apps. But, certainly, we're seeing a lot of creativity around user experiences, and developers are pushing the bounds of what mobile devices can do. I guess if there's one thing we can learn, it is that the future is still very much up for grabs.

How much backward compatibility should mobile developers incorporate into their web apps?

James Pearce: This depends a lot on the type of app. If you are creating a relatively simple web application or site, then it's a good idea to employ techniques that allow it to function on down-scale devices. Normally, this means starting with simple content or markup and progressively enhancing the application in response to the additional capabilities the browser contains.

For full-fledged web applications, this is not at all easy to do, and at some point you need to draw the line with regard to the functionality the device must absolutely have in order to support your application's purpose. For example, a photo-sharing application is almost no use on a device with no camera API. A location-based application is pretty useless if the device can't determine its location and convey it to the browser, and so on.

What is the best way to test across devices without owning those devices?

James Pearce: That's an eternal challenge in mobile. The Apple and BlackBerry emulator tools are great. There's also one in the Android SDK, though sadly, of lower fidelity right now. Services like DeviceAnywhere allow you to remotely connect to devices — and Nokia runs a similar service — but ultimately, there's no alternative to the real thing to truly judge how your app's user experience shapes up.

TOC NY 2012 — O'Reilly's TOC Conference, being held Feb 13-15, 2012, in New York City, is where the publishing and tech industries converge. Practitioners and executives from both camps will share what they've learned and join together to navigate publishing's ongoing transformation.

Register to attend TOC 2012

This interview was edited and condensed.

Related:

July 14 2011

There are bigger issues surrounding the .gov review

The United States federal government is reforming its Internet strategy. In the context of looming concerns about the debt ceiling, high unemployment, wars abroad, rising healthcare costs, and the host of other issues that the White House and Congress should be addressing, that might seem like a side issue.

It's not. The federal government spends some $80 billion dollars every year on information technology. If you're paying any attention at all to government IT, you know that Uncle Sam is not getting his money's worth from that investment. Furthermore, the federal government has not been getting the kinds of returns in productivity or efficiency that the private sector has enjoyed over the past decade or so. Former White House OBM director Peter Orzag called that disparity the "IT gap" last year. So when the Obama administration launched a new initiative on Web reform this summer, it might have seemed a little overdue.

Better late than never, and better now than later.

Citizens are turning to the Internet for government data and services in unprecedented numbers, and they're expecting to find answers, applications and, increasingly, people. While public servants can join the conversations on social networks to address the latter demand, delivering improved information and e-services confronts the federal government with some tough choices, given budget constraints. That challenge is one reason that they're looking to the general public and the private sector for some ideas on how they can improve their strategy.

This week, in service of that goal, the White House hosted a livechat on improving federal websites with Macon Phillips, White House director of digital strategy, Vivek Kundra, the federal chief information officer, and Sheila Campbell, director of the GSA’s Center for Excellence in Digital Government. The chat, which has become a standard tool in the White House's online communications toolkit over the last year, included a livestream from WhiteHouse.gov/live, a Facebook chat and an active Twitter backchannel at the #dotgov hashtag. The White House also took questions through a form on WhiteHouse.gov and its Facebook wall.

These issues aren't new, of course, even if the tools for discussion have improved. And if you've been following the Gov 2.0 movement over the years this issue of how the government can use the Internet and associated technologies to work better has been at the core of the discussion throughout. Success online used to be measured by having a website, said federal chief information Vivek Kundra. As he observed immediately afterwards, "those days are long gone."

If the federal government is going to reform how it uses the Internet, it will need to learn and apply the lessons that Web 2.0 offers to Gov 2.0, whether it's standing up open government platforms, leveraging the cloud, crowdsourcing, or making data-driven policy.

Government is also going to need to stop creating a new .gov website for every new initiative, particularly if they're not optimized for search engines. There's some good news here: "Every month, historically, federal agencies would register 50 new domain names, said Kundra on Tuesday. "That's been halted."

This proliferation of federal .gov websites has been an issue for some time — call it ".gov sprawl" — and that what's driven the .gov reform effort in the context of the Obama administration's campaign to cut government waste. This week, for the first time, a dataset of federal executive branch Internet domains has been published as open government data online. The dataset of federal gov domains is hosted on Data.gov and has been embedded below:

Federal Executive Branch Internet Domains

"This dataset lists all of the executive branch second-level domains within the top-level .gov domain, and which agencies own them," commented General Services Agency new media specialist Dan Munz in the Community page for the dataset. "As White House Director of Digital Strategy Macon Philips has pointed out (see "TooManyWebsites.gov"), while many of these domain names point to sites that are valuable, some are duplicative or unnecessary. That makes it harder to manage the .gov domain, impairs agencies' ability to distribute information, and creates a user experience for citizens that just isn't as good as it could or should be. How can we fix that? Over the coming months, we'll have a plan for streamlining the federal executive branch webspace, and we want to invite you into the conversation. We're releasing this dataset as a first step, so that you can explore, comment, remix, and maybe even use the data to map the .gov domain in ways we haven't seen before."

OSCON Data 2011, being held July 25-27 in Portland, Ore., is a gathering for developers who are hands-on, doing the systems work and evolving architectures and tools to manage data. (This event is co-located with OSCON.)

Save 20% on registration with the code OS11RAD


Why reforming .gov matters

This effort is not impressing all observers. Micah Sifry, the co-founder of the Personal Democracy Forum, has called the move to delete redundant websites "cheap, dumb and cynical" at techPresident. "Redundant government websites probably cost the taxpayer a fraction of what we spend on military bands, let alone what we spend on duplicative and unnecessary government websites promoting the Army's, Navy's, Air Force's, Merchant Marine's, Naval Academy's, and Coast Guard's bands' websites! (According to NPR, the Marines spend $50 million a year on their bands, and the Army $198 million." In a larger sense, Sifry argued, "if you are really serious about eliminating stupid and pointless spending, then you'd be pushing for laws to strengthen protections for government whistleblowers (instead of going an stupid and pointless rampage to prosecute them!), since insiders know where the real waste is hidden."

Sifry is absolutely right on one count: the amount of money up for savings through reducing federal .gov websites is dwarfed by what is saved by, say, reducing Medicare fraud using new data analytics tools, or in finding cost savings in defense spending. Reducing the number of federal .gov websites by 90% would not significantly address the federal deficit. The biggest federal cost savings from this week's .gov livechat were likely cited by Kundra, when he said that 137 federal data centers were closed by the end of this calendar year, each of which consumes immense amounts of energy.

Where Sifry may have been overly harsh in his critique is in not acknowledging how progressive a perspective the White House appears to have embraced here. (Progressive meaning "forward-thinking," not political ideology, in this case.) Democratizing healthcare data so that it showed up in search engine results or is integrated into applications makes it more useful, argues Kundra, citing the improvements to hospitalcompare.gov. Moving from a static website to a universe of applications and services provisioned by open government data is shifting from a Web 1.0 vision to 2.0 reality. In a country where 35% of citizens have a smartphone, delivering services and providing information to a mobile audience has to be factored into any online strategy, whether in the public or private sector. And, in most cases, it's the private sector that will be able to create the best applications that use that data, if government acts as a platform to empower civic coders. Phillips acknowledged that explicitly. "The best mobile apps," he said, "are going to be driven by the private sector making use of public data."

If e-government is going to move toward "We-government" — as Sifry has described the growing ecosystem of civic media, technology-fueled transparency advocates and empowered citizens — government data and services will need to be discoverable where and when people are looking for them. That is ultimately, in part, what getting .gov reform right needs to be about, versus straightforward cost-savings.

Kundra asked the broader community to "help us think through how we're going to provide services over the mobile Internet." If, as he said, search is the default way that people search for information now, then releasing high quality open data about government spending, the financial industry, healthcare, energy, education, transportation, legislation and campaign finance would be a reasonable next step. Tim O'Reilly has been sharing a simple piece of advice to the architects of platforms for years: "Don't make people find data. Make data find the people."

The .gov reform, in that context, isn't just about reducing the number of websites and saving associated design or maintenance costs. It's about reducing the need to ever visit a website to retrieve the information or access a citizen requires. In the years ahead, it will be up to Congress and Kundra's successor as federal CIO — along with whomever he or she reports to in the Oval Office — to get that part of "web reform" done.

May 04 2011

Four short links: 4 May 2011

  1. Maqetta -- open source (modified BSD) WYSIWYG HTML5 user interface editor from the Dojo project. (via Hacker News)
  2. Hacker News Analysis -- interesting to see relationship between number of posts, median score, and quality over time. Most interesting, though, was the relative popularity of different companies. (via Hacker News)
  3. Real Time All The Time (Emily Bell) -- Every news room will have to remake itself around the principle of being reactive in real time. Every page or story that every news organisation distributes will eventually show some way of flagging if the page is active or archived, if the conversation is alive and well or over and done with. Every reporter and editor will develop a real time presence in some form, which makes them available to the social web. When I say "will" I of course don't mean that literally . I think many of them won't, but eventually they will be replaced by ones who do. (via Chris Saad)
  4. Changes in Home Broadband (Pew Internet) -- Jeff Atwood linked to this, simply saying "Why Web 1.0 didn't work and Web 2.0 does, in a single graph." Ajax and web services and the growing value of data were all important, but nothing's made the web so awesome as all the people who can now access it. (via Jeff Atwood)

December 31 2010

What lies ahead: Gov 2.0

Tim O'Reilly recently offered his thoughts and predictions for a variety of topics we cover regularly on Radar. I'll be posting highlights from our conversation throughout the week. -- Mac


Is open government moving from theory to practice?

Tim O'ReillyTim O'Reilly: The initial rush of interest in open government and transparency is wearing off and people are getting to work. Gov 2.0 startup founders are figuring out business models -- such as advertising and providing back-end services for cities -- and the first crop of startups are being funded. Early entrants, like SeeClickFix and CrimeReports, are growing. I think we'll see a number of new startups in this space. [Disclosure: O'Reilly AlphaTech Ventures is an investor in SeeClickFix.]

Open government's transition is also leading to practical applications. The Blue Button initiative from Veterans Affairs, which allows veterans to easily download their medical records, is an idea that's bound to spread. Blue Button is a direct outcome of the Open Government Initiative, but that connection probably won't be recognized. As is so often the case, the things that really make a difference get put into a different category than those that fail.

Along those lines, people might say that open government failed because many of the items on the punch list didn't happen the way they were originally envisioned. When we look back, we'll realize that open government is not just about transparency and participation in government decision making, but the many ways that open data can be put to practical use.

There are profound open government projects taking shape. For example, the Department of Health and Human Services (HHS) could transform our healthcare system through open data and medical records. HHS Connect and The Direct Project are all about creating the standards for interoperability between medical records. We'll eventually see and benefit from larger efforts like these.

Another open data project that I'm fond of that started very early in the open government process is GTFS, the General Transit Feed Specification. That's the data standard that lets transit districts feed their bus and train arrival times to applications like Google Transit, or any of the many smartphone apps that help you plan your trip on public transit. This standard started as a collaboration between Google and the city of Portland, but is now available from many cities. It's a great example of how governments can think like platform providers. They have to equip their buses and trains with GPS, and report out the data. They could report it just to their own bus stops and train stations, or they could make it available to third parties to deliver in a hundred ways. Which is better for citizens? It's pretty obvious.

And of course, this is government data following in the footsteps of great open data projects of the past, such as the satellite weather data released by NOAA to power the world's weather forecasters, or even the GPS signals that were originally designed only for military use but then released for civilian use.

Much of what you're describing sounds like the Web 1.0-to-2.0 trajectory. Do you see similarities?

Tim O'Reilly: At the end of the Web 1.0 era, some people claimed the web had failed because banner advertising, pop-overs, pop-unders and all the increasingly intrusive forms of advertising didn't work. Then Google came along with a better idea. I think something similar will happen with Gov 2.0. Many of the things people label as "Gov 2.0" are really the early signals and efforts of a "Gov 1.0" period. Important shifts will eventually occur, and they won't have anything to do with government agencies putting up wikis or using Twitter. We will see many unexpected outcomes over time.


A collection of posts that look ahead to 2011 can be found here.





Related:




December 17 2010

Four short links: 17 December 2010

  1. Down the ls(1) Rabbit Hole -- exactly how ls(1) does what it does, from logic to system calls to kernel. This is the kind of deep understanding of systems that lets great programmers cut great code. (via Hacker News)
  2. Towards a scientific concept of free will as a biological trait: spontaneous actions and decision-making in invertebrates (Royal Society) -- peer-reviewed published paper that was initially reviewed and improved in Google Docs and got comments there, in FriendFeed, and on his blog. The bitter irony: Royal Society charged him €2000 to make it available for free download. (via Fabiana Kubke)
  3. Bixo -- an open source web mining toolkit. (via Matt Biddulph on Delicious)
  4. How Facebook Does Design -- podcast (with transcript) with stories about how tweaking design improved the user activity on Facebook. One of the designers thought closing your account should be more like leaving summer camp (you know a place which has all your friends, and you don’t want to leave.) So he created this page above for deactivation which has all your friends waving good-bye to you as you deactivate. Give you that final tug of the heart before you leave. This reduced the deactivation rate by 7%.

November 15 2010

Hiring trends among the major platform players

After re-reading Tim's post on the major internet platform players, I looked at recent hiring trends* among the companies he highlighted. First I examined year-over-year changes in number of job postings (from Aug to Oct 2009 vs. Aug to Oct 2010). Consistent with the recent flurry of articles about hiring wars, all the companies (except for Yahoo) increased** their number of job postings. Winning the battle for the Internet's points of control requires amassing talent:

pathint


Below is the breakdown by most popular occupations over the last three months:

pathint


[For a similar breakdown by location (most popular metro areas), click HERE.]




(*) Using data from a partnership with SimplyHired, we maintain a data warehouse that includes most U.S. online job postings dating back to late 2005. Since there are no standard data formats for job postings across employment sites, algorithms are used to detect duplicate job postings, companies, occupations, and metro areas. The algorithms are far from perfect, so the above results are at best extremely rough estimates.



(**) As a benchmark, the total number of job postings in our entire data warehouse of jobs grew 68% from Aug/Oct 2009 to Aug/Oct 2010.


October 28 2010

Getting closer to the Web 2.0 address book

Tim O'Reilly has often asked a good question: seeing as my phone and email know who I communicate with, my company directory knows who I work with, and Amazon knows who has written books for my company, why can't these various things somehow combine to automatically deal with friend requests coming from social networks?

This specific problem is just one example of a wider issue. Given that so much diverse and overlapping information about each of us is spread between many applications, why are seemingly simple things based on combinations of information -- like automatically reacting to known friend requests -- still not possible?

Tim summarizes this by asking "Where is the web 2.0 address book?"

The traditional reaction to Tim's need would be to begin work on the design of a new application to provide the suggested features. This might give rise to an online social address book incorporating his specific suggestions, along with anything else the application designers might dream up to provide additional functionality.

While such an application might be cool and work well in the short term, I believe that approach is doomed to failure. The main reason is that we can't anticipate the future. Baking decisions about what to support or not into applications is almost certain to leave us wanting and needing more. At that point we're beholden to application designers' willingness, priorities, and ability to further adapt.

What happens when Tim comes up with another idea, or when you or I do? While an application might be designed to be open -- for example, by providing an API or a plugin system -- at the end of the day it's just another application sitting in front of its collection of data. This problem is particularly severe in our specific case: Tim would ideally like the incorporation not just of his phone's data and his email, but specific information from the O'Reilly org chart and from Amazon. That level of specificity -- or personalization, if you prefer -- is well beyond the interests of anyone writing a general phone book application.

For these reasons, among others, I believe the traditional "let's build a cool new app" reaction to Tim's dilemma is the wrong approach.

The other answer

An alternate answer begins with a sharp departure from this thinking. I suggest that the Web 2.0 address book need not take the familiar shape of an application. Instead, like a physical address book, it might be mainly about the data. That is, centered in a common data store through which a variety of loosely coupled applications interact with users and with one another.

In such a world, a friend request to Tim would be added to a shared online data store. There it would be noticed by an application running on a mobile phone, a periodic script that scans Tim's inbox, or an internal O'Reilly script with access to the company's staff list. Any application recognizing the requesting user's details could add information to the request object to indicate recognition. The initiating application could detect this and act on it.

Several aspects of this idealized scenario are worth close consideration:

  1. Such a system would require an underlying storage -- a "data fabric" as Tim has called it -- that was shared not just for reading, but also for writing. The shared writable data fabric would be open to future applications. They could contribute or consume information as they saw fit, without requiring the permission of existing applications, possibly without knowing or caring about each other's existence. Future applications would interact seamlessly with existing ones by simply following established data conventions.
  2. While any application should able to join the fun and contribute, a permissions system would be needed to protect existing information and to selectively share it among authorized applications. For example, Tim might authorize an application on his phone to add information about numbers he has dialed or email addresses he has been in contact
    with. He could authorize another application -- in our case an automatic friend request resolver -- to read that information and to create more information on his behalf to indicate existing friendships.
  3. This data-centric answer to the "Where is the web 2.0 address book?" question points to a wide class of future applications which cooperate and interact not through synchronous pre-ordained API calls, but via asynchronous data protocols which follow open-ended conventions. This is in strong contrast to applications which hold information in databases with relatively inflexible structure behind APIs that strictly delimit possible interactions. Applications using a shared writable data storage can adopt conventions and data protocols by which they cooperate to achieve joint actions. Applications that operate in this way leave the door open for new conventions, for additional unanticipated data, and for future applications that adopt the existing conventions, or introduce new ones.

At Fluidinfo we're building a shared online "cloud" data system, called FluidDB, with the characteristics outlined above. FluidDB aims to usher in a new class of applications, as described above, in which data can be thought of as being social. The data allows diverse applications to interoperate and communicate asynchronously using shared conventions. [Disclosure: Tim O'Reilly is an investor in FluidInfo.]

In FluidDB, all objects may have information added to them by anyone or any application at any time -- with no questions asked. Whereas a more traditional approach locks down entire database objects, FluidDB has a simple and flexible permissions system that operates at the level of the tags that comprise objects. Objects are always writable, including by future applications and by idiosyncratic applications that know, for example, how to access the O'Reilly staff list.

To put things in a more general light, Tim's "How ridiculous is this?" complaint (see image, below) is symptomatic of a wider problem. It highlights the awkwardness of a computational world in which applications keep their data behind UIs and APIs that are designed for specific purposes. These prevent augmentation, sharing and search, and they ultimately prove inflexible. It is ridiculous that even simple operations combining information in obvious ways are still beyond our reach.

How ridiculous is this slide
Slide from Tim O'Reilly's "What is Web 2.0?" presentation at Web 2.0 Expo Berlin 2007.

Relief does not lie in the direction of more applications holding more data behind more APIs. It lies instead in allowing related data to coexist in the same place. Both Google and Wikipedia have demonstrated, in very different ways, the massive value that accrues from co-locating related data. There's a real need for a Wikipedia for data. Writable and with an object for everything, like a wiki, but also with permissions, typed data, and a query language to make it more suitable for applications. Such a store can hold the data, the metadata, support search across these, and provide a locus for applications to communicate.

The Web 2.0 address book then becomes a ball of data, not an application. Something we access as we please, using a collection of tools that individually suit us, and without any application having the final word on what's possible.



Related:




October 26 2010

Dancing out of time: Thoughts on asynchronous communication

I find it useful to try to think clearly about communication systems. This includes the ways in which they are synchronous or asynchronous (or a mixture), and the disruption that occurs when new technology allows us to replace synchronous systems with asynchronous ones, or to deliver new forms of asynchronous communication.

This post is the first of a two-part series. Below, I take a look at the core differences between synchronous and asynchronous techniques, and I suggest an alternative to traditional synchronous API-based communication between applications. Part two, coming later this week, will address Tim O'Reilly's question: "Where is the Web 2.0 address book?"

Synchronous and asynchronous communication

Synchronous communication requires that the participating parties -- information producers and consumers -- are simultaneously involved. Examples abound: telephone calls, in-person meetings, a parent teaching a child a song, an audience listening to a public lecture or a live concert, buyers and sellers negotiating prices at a market, the broadcasting and simultaneous consumption of live events, Morse code flashed between ships at sea, and the Gotham City bat signal.

Communication can also be asynchronous. In this case, information is somehow -- broadly speaking -- published and is only later consumed.

The history of technology has many examples of the tremendous impact that can result when a form of synchronous communication is replaced with something asynchronous. The most obvious example is the invention of writing, which allowed information to be written and consumed later. There are many familiar examples, such as CDs and DVDs, the VCR (and its modern derivatives like the TiVo and Boxee), voicemail, etc.

There are many other less obvious examples, too. Post-it Notes are asynchronous: they let us publish information in physical locations that we choose carefully so the intended recipient will likely encounter them later. Blog posts, including this one, are asynchronous. There are also plenty of non-human examples, such as dogs urinating on lamp posts.

Communication doesn't always divide cleanly into synchronous or asynchronous. Many forms of communication have some elements of both. How would you classify the Pony Express, smoke signals, sky writing, and telegrams? Susan Boyle's performance on "Britain's Got Talent" was experienced synchronously by a few thousand people, and asynchronously by 50 million. And when Charlie bit his brother's finger (again) in the back seat of the car, the event was experienced synchronously by just three people, but later, asynchronously, in 233 million online viewings.



The requirements of asynchronous communication


The main requirement for asynchronous communication is a mechanism for interim information storage. This can take many forms: wet concrete, the soft bark of a tree, a wall, or one's own skin. Along with storage, one must also have a means by which to deposit information. In our previous examples that would be a finger, a pen-knife, spray paint, and a tattoo needle.

It's difficult to overstate the impact of technology on our ability to communicate asynchronously. Modern digital systems provide us with virtually unlimited amounts of cheap storage, along with the means to efficiently deposit/retrieve information into/from this storage. Consider Slideshare as an example: You can post presentation slides and have them read asynchronously by thousands or even millions, rather than just seen by the dozens who might attend a presentation in person.

The other requirement is a set of conventions about the communication. These may be high-level and taken for granted. For example, the language of communication. Conventions are often more explicit, and can become specialized over time, like apartment ads: lg 3br apt, d/w & a/c, hdwd flrs. Conventions can also take the form of protocols, in which parties exchange information using the shared storage. Examples include quoting conventions in email threads, and people using @name to address others and check for mentions in Twitter.

Asynchronous communication lends itself to scale

Asynchronous communication almost always scales better than synchronous. Synchronization is usually centralized, complicated, and expensive because participants are often forced to waste time waiting.

The main requirements for asynchronous communication are things that modern technology is providing in abundance. Experiences that were formerly limited to a small set of synchronous participants are now regularly experienced asynchronously by a significant fraction of the people on the planet.

At smaller scales, there are many familiar examples -- including non-digital ones -- of how asynchronous communications scale so easily and cheaply. Putting up a billboard beside a busy highway is a cheap and easy way of delivering a message to many. Using "Wanted" posters is better than having the sheriff stand on the corner describing a fugitive to passers by. Having a mailbox in front of your house provides temporary mail storage, which ironically can be faster and more convenient than a synchronous meeting with a courier service who will put your package back in their truck if you're not home to sign for it. Higher tech examples are abundant, as mentioned above.

Because asynchronous systems scale so much better than synchronous ones, coming up with ways to replace the latter with the former can result in the creation of extraordinary value. For example, consider Twitter and YouTube.

APIs, data protocols, and FluidDB

My main interest in thinking about the possibilities of asynchronous communications is in the area of how applications communicate (and might come to communicate). This is often done via APIs that applications provide, and in much of the computational world communication between applications takes the form of synchronous calls to these API methods. I believe this presents an opportunity for an asynchronous alternative, and that the alternative can have an impact similar to those we've seen historically when synchronous systems are replaced by asynchronous ones.

In particular, I'm currently involved in building FluidDB, a new kind of database under development at Fluidinfo. Among other things, FluidDB aims to provide shared always-writable storage that applications can use to exchange information. [Disclosure: Tim O'Reilly is an investor in FluidInfo.] I believe that predefined API calls between applications are generally less flexible and powerful than asynchronous communications.

In the second part of this series, coming later this week, I'll give an example of asynchronous flexibility by offering an answer to Tim O'Reilly's question "Where is the Web 2.0 address book?" His question arises from a frustration with the lack of communication between applications. Application-independent shared writable storage and simple conventions for communication look to me like the best way forward.



Related:




October 07 2010

Four short links: 7 October 2010

  1. How to Manage Employees When They Make Mistakes -- sound advice on how to deal with employees who failed to meet expectations. Yet again, good parenting can make you a good adult. It’s strange to me that in the technology sector we have such a reputation for yellers. Maybe it’s business in general and not just tech. [...] People stay at companies with leaders who rule like Mussolini because they want to be part of something super successful. But it does tend to breed organizations of people who walk around like beaten dogs with their heads down waiting to be kicked. It produces sycophants and group think. And if your company ever “slips” people head STRAIGHT for the door as they did at Siebel. I’d love to see a new generation of tech companies that don’t rule through fear. (via Hacker News)
  2. Information Wants to be Paid (Pete Warden) -- I want to know where I stand relative to the business model of any company I depend on. If API access and the third-party ecosystem makes them money, then I feel a lot more comfortable that I'll retain access over the long term. So true. It's not that platform companies are evil, it's just that they're a business too. They're interested in their survival first and yours second. To expect anything else is to be naive and to set yourself up for failure. As Pete says, it makes sense to have them financially invested in continuing to provide for you. It's not a cure-all, but it's a damn sight better than "build on this so we can gain traction and some idea of a business model". Yet again, Warden reads my mind and saves me the trouble of finding the right words to write.
  3. 0Boxer -- Chrome and Safari extensions to turn gmail into a game. (via waxy)
  4. Twitter's New Search Architecture (Twitter Engineering Blog) -- notable for two things: they're contributing patches back to the open source text search library Lucene, and they name the individual engineers who worked on the project. Very classy, human, and canny. (via straup on Delicious)

October 06 2010

10 Lessons for Gov 2.0 from Web 2.0

What is Web 2.0? In 2005, it meant geeks embracing a set of principles and practices: using the web as a platform, harnessing collective intelligence, data is the new "Intel inside," and others. By 2010, many of the dominant companies and services that embody or fuel Web 2.0 have become global brands: Google, Craigslist, Amazon, Facebook, Twitter, and a host of new mobile communities or platforms. These companies are often defined by what they allow users to do: upload pictures or video, stay connected to friends, track and discover news, save bookmarks, create communities, etc.

For non-geeks, Web 2.0 meant the online world became a place you could publish and participate. It became about everyone with an Internet connection exploring an interactive web. Instead of static brochureware, the web became a read/write medium, with a social layer that accelerated quickly. As with most technical innovations, the evolution of an Internet operating system has been incremental and cumulative.

The same is true of government 2.0, or "Gov 2.0," which Tim O'Reilly defined as thinking like a platform provider that can bring services to citizens using government data and the creative power of the private sector.

Gov 2.0 has often been defined by its utility to help citizens or agencies solve problems, either for individuals or the commons. That's what the giants of the Web 2.0 era have been able to do successfully outside of the government world, and that's the paradigm that many Gov 2.0 events have been exploring.

In that vein, Gov 2.0 is not defined by social media any more than Web 2.0 is. Collaborative software -- including blogs, wikis, RSS, interactive video and social networks -- is an elemental feature of Gov 2.0, but it does not encompass all of it. For example, a congressional hearing this summer defined Government 2.0 in the context of Web 2.0 technologies, balancing potential security and privacy issues against innovation and cost savings.

So what does Web 2.0 mean to Gov 2.0? Many aspects cannot be discerned at this point, but one thing is certainly clear: It's about all of us. Creating a smarter, more innovative government matters to every citizen.

In their analysis of "Web 2.0 five years on, John Battelle and Tim O'Reilly wrote that "if we are going to solve the world's most pressing problems, we must put the power of the web to work -- its technologies, its business models, and perhaps most importantly, its philosophies of openness, collective intelligence, and transparency. And to do that, we must take the web to another level. We can't afford incremental evolution anymore."

In his advice on the direction of the first Government 2.0 Summit, federal CTO Aneesh Chopra urged the technology community gathering for the Gov 2.0 Summit not to focus on the successes of Web 2.0 in government, but rather on the unsolved problems that confront the country.

That community that Chopra has looked to for ideas came together at the Web 2.0 Expo last week in New York City. In no particular order, following are 10 lessons from Web 2.0 that could be applied to government. If you also attended the conference or have been thinking about the topic, please share your thoughts in the comments.

Work on stuff that matters

Thinking about the future is an obvious place to start when looking at the lessons of Web 2.0 for Gov 2.0. Tim O'Reilly recently spoke about the big issues that we all confront and some of the long-term trends that confront the tech industry, citizens and government alike: financial crises, income inequality, soaring healthcare costs, to name a few. Humanity as a whole has even greater challenges.

What does this mean to government? Can citizens collaborate with officials, workers and one another to apply a civic surplus to open government? During the 2008 election, then Senator Barack Obama said that "the challenges we face today -- from saving our planet to ending poverty -- are simply too big for government to solve alone. We need all hands on deck." As President, finding solutions to grand challenges means that Obama is looking again to the technology community for answers. Whether he finds them will be a defining element in judging whether a young Senator from Illinois that leveraged Web 2.0 to become President can tap into that collective intelligence to govern in the Oval Office.

Explore the power of platforms

This year, the overarching theme for the Web 2.0 Expo was platforms. In an increasingly mobile, social, and real-time world, Facebook and Twitter have continued to grow in popularity as the globe's No. 1 and No. 3 social networking websites.

This year Anil Dash, director of Expert Labs, talked with Ryan Sarver, Twitter's director of platform, and Bret Taylor, the chief technology officer of Facebook. Both have been entrusted with maintaining and improving the application programming interfaces (API) that enable millions of developers to build apps upon their platforms. Twitter's success has been driven by acting as a platform for thousands of such apps, allowing third parties to add functionality to Twitter that the relatively tiny company could not. While that strategy has evolved as Twitter itself has matured, Facebook has gone even further toward allowing value to be created on its platform, in particular social gaming. Zynga, the makers of the wildly popular Farmville game, has grown dramatically since 2007, with a potential valuation of some $5 billion. Facebook's platform for adding a social layer is well-liked: Five months after launching social plugins, Taylor said that about 2 million websites have added them.

What could embracing platforms mean for government? Making community health data as useful as weather data has potential. Open source may improve healthcare through NHIN Direct. Standardizing APIs, empowering users, and working with developers can be a crucial complement to publishing open data online, as may be true in rebooting FCC.gov. As Nat Torkington has pointed out, however, "government-as-platform doesn't absolve us from asking what services should be provided by a government."

Make better data-driven decisions

How will the e-commerce platforms of tomorrow compare to the engines of today? Given Charlie Kim's analysis on data-driven e-commerce, we can expect them to be smarter, leaner and more efficient. Next Jump, which Kim founded during the .com boom, uses data to turn browsing into buying. There's no question that real-time monitoring is important to cybersecurity and fighting insider threats. Just as sentiment analysis at the point of sale and understanding of customer behavior improves Web 2.0 businesses, real-time monitoring of log management is crucial to improved cybersecurity.

What does data-driven e-commerce mean for government? If the government is to provide citizens with better e-services, save energy through data center consolidation or unlock innovation through healthcare information, the lessons of data-driven e-commerce are relevant.

Crowdsourcing can empower citizens

Yes, the term "crowdsourcing" has become a buzzword. That does't mean the underlying phenomenon doesn't have immense power. After Lukas Biewald of CrowdFlower and Leila Chirayath Janah of Samasource stepped off stage to talk about "the future of work," they'd offered a reminder of how channeling crowdsourcing into distributed work effects government, research and the workforce.

What does crowdsourcing mean to government? The power of empowered citizens to improve communities and collaborate with government showed how this Web 2.0 idea is an elemental component of the future of Gov 2.0 and participatory platforms.

Location, location, location

Location isn't just significant in the real estate game. No one was surprised to see Dennis Crowley focusing on the power of location-based social networking, APIs and location data at the Web 2.0 Expo. After all, he founded Foursquare, which just passed 3 million users. The bigger question that the success of Foursquare inspires might be whether location-based services could increase civic engagement in millennials. In September, Foursquare announced a partnership with CNN that would give a "healthy eater" badge to anyone who checks in at one of 10,000 farmers' markets.

What does this mean for government? Don't expect Michelle Obama to check in after she "gets up and moves" to the White House farmers' market quite yet. But citizens may be another matter. "We've seen time and time again how Foursquare can be used to drive people to action, and CNN's campaign is a perfect example of how brands can use the platform to promote good behavior, such as healthy eating," said Crowley. Foursquare has embraced a platform approach in allowing other services to build on top of its API. If it goes ahead and creates civic badges for volunteering or registering to vote, Foursquare could become a platform for civic involvement. The 1,600-pound gorilla, however, is Facebook. If the social networking giant puts more resources behind its Places product, government could have both new opportunities and privacy risks to consider.

Find better filters for information overload

Katie Couric at Web 2.0 ExpoWhen CBS Evening News anchor Katie Couric said that we need better filters for a tsunami of news at Web 2.0 Expo, she was was expressing a frustration common to government officials, media and citizens alike. Will hanging out with the geeks improve the quality network news? Judging from Couric's comments, she's using online platforms to share what she's reading and interact with the people formerly known as her audience. Now that she's joined the online conversation, however, she's beset with the same challenges of finding relevant news, sourcing information and attributing material. Given her old school journalism chops, much of that will come easily. But identifying and mastering digital tools that make the real-time web relevant while retaining a healthy flow of news won't be a walk in the park.

Will hanging out with the geeks change government as well as broadcast media? Maybe. The consequences of successfully applying the lessons of Web 2.0 to Gov 2.0 have the potential be even more far reaching in government. It's in the self interest of traditional media to support a more educated citizenry with greater digital literacy. Living in an information bubble with like-minded people is both "limiting and dangerous to a democracy," said Couric during her talk. That's one reason that the Knight Commission was created, and why the information needs of this democracy must be considered as technology continues to evolve.

Design for how people live and work

"Design is really part of life. In particular, it's a fundamental ingredient for progress," said Paola Antonelli. senior curator in the Department of Architecture and Design at the Museum of Modern Art (MoMA). "When technology people and when scientists create revolutions or create something new, designers are the ones who make these revolutions into objects that people can use."

What's the Web 2.0 lesson on design for Gov 2.0? It's simple: Design matters. Well designed websites help citizens find key services. Better designed signs make for safer traffic patterns, evacuations, safety precautions or access to key utilities. Infographics can explain crucial public health messages.

Live in the future

What can pornography, neuroscience and maps teach people about the future of media and business? Plenty. Each has driven the web's development. Today is no different. Congressional representatives may be unhappy with people accessing pornography from government computer equipment, but the technology that's driven the industry deserves further study on its own merits. As New York Times technology writer Nick Bilton observed in his talk, we're now living in the future that science fiction authors imagined in decades past.

What do changes in media consumption mean for government? Citizens are searching more for information online, and in more places, than ever before. They will interact with legislators and consume e-services outside of normal business hours. Government officials building the infrastructure for a more participatory digital republic should consider how to reach those digital natives while retaining real world facilities for those left behind by the digital divide.


Look to HTML5 and mobile access

It's not hard to see that the future is mobile, as cellphone ownership skyrockets around the world and wireless broadband data usage soars. These days, it seems like there's an app for nearly anything and everything. When Daring Fireball's John Gruber talked about Apple and the Open web, the importance of learning HTML5 as a means of offering the best experience to all mobile platforms shone through. "Web 1.0 is http in your browser. Web 2.0 is http everywhere," said Gruber. If Web 2.0 means people will be accessing the web everywhere, delivering rich media experiences that are platform-neutral is crucial.

What does HTML5 mean for government? As government entities create new .gov sites and invest in new apps, it's important to think of how the open web has evolved. Sure, Flash and HTML may be converging, but some actions speak volumes: "One of the major technology decisions we're dealing with is mobile platforms," said Brett Taylor, the CTO of Facebook. "We're doubling down on HTML 5." Why? It's about giving the most people the best mobile experience possible, without requiring people to buy an expensive smartphone to use an app.

Learn how to Mayor

Web 2.0 has also been driven by laughter, from the LOLcats to Rickrolling to the steady march of shared funny news. That's why it makes sense to end with Baratunde Thurston's talk on "how to Mayor," where he takes "Foursquare politics to the next level." The Onion's web editor understands the confluence of technology and humor better than most, and at Web 2.0 Expo he demonstrated that mastery to a live audience.

What does humor mean to government? Business, politics and government, after all, are often deadly serious. If Defense Secretary Robert M. Gates manages to "leave them laughing," however, you can see how humor can be a useful tool. When Thurston added humor to a healthy Foursquare competition between friends, the result was a valuable lesson in how new media can be put to good use for campaigns. In the heated election season to come, a little laughter to the partisan wrangling will be a welcome respite. While the political videos that go viral are often the misfires, using technology to discover and speak to the needs, desires and sentiments of constituents where they live and work will still matter. As former Speaker of the House Tip O'Neill famously said, "all politics is local." In a world where Web 2.0 includes location data, mapping mashups and social media, we can expect that aphorism to continue to ring true.

Credit: Katie Couric photo by James Duncan Davidson.



Related:




September 16 2010

Four short links: 16 September 2010

  1. jsTerm -- ANSI-capable telnet terminal built in HTML5 with Javascript, Websocket, and Node.js. (via waxpancake on Twitter)
  2. MySQL EXPLAINer -- visualize the output of the MySQL EXPLAIN command. (via eonarts on Twitter)
  3. Google Code University -- updated with new classes, including C++ and Android app development.
  4. Cloudtop Applications (Anil Dash) -- Anil calling "trend" on multiplatform native apps with cloud storage. Another layer in the Web 2.0 story Tim's been telling for years, with some interesting observations from Anil, such as: Cloudtop apps seem to use completely proprietary APIs, and nobody seems overly troubled by the fact they have purpose-built interfaces.

August 31 2010

Points of Control: The Web 2.0 Summit Map

In my blog post State of the Internet Operating System a few months ago (and the followup Handicapping the Internet Platform Wars), I used the analogy of "the Great Game" played out between England and Russia in the late Victorian era for control of access to India through what is now Afghanistan. In our planning for this year's Web 2.0 Summit, John Battelle and I have expanded on this metaphor, exploring the many ways that Internet companies at all levels of the stack are looking for points of control that will give them competitive advantage in the years to come.

Now, John has developed that idea even further, with a super-cool interactive map that shows the Internet platform wars in a kind of fantasy landscape, highlighting each of the players and some of the moves they might make against each other. Click on the link at the top of the image below to get to the full interactive version. You might also want to read John Battelle's description of the points of control map and how to use it.

Some of the battlegrounds are already clear, as Google has entered the phone hardware market to match Apple's early lead, while Apple is ramping up its presence in advertising and location-based services to try to catch up to Google. Meanwhile, Facebook adds features to compete with Twitter and Foursquare, Google (and everyone else) keeps trying to find the magic bullet for social networking, and tries to eat Yelp's lunch with Place Pages, Microsoft gains share in search and tries again to become a player in the phone market. Areas like social gaming, payment, speech and image recognition, location services, advertising, tablets and other new form factors for connected devices, are all rife with new startups, potential acquisition plays, and straight-up competition.

In the map, we've tried to highlight some of the possible competitive vectors. I'm sure you'll have other ideas about companies, possible lines of attack, and possible alliances. We hope to hear your feedback, and we hope to see you at the Web 2.0 Summit, where we'll be talking with many of the key players, and handicapping the next stage of the Great Game.




Related:

August 10 2010

"Knowledge is a mashup"

These days, we hear a lot about open data, open government, and Gov 2.0. President Obama's Open Government directive has given us access to huge data sets through avenues such as data.gov. But we have a lot more assets as a country than just digital 0s and 1s in CSV files. We also have artifacts and science and history and experts. Can open government apply to those assets as well?

smithsonian-logo.png

If the Smithsonian Commons project is any indication, the answer is yes. I talked to Michael Edson, director of Web and New Media Strategy for the Smithsonian about the project.

The Commons, which is currently being prototyped, is one of the best examples I've seen of "Gov 2.0 in action." Highlights include:

  • Open access to data and knowledge, applied in a way that matters to people's lives.
  • It's transparent, participatory, and collaborative in a way that harnesses what Clay Shirky has called "the value of combinability" -- an end result greater than the sum of its parts.
  • It is made significantly more useful by our contributions.

If these things are important to you -- if you see the power of freeing real objects and history and culture -- then go check out the prototype and let the Smithsonian staff know what you think. They need to hear from you before they can go on to the next phase in the project.

What is the Smithsonian Commons project?

Gov 2.0 Summit, 2010The Smithsonian Commons project is an effort to provide online access to Smithsonian research, collections, and communities. After all, not everyone can pop into one of the Smithsonian museums anytime they want. Even if they could, the buildings hold less than 2 percent of the collection. And anyway, if you're a teacher and want to borrow some American history for a class lesson, I hear the people that work at the museums don't like it much when you collect a bunch of that history in a shopping bag and fly back to Seattle with it.

But that description makes it sound like the project is about making a web site and putting pictures of stuff online, and that's not really it at all.The project goes well beyond just access. This is key, as making information available should be merely the first step. It also has to be applicable and useful to people's lives and it has to be the foundation for collaboration that makes the whole greater than the sum of the parts.

How can people use the information? In the case of the Smithsonian Commons, Edson says the idea is for it to be a catalyst and a platform to empower "innovators, explorers, and sense-makers."

In addition, Edson says the Smithsonian Commons "isn't just a commons of stuff the Smithsonian owns and controls ... Rather, the commons is about seeking powerful network effects from the sum total of the stuff, and the data, and the expertise, and the activities and programs, and the communities that form where stuff needs doing and figuring out in the real world."

He notes that, "in the 19th and 20th centuries the best way to do this was to build big bricks-and-mortar institutions and throw all the stuff and experts in there together so organized work could happen." But now, "we have important and audacious new goals that won't yield easily to the old ways of getting stuff done."

The prototype home page shows how this could work. A teacher can search through the vast collection for material for her class because the public has collaborated on tagging, recommending, discussing physical objects (from across all Smithsonian museums), and assembling videos, articles, and community-uploaded material. The teacher can filter by grade level and can download or share what she gathers, as well as store it in her personal collection. The sample video associated with the prototype shows the teacher downloading the information to a PowerPoint slide, but she could just as easily share the information to a Facebook page. (OK, maybe students don't want teachers to know about their Facebook accounts. But you get the idea.)

smithsonian.gif

Why is the Smithsonian Commons project important?

We tend to think of information as data on a screen, but as Edson points out, the physical items that museums house represent ideas, science, culture, and history:

"I think museums, libraries, archives, and research organizations have a critical role to play in building the preconditions for sustained rational thought and discourse in society," he said. "And we can and should be an engine for creativity and innovation. I think we provide the building blocks for this by publishing our collections and research data in as open and free a way as possible. We provide scaffolding through scholarly research, exhibitions, publications, and public programs. But the mortar — the connective tissue that holds it all together — comes from the curiosity and activity and participation of millions of users, makers, and participants."

How does the Smithsonian Commons project impact developers, makers, and innovators?

When I see the work of the open government movement, I am impressed by how much has been accomplished, but also see there is so much more to be done. Developers need to take that raw data and make it applicable to the every day lives of citizens. Makers, hobbyists, and experts can take what the Smithsonian Commons project hopes to provide as a foundation for collaboration, innovation, and relevance to our every day lives.

And that ability is one of the core attributes of the Smithsonian Commons project. Edson explains:

"I often describe the Smithsonian Commons in Maker terms — that a commons is a kind of organized workshop where the raw materials of knowledge creation can be found and freely assembled into new things, by us, by you, by anybody. Cory Doctorow or Mister Jalopy might say that the Smithsonian Commons is a museum and research complex as it would exist if reconstructed around the Owner's Manifesto. Knowledge is a mashup!"

Edson says this collaboration is important in achieving the Smithsonian's five-year plan of "unlocking the mysteries of the universe, understanding and sustaining a biodiverse planet, valuing world cultures, and understanding the American experience" which Edson notes is doing what "Tim O'Reilly would call "stuff that matters."

Why the Smithsonian project is "crazy good"

"We say that the Smithsonian Commons will be vast, findable shareable, and free," Edson says. "These four things together give us something powerful and unique. Take away one and you get something good, but not crazy good."

What does this mean in practice?

  • Vast: Anyone can have access to the entire Smithsonian collection, staff, vistors, and partners.
  • Findable: Search, navigation, and user experience design, recommendations, comments, and social networks come together to help users find exactly what they need
  • Sharable: The project encourages use and reuse for work, pleasure, education, online and offline
  • Free: "The Smithsonian Commons will be built on the premise that free, high-quality resources will spread farther and create more opportunities for discovery and creation than those that are restricted by unnecessary fees and licenses," Edson says.

Who is the Smithsonian Commons project for?

Smithsonian Commons is for everyone, of course. But in the beginning, the makers and innovators are key. The Smithsonian wants to operate the Commons project a bit like a Web 2.0 startup: launch early and often. Iterate based on how people use it and what they really need. Who can make the best use of what the project has to offer and what is most useful to them?

The Smithsonian Commons prototype is the first step in that process. Publish some ideas and get feedback. Iterate. Repeat. Then ramp things up once the best direction becomes clear. Edson notes that getting permission, or at least forgiveness, for working this way is perhaps one of the greatest challenges in the Gov 2.0 movement: "In Gov 1.0 and in most large organizations, we like to design things in toto, pour the concrete, and be done with it. Varying that process requires a lot of stamina."

I think this is an awesome approach. Now that we can put things online easily and let people use things the way they want to rather than force our audiences into a particular model, why not take the best advantage of that? Edson says that it's easy to make generalizations about the Smithsonian audience, but in reality the Smithsonian is "the consummate long-tail business".

This project will be a great experiment to see how a large government organization can operate like a Web 2.0 startup and learn the needs of the audience as the project evolves.

The power of what the web can be

This project is an amazing example of the true capabilities of the web. It merges offline and online information, makes experts available in any topic we want, provides global collaboration, and gives all of us access to valuable knowledge as building blocks for something even greater. In "Cognitive Surplus" -- and noted above -- Clay Shirky talks about "the value of combinability." This project is a perfect example of what he describes. As I wrote about this concept on my blog:

"Shirky writes “if you have a stick, and someone gives you another one, you have two sticks. If you have a piece of knowledge — that rubbing two sticks together in a certain way can make fire — you can do something of value you couldn’t do before.” And here too is another new surplus the culture of the web gives us. By sharing knowledge, tools, failures, successes, ideas, we can better combine them for sums much greater than the parts. He notes that the community size has to be big enough, sharing has to be easy, there should be a common format or way of understanding the information, and then, there’s the last component, the one that technology can’t solve — people. Can we work well together? Do we understand each other, trust each other, want others to make what we do better?"

Edson says:

"The thing that makes the Smithsonian Commons different than a commons developed by a commercial entity is that the Smithsonian is in the forever business. By putting something in the Smithsonian Commons we're asking people to trust us. We're not going to scam you. We're not going to violate your privacy. We're not going to get bought by a competitor or just decide to go out of business one day. We're going to be honest about what we do and don't know, we're going to be open to new ideas and points of view, we're going to help each other figure out the world, and these promises are good, forever. Museums and libraries and archives are some of the few organizations in our culture that enter into those kinds of promises, and we take that responsibility very seriously."

So what's next?

Edson says that the Smithsonian has never done a project like this before, so they've got no real process for it. Right now, they are soliciting feedback and comments. You can head over to the prototype right now and tell them what you think, what you would like the project to be, and how you'd best be able to use it. The reaction so far has been overwhelmingly positive. But the Smithsonian wants to hear from as many people as possible before going forward so, ultimately, they build what people really want rather than what they think people might want. That's a true Web 2.0 approach to Gov 2.0.


Related:

July 23 2010

Web 2.0 risks and rewards for federal agencies

usgs-earthquake-tweets.jpgThe nature of record keeping and government transparency in the information age is rapidly changing. Officials can text, tweet, direct message, send "Facemail," IM or Skype, all from a personal smartphone. That's why yesterday's testimony of David Ferriero, Archivist of the United States, at a hearing on "Government 2.0: Federal Agency Use Of Web 2.0 Technologies" was both critically relevant and useful. (It's embedded below, after the jump.)Officials are "free to use external accounts as long as emails are captured into records management systems," he said. "Every new technology provides new challenges to what is a record." Ferriero said that new guidance on government use of social media will be released this fall, updating the 2009 guidance issued by the National Archives and Records Administration (NARA).

The biggest challenge, said Ferriero, is whether the record is the whole site or just a portion. "Web 2.0 offers opportunities unimagined a decade ago," he said.

David McClure, associate administrator for citizen services and innovative technologies at the General Services Administration, echoed that sentiment in his testimony."Web 2.0 isn't fundamentally about the technology itself but how people are coming together to achieve extraordinary results," he said, pointing to uses for idea management, ranking or ranking ideas, communication and more. "From an efficiency perspective, a lot of software meets those needs without the need for the agency to build tools, when the market is as robust as it is today."

More on the House subcommittee hearing on Government 2.0 after the jump, including a United States General Accountability Office (GAO) report on Web 2.0 and security in government and videos.

The potential of Web 2.0 technology, as illustrated by the many examples McClure provided in his testimony, is balanced by both risk and privacy concerns. "The expectation is that any tool for government use adheres to the Privacy Act and a Privacy Impact Assessment," said McClure. Applications must be compliant with relevant regulations to be on Apps.gov, said McClure.

Gregory Wilshuen, director of information security issues for the GAO, testified at the hearing about government security challenges posed by the use of Web 2.0 platforms by federal agencies. He delivered a new report, embedded below, and said that the GAO will be looking at the preparation of agencies to retain records from social media platforms. "We've found a number of agencies using technology to interact well," he said. "Several are using technology in an effective manner using videos and blogs." Wilshuen said that the GAO will examine whether information maintained by third-party providers is subject to Freedom of Information requests, which is, as he put it, as "rather strenuous."

In reviewing federal activity, Wilshuen said the GAO found most agencies are using social media platforms. He highlighted three effective examples:

The challenge throughout all of these applications lies in privacy, security and records management, said Wilshuen. "Are these federal records?"

Testimony of United States Archivist

"If we're going to be advising other agencies on how to use these tools, we need to use them ourselves," said Ferriero. For instance, Ferriero said that given the severe budget conditions expected for the year ahead, they're using IdeaScale to crowdsource ideas. Similarly, the National Archives crowdsourced the redesign of its website.

"We need to rethink the definition of a record," said Ferriero. "What part of technology is permanent that we need to keep in perpetuity?" When asked about the areas the committee should focus oversight upon, Ferriero said that it's clear agencies have identified a "moderate to high risk" regarding archiving electronic agencies and that NARA needs to provide more guidance." His written testimony is embedded below:

Testimony of David McClure on Web 2.0 in government

McClure offered one direction for archiving: distinguishing between official government business versus personal use. Some officials have chosen to use multiple accounts for that reason, though gray areas are a real risk given the closeness of Washington society.

McClure was crystal clear on one point: the rapid changes to technology have changed American society. "Web 2.0 tools are essential for responding to shifting expectations of government," he said, citing the hundreds of billions of pieces of content shared on social networks and viewership of YouTube. McClure said that government is "expected to engage" on these new platforms. Using them, however, "should be aligned with core government principles," much as social media is used for business purposes in the private sector. McClure pointed to the Library of Congress, State Department and the TSA's IdeaFactory as examples of agencies using social media to deliver on their missions.

Consumer watchdog: More scrutiny of Web 2.0

Simpson urged the subcommittee not only to look at abstract technologies but also to compare those providing services, their approaches to privacy. In cloud computing and security, there's a "tendency of tech companies to overpraise." Simpson said Google missed the deadline for its Los Angeles cloud implementation, as the city had to come up with more money after Google did not meet security requirements set by the Los Angeles police department. (Google and partner Computer Sciences Corp. agreed to reimburse the city for the cost of the delay, which, according to MarketWatch, should only reach about $135,000).

Simpson reflected that his personal experience of Web 2.0 on the Obama campaign showed him that they can be powerful tools to "improve government transparency, responsiveness and citizen involvement." He balanced that potential with challenges to consumer privacy, including Google's "Wi-Spy" and Buzz missteps and Facebook's changing privacy policy. Simpson also raised concerns about federal agencies implicitly endorsing individual Web 2.0 technology companies, advocating for the enactment of robust privacy laws by Congress with meaningful warnings.

His testimony is embedded below:

Government 2.0, meet politics

Due to continued friction between Republicans and the Obama White House, the oversight hearing got off to a bit of a bumpy start. Politics overshadowed the technology. At issue was the Google Buzz imbroglio that involved White House deputy CTO Andrew McLaughlin and the absence of White House deputy CTO Beth Noveck, who had originally been slated to testify in front of the subcommittee on June 24th. Contrary to earlier reports, Consumer Watchdog's John Simpson did testify.

The substance of the Republican concern goes to whether communications by members of the executive branch are subject to the Presidential Records Act and thus must be disclosed. The position taken by Representative Lacy Clay (D-IL) was that the White House Office of Science and Technology Policy (OSTP), where McLaughlin works, is subject to the Federal Records Act, wherein an individual using a third-party electronic messaging system, such as Gmail, needs to ensure that a record gets into a proper records management system. Given that McLaughlin produced the email records due to a FOIA request and was reprimanded by the White House, Clay considered the matter closed.

Rep. Patrick McHenry (R-SC) did not, nor did Rep. Darrell Issa (R-CA). Issa, the ranking member on the subcommittee, has continued to voice concerns about the administration's Google ties. McHenry also raised concerns about reports that White House staff met lobbyists at Caribou Coffee or used personal email accounts to communication, thereby avoiding visitor logs or records management systems.

The issue of officials and staff in the executive branch using non-federal email systems is far from new, however, as Rep. Eleanor Holmes Norton noted in her response to McHenry. Among the thousands of emails from the Bush Administration were those sent by Karl Rove, which now appear lost to history.

It is unfortunate that, due to the politics surrounding the hearing, Noveck did not testify, a choice perhaps driven by media reports of a "showdown," as her knowledge of the use of social software by the federal government would have offered insight to both the subcommittee and the American people, to whom she and the Representatives are both ultimately responsible. After votes to subpoena Noveck and adjourn the hearing were denied on party lines, 5-4, the subcommittee heard from the four witnesses. Hillicon Valley reported further on the political wrangling yesterday.

On video, transparency and Government 2.0

The Oversight Committee posted videos of the testimony and the back and forth between legislators. Additionally, House IT staff said the government hearing on Government 2.0 would be streamed at oversight.house.gov. It was impossible to miss, however, that the committee Twitter account, @OversightDems, didn't issue a single tweet about the hearing. An ironic committee policy came to light as well: only credentialed press were allowed to use laptops at the hearing, hampering the ability of government staff, bloggers or citizens to participate in documenting, discussing or posting status updates about the event online. Emily Long, a reporter from NextGov, didn't find the hearing to be social at all. Given the focus on the elements of open government in 2010, participation, transparency and collaboration, or Government 2.0 as described by McClure, wherein lightweight tools are used to share information about government activities, that policy deserves to be revisited.

Part 1

Part 2

Part 3

Part 4


Related:

July 08 2010

June 03 2010

Velocity Culture: Web Operations, DevOps, etc...


Velocity 2010 is happening on June 22-24 (right around the corner!).  This year we've added third track, Velocity Culture, dedicated to exploring what we've learned about how great teams & organizations work together to succeed at scale. 

Web Operations, or WebOps, is what many of us have been calling these ideas for years.  Recently the term "DevOps" has become a kind of rallying cry that is resonating with many, along with variations on Agile Operations. No matter what you call it, our experiences over the past decade taught us that Culture matters more than any tool or technology in building, adapting, and scaling the web.

Here is a small sample of the upcoming Velocity Culture sessions:

Ops Meta-Metrics: The Currency You Use to Pay For Change
John Allspaw (Etsy.com)
Change to production environments can cause a good deal of stress and strain amongst development and operations teams. More and more organizations are seeing benefits from deploying small code changes more frequently, for stability and productivity reasons. But how can you figure out how much change is appropriate for your application or your culture?

A Day in the Life of Facebook Operations
Tom Cook (Facebook)
Facebook’s Technical Operations team has to balance this need for constant availability with a fast-moving and experimental engineering culture. We release code every day. Additionally, we are supporting exponential user growth while still managing an exceptionally high radio of users per employee within engineering and operations.

This talk will go into how Facebook is “run” day-to-day with particular focus on actual tools in use (configuration management systems, monitoring, automation, etc), how we detect anomalies and respond to them, and the processes we use internally for rapidly pushing out changes while still keeping a handle on site stability.

Change Management: A Scientific Classification
Andrew Shafer (Cloudscaling)
Change management is the combination of process and tools by which changes are made to production systems. Approaches range from cowboy style, making changes to the live site, to complex rituals with secret incantations, coming full circle to continuous deployment. This presentation will highlight milestone practices along this spectrum, establishing a matrix for evaluating deployment process.

There is a tremendous amount happing in our space in the coming weeks in addition to the conference itself.  First, the "Web Operations" book which John Allspaw & I edited goes to print on June 15th.  We're really excited about how it came together.  Then, immediately after Velocity is DevOpsDays, which is a great community event that continues the conversation after Velocity (and is free).  Hope to see you all there!

May 21 2010

Four short links: 21 May 2010

  1. Infrastructures (xkcd) -- absolutely spot-on.
  2. The Michel Thomas App: Behind the Scenes (BERG) -- not interesting to me because it's iPhone, but for the insight into the design process. The main goal here was for me to do just enough to describe the idea, so that Nick could take it and iterate it in code. He’d then show me what he’d built; I’d do drawings or further animations on top of it, and so on and so on. It’s a fantastic way of working. Before long, you start finishing each others’ sentences. Both of us were able to forget about distinguishing between design and code, and just get on with thinking through making together. It’s brilliant when that happens.
  3. Open Government and the World Wide Web -- Tim Berners-Lee offered his "Five-Star" plan for open data. He said public information should be awarded a star rating based on the following criteria: one star for making the information public; a second is awarded if the information is machine-readable; a third star if the data is offered in a non-proprietary format; a fourth is given if it is in Linked Data format; a fifth if it has actually been linked. Not only a good rating system, but a clear example of the significantly better communication by semantic web advocates. Three years ago we'd have had a wiki specifying a ratings ontology with a union of evaluation universes reconciled through distributed trust metrics and URI-linked identity delivered through a web-services accessible RDF store, a prototype of one component of which was running on a devotee's desktop machine at a university in Bristol, written in an old version of Python. (via scilib on Twitter)
  4. Data Access, Data Ownership, and Sharecropping -- With Flickr you can get out, via the API, every single piece of information you put into the system. Every photo, in every size, plus the completely untouched original. (which we store for you indefinitely, whether or not you pay us) Every tag, every comment, every note, every people tag, every fave. Also your stats, view counts, and referers. Not the most recent N, not a subset of the data. All of it. It’s your data, and you’ve granted us a limited license to use it. Additionally we provide a moderately competently built API that allows you to access your data at rates roughly 500x faster then the rate that will get you banned from Twitter. Asking people to accept anything else is sharecropping. It’s a bad deal. (via Marc Hedlund)

April 30 2010

State of the Internet Operating System Part Two: Handicapping the Internet Platform Wars

This post is Part Two of my State of the Internet Operating System. If you haven't read Part One, you should do so before reading this piece.


As I wrote last month, it is becoming increasingly clear that the internet is becoming not just a platform, but an operating system, an operating system that manages access by devices such as personal computers, phones, and other personal electronics to cloud subsystems ranging from computation, storage, and communications to location, identity, social graph, search, and payment. The question is whether a single company will put together a single, vertically-integrated platform that is sufficiently compelling to developers to enable the kind of lock-in we saw during the personal computer era, or whether, Internet-style, we will instead see services from multiple providers horizontally integrated via open standards.


There are many competing contenders to the Internet Operating System throne. Amazon, Apple, Facebook, Google, Microsoft, and VMware all have credible platforms with strong developer ecosystems. Then there is a collection of players with strong point solutions but no complete operating system offering. Let's take them in alphabetical order.

Amazon

With the introduction in 2006 of S3, the Simple Storage Service, and EC2, the Elastic Compute Cloud, Amazon electrified the computing world by, for the first time, offering a general-purpose cloud computing platform with a business model that made it attractive to developers large and small. An ecosystem quickly grew up of companies providing developer and system-management tools. Companies like RightScale provide higher level management frameworks; EngineYard and Heroku provide Ruby-on-Rails based stacks that make it easy to use familiar web tools to deploy applications against an Amazon back-end; the Ubuntu Enterprise Cloud and Eucalyptus offer Amazon-compatible solutions.

A number of competitors, including Rackspace, Terremark, Joyent, GoGrid, and AppNexus are going head to head with Amazon in providing cloud infrastructure services. Many analysts are simply handicapping these providers and comparing them to offerings from Microsoft and Google. But to compare only cloud infrastructure providers is to miss the point. It's a bit like leaving out Microsoft while comparing IBM, Compaq, and Dell when handicapping the PC operating system wars. Hardware was no longer king; the competition had moved up the stack.

The key subsystems of the Internet Operating System are not storage and computation. Those are merely table stakes to get into the game. What will distinguish players are data subsystems.

Data is hard to acquire and expensive to maintain. Delivering it algorithmically at the speeds applications required for reasonable real-time performance is the province of very few companies.

In this regard, Amazon has three major subsystems that give