Newer posts are loading.
You are at the newest post.
Click here to check if anything new just came in.

September 27 2011

The Web 2 Summit Points of Control map has a new data layer

This post was originally published on John Battelle's Searchblog ("The Web 2 Summit Data Layer Is Live").

Earlier this year I posted about an idea we've come up with to create a new "data layer" on top of last year's popular "Points of Control" map. We created this map to visualize the theme of the Web 2 Summit conference, which is coming up again in a few weeks.

As you can see from the map, we've visualized eight key Internet players as cities, with each of the buildings representing storehouses of key data types. Cities are scaled by the size and engagement of their audiences, with data driven by our partner Nielsen and also company-reported sources. A detailed legend is here.

The map is still a work in progress, and there's plenty of opportunity for you to comment on it. And there's more coming — soon anyone will be able to create their own city, based on their own company, or one they think should join the map. Check it out, and stay tuned for more news.

Web 2.0 Summit, being held October 17-19 in San Francisco, will examine "The Data Frame" — focusing on the impact of data in today's networked economy.

Save $300 on registration with the code RADAR


June 07 2011

How can we visualize the big players in the Web 2.0 data layer?

This post was originally published on John Battelle's Searchblog ("Web 2 Map: The Data Layer - Visualizing the Big Players in the Internet Economy").

As I wrote last month, I'm working with a team of folks to redesign the Web 2 Points of Control map along the lines of this year's theme: "The Data Frame." In the past few weeks I've been talking to scores of interesting people, including CEOs of data-driven start ups (TrialPay and Corda, for example), academics in the public data space, policy folks, and VCs. Along the way I've solidified my thinking about how best to visualize the "data layer" we'll be adding to the map, and I wanted to bounce it off all of you. So here, in my best narrative voice, is what I'm thinking.

First, of course, some data.

Data layer chart

On the left hand side are eight major players in the Internet Economy, along with two categories of players that are critical, but which I've lumped together — payment players such as Visa, Amex, and Mastercard, and carriers or ISP players such as Comcast, AT&T, and Verizon.

I've given each company my own "finger in the air" score for seven major data categories, which are shown across the top (I don't claim these are correct, rather, clay on the wheel for an ongoing dialog). The first six scores are in essence percentages, answering the question "What percentage of this company's holdings are in this type of data?" The seventh, which I've called Wildcard data, is a 1-10 ranking of the potency of that company's "wildcard" data that it's not currently leveraging, but might in the future. I'll get to more detail on each data category later.

Toward the far right, I've noted each company's overall global uniques (from Doubleclick, for now, save the carriers and payment guys — I've proxied their size with the reach of Google). There is also an "engagement" score (again, more on that soon). The final score is a very rough tabulation computing engagement over uniques against the sum of the data scores. There are pivots to be built from this data around each of the scores for various types of data, but I'll leave that for later. This is meant to be a relatively simple introduction to my rough thinking about the data layer. Hopefully, it'll spark some input from you.

Now, before you rip it apart, which I fully invite (especially those of you who are data quants, because I am clearly not, and I am likely mixing some apples and watermelons here), allow me to continue to narrate what I'm trying to visualize.

As you know, the map is a metaphor, showing key territories as "points of control." The companies I've highlighted in the chart all have "home territories" where they dominate a sector — Google in search, Facebook in social, Amazon and eBay in commerce, etc. What I plan to do is create a layer based on the data in the chart that, when activated, shows those companies' relative size and strength.

But how?

Web 2.0 Summit, being held October 17-19 in San Francisco, will examine "The Data Frame" — focusing on the impact of data in today's networked economy.

Save $300 on registration with the code RADAR

Well, the best idea we've come up with so far is to show each as a small city of sorts, where the relative height of the buildings is determined by a corresponding data point. So Twitter, for example, will have a tall building in the middle of its city, representing "Interest data." Google's tallest building will be search. Facebook's will be social, and so on. And of course the cities can't be all on the same scale, hence our use of total global uniques, and total engagement. Yahoo may be nearly as big as Facebook, but it doesn't have nearly the engagement per user. So its city will be smaller, relatively, than Facebook's.

Building previewWhat is interesting about this approach is that each company's "cityscape" emerges as distinct. Microsoft's is wide but not tall — they have a lot of data in a number of areas. It will probably end up looking like a suburban office park — funnily enough, that's what Microsoft really looks like, for the most part. Amazon and eBay will have high towers of payment data, with a smattering of shorter buildings. And so on. I don't have a good visualization of this yet, but the designers I'm working with at Blend have sketched out a very rough early version just so you can get the idea (see image to the right). The structures will be more whimsical, and of course be keyed with color. But I think you get the idea.

I'm even thinking of adding other features, like "openness" — i.e., can you access, gain copies of, share, and mash up the data controlled by each company? If so, the city won't be walled. Apple, on the other hand, may well end up a walled city, with a moat, on top of a hill.

Now, a bit more detail on the data categories. You all gave me a lot of really good input on my earlier post, where I posited these original categories. But I've kept them the same, save the addition of the wildcard data. Why? Because I think each can be interpreted as larger buckets containing a lot of other data. I'll go through each briefly in turn:

Purchase Data: This is information about who buys what, in essence. But it's also who almost buys what (abandoned carts), when they buy, in what context, and so on.

Search Data: The original database of intentions — query data, path from query data, "intent" data, and tons more search signals.

Social Data: Social graph, but also identity data. Not to mention how people interact inside their graphs, etc.

Interest Data: This is data that describes what is generally called "the interest graph" — declarations of what people are interested in. It's related to content, but it's not just content consumption. It includes active production of interest datapoints, like tweets, status updates, checkins, etc.

Location Data: This is data about where people are, to be sure, but also data about how often we are there, and other correlated data — i.e., what apps we use in location context, who else is there and when, etc.

Content Data: Content is still a king in our world, and knowing patterns of content consumption is a powerful signal. This is data about who reads/watches/consumes what, when, and in what patterns.

Wildcard Data: This is data that is uncategorized, but could have huge implications. For example, Microsoft knows how people interact with their applications and OS. Microsoft and Google have a ton of language data (phonemes, etc.). Carriers see just about everything that passes across their servers, though their ability to use it might be regulated. Google, Yahoo and Microsoft have tons of email interaction data. And so on ...

Now, of course all these data categories get more powerful as they are leveraged one against the other, and of course, I've left tons of really big data players off the map entirely (small startups like Tynt, Quora, or Sharethis have massive amounts of data, as do very large companies like Nielsen, Quantcast, etc.). But you have to make choices to make something like this work.

So, that's where we are with the Web 2 Summit map data layer. Naturally, once the data layer is live, it will be driven by a database, so we can tweak the size and scope of the cities and buildings based on the collective intelligence of the map users' feedback.

What do you think? What's your input? We'll be building this over the next two months, and I'd love your feedback before we get too far down the line. Thanks!


November 19 2010

We're entering the talent economy

Jeff WeinerNews that Google has more than 2,000 job openings reinforced a point LinkedIn CEO Jeff Weiner made at Web 2.0 Summit: We're entering the "talent economy."

During our interview, I asked Weiner about near-term drivers of the Internet economy. Here's what he said:

... The economy, generally, is going to be increasingly driven by talent. The world has evolved. If you look back at history, we've moved from an agrarian age to the industrial revolution, followed by an information age -- arguably a "meta" information age -- and I think we're transitioning into a talent economy. Where it's not just about the information you know, but about who you know and the information they possess.

... Knowledge is now evolving so quickly, I think it's equally, if not more important, to have access and be connected to the people who have the knowledge you most need to get your job done ... Talent is going to be driving where value gets created.

Weiner touched on a number of other topics during our discussion, including:

  • How LinkedIn uses data science to create relevance from massive streams of information. Case in point: aggregated statistics reveal how companies stack up against competitors in areas like R&D and employee tenure.
  • How new features, like the Career Explorer beta, are combining personal networks and predictive tools to help job seekers find career paths. Put another way: Data tools are shifting the focus from what did happen to what can happen.

The following video contains the full interview:

The marriage of data science and data products will be discussed in-depth at the upcoming Strata Conference (Feb. 1-3, 2011 in Santa Clara, Calif.). Save 20% on registration with the code "STR11RAD."


November 15 2010

Livestream from Web 2.0 Summit

Leaders from across the Internet Economy are converging in San Francisco this week for Web 2.0 Summit. They'll be discussing "points of control" -- the key companies, technologies and platforms that will shape the future of the Internet.

A free (and registration-free) livestream of the presentations is available below. We'll also be posting interviews and sessions on the Web 2.0 Summit site.

Note: Sessions begin at 2:30 PT today (check out the full schedule here). Highlights from past events will be streamed when live sessions aren't be held.

October 29 2010

Points of control = Rents

I watched Tim O'Reilly and John Battelle's "Points of Control" webcast on Wednesday (archived video will be available here soon). I thought it was great and I dug the map. But as I listened, I kept seeing the "Points of Control" notion through a slightly different frame: economic rents.

Web 2.0 Summit 2010Economists use the term "rent" to mean "a return in excess of the resource owner's opportunity cost." That basically means the amount you pay people in excess of what you really have to to get them to do something. In a way, the history of computing has been a history of the evolution of rent-taking within the industry. The fact that we are now talking about "Points of Control" is at least partially because the sources of rent aren't what they used to be, and in our guts that seems bad.

At the very beginning, innovation was the only source of rent in our industry. In a perfectly functioning Schumpeterian system, that's where all rents would come from. But rents from innovation are ephemeral and they are quickly eroded by competition. People and companies get paid enough to keep doing the cool things we want them to do, but they have to keep innovating if they want to own a yacht suitable as a platform for mooning.

Microsoft was early in figuring out that innovation in an open ecosystem could create a network effect, and network effects were very effective barriers to exit. Microsoft's ability to extract rent has been amazing. Innovation was the catalyst, but the rents grew on the back of that network effect to be way out of proportion to the effect of innovation alone. (This weighs heavily on my mind as I contemplate shelling out $279 for Office for Mac so I can effectively share docs with my work colleagues).

A decade or so later, the Internet created new ways to build network effects, and rents at scale. (e.g. eBay).

Now just to be clear, it doesn't mean that we didn't (or don't) get lots of value out of these things. It just means that economically, we didn't have to pay Meg Whitman enough to fund the most expensive campaign for governor to get her (or someone like her) to do what she did. She would have done it for less.

Open source is a really interesting twist in the midst of all this. Software businesses with profit margins greater than the current Treasury yield hate open source because it mostly eliminates rents. Forkability is a rent vaccine, so open source "products" tend to be sold or serviced at just about their producer's opportunity cost. In the case of community based software, it is by definition at opportunity cost, but that cost is as likely to be paid in reputation as in dollars -- making this a conversation on sociology and psychology rather than economics.

In any case, open source software is a leverage-less wasteland from the point of view of anyone that has an MBA. Or, it's a wonderfully rich source of innovation for the people that never liked having rent forcibly extracted. You can see this by comparing the relative market caps of Red Hat and Microsoft as a multiple of revenues when they were at similar stages of growth (number of processors they are running on or similar measures). Microsoft probably had a one- to two-order of magnitude advantage on this measure at any point along their growth curves. Or, as someone from Red Hat once told me, "we love making billion-dollar businesses into hundred-million-dollar businesses."

Obviously digital distribution has also damaged the traditional channel model of the music, film, and photography markets. The impact of this is that the tail-end of the curve can probably shift business models and still make the same money (by touring, selling FLAC files, whatever). But the head -- where the record companies are -- will struggle to extract rents like they used to. As they realize this, they do what rent holders who are losing always do: dispense patronage from their existing franchise and try to influence the law to make their rents more permanent.

Apple has historically lived on rents derived from superior design, which is a very hard thing to do consistently. So they've earned their rents so far. Recently, they've gotten even smarter. The App Store is an MBA's dream because it combines network effects with classic distribution channel control and slotting fees. It also has strong barriers to exit. Interestingly, Foxconn (and its employees) mostly continue to work at opportunity cost levels of renumeration. Rents stay with the leverage and are not evenly distributed through the supply chain.

Apple also finds itself in the odd position of Karmic enforcer. The software developers that once helped destroy content owners' iron-clad grip on distribution now find themselves selling their creations for 30 percent of $.99. Karma is a bitch.

Google extracts amazing rents through a combination of innovation and network effects, although they have really struggled to duplicate their core search / AdSense monopoly. Innovation is keeping Google ahead of Schumpeter for now, but hasn't yet created a second vortex of network effect monopoly. So Bing is an important threat if its share continues to grow. Emerging and effective competition in the area where you are extracting rents will have a non-linear impact on your bottom line. If all goes well (in a Schumpeterian sense), both Bing and Google's search franchises will be rent free in an economic sense. Good for people buying ads, bad for people that hope Google will keep taking the cash thrown off to innovate in other areas (like creating an Office rent-neutralizing alternative in the cloud). It's like watching a pair of Ultra Kaiju trying to choke each other out over Tokyo.

Twitter is the odd case of the network effect without the rent. I like it even more for that.

Both Apple and Google are innovating (obviously) so their network effect and distribution channel rents are at least initiated by innovation. However, we should be observant to those signals that they may be following in Microsoft's, or the record labels' footprints, and attempting to make ephemeral innovation rents more permanent through lobbying, aggressive (and abusive) patent strategies, or other approaches that help them avoid Schumpeterian logic. After all, no one hated the Robber Barons when they were laying track and industrializing America. They were heralded. It wasn't until later, when they distributed rents as patronage to buy politicians (and policies) to lock in the rent streams from those maturing investments, that they became the subjects of political cartoonists.

Mobile telecom providers are my favorite because they are just so obvious. Spectrum auctions are the modern equivalent of the East Indies Trading Company. It's a royal monopoly charter with a direct lineage from the original form. This is where the word "rent" comes from.

We give these charters because we think the physics can't be dealt with without ceding monopolies. Too bad, because they are really bad for innovation. The culture of monopoly becomes part of their DNA. Now as computing goes mobile, that same DNA has found a viral vector and is splicing itself into software and computing companies. Obviously Apple, with its historical predilections, is a more accepting host for the splice, but don't think Google and others won't be immune to rents. The government auctions spectrum to control it, but what they also get is a patronage network of massive proportions.

Facebook is a hard one to describe in the standard terminology. Innovation, yes. Network effect on distribution, absolutely. But also there is this weird bit about knowing more about us than we ourselves know in a conscious way. It's a network effect, but the first to be based on our most personal social network. Maybe this will come to be called "Faustian rent."

Another way of thinking about it is this: for every technology company whose stock we are proud to tell our friends we own, there are significant economic rents being extracted on the other side -- otherwise the money would be equally well invested in treasury bonds. No one brags to their friends about that.

The question is, what is the source of the rents? A company that lives by "deliver more value than you extract" is probably living on innovation rents, at least at the catalyzation point. Then it moves faster than the competition to stay out of the chute of Schumpeter's creative destruction chipper. However, many companies are finding other less palatable sources of rent, or are moving toward them just as fast as their newly minted lobbying crew in DC can make it happen.

As this industry matures, it will either look, economically at least, like a DuPont Styrene plant or it will look like most of Microsoft (the non-Xbox part). A company whose sources of rent have largely tipped from innovation to those based on barriers to exit (lock in), policy (lobby against open source), and patronage (let's bring the entire USAF IT staff to Redmond for a conference and feed them ice cream at every break!). The thing is, those of us that love doing this kind of work love it for the innovation. I hope we aren't getting comfortable with the idea of doing it so that we can build our own Burg Pfalzgrafenstein in software.

From the perspective of buyers, we don't notice rents so much when they are based on innovation. It's early in Apple's transition and the faces of the fangregation are still glowing as they come up Peter Bohlin's 5th Ave. spiral glass staircase. They just spent twice what the device would cost if it was a commodity, but they are grinning as they fondle those little white backpack bags.

On the other hand, rents of the less palatable lock-in variety taste like cod liver oil. And there are some early signs that even for Apple, shiny isn't always adequate salve for that feeling of being taken. Microsoft is the company we love to hate, but Apple may well become the one we hate to love as it relies less on the shiny and more on the locks to keep us paying.

On a related note, last night my Dad said to me: "I can't stand Microsoft and avoid it as much as I can. I've switched to Ubuntu because I got tired of paying Bill Gates a tax so he could run a charity." I thought that was funny.

Associated image on home page courtesy Apple.


October 25 2010

The battle for the Internet Economy

Battles are brewing for the Internet's points of control: search, location, identity, commerce and more. How these battles play out and who emerges victorious will shape virtually everything and everyone involved in the Internet Economy.

Tim O'Reilly and John Battelle will examine the people, organizations, and chokeholds relevant to points of control during a one-hour webcast on Wednesday (Oct. 27) at 1 pm PT / 4 pm ET. Registration and attendance are both free.

"Points of control" is also the theme of this year's Web 2.0 Summit, being held Nov. 15-17 in San Francisco. Learn more about Web 2.0 Summit at the conference site and check out the illustrated Points of Control map.

September 13 2010

Why Twitter's is a game changer

TwitterTwitter has been open with its data from the start, and widely available APIs have created a huge variety of applications and fast adoption. But by making their platform so open, Twitter has fewer options for monetization.

The one thing they can do that nobody else can -- because they're the message bus -- is to rewrite tweets in transit. That includes hashtags and URLs. Twitter could turn #coffee into #starbucks. They could replace a big URL with a short one. And that gives them tremendous power.

Twitter recently announced a new feature that makes this a reality. The URL shortener -- similar to those from,, and tinyURL -- might seem like a relatively small addition to the company's offering. But it's a massive power shift in the world of analytics because now Twitter can measure engagement wherever it happens, across any browser or app. And unlike other URL shorteners, Twitter can force everyone to use their service simply because they control the platform. Your URLs can be shortened (and their engagement tracked by Twitter) whether you like it or not.

Web marketers obsess over the "funnel" -- the steps from first contact to purchase. They try to optimize it constantly, tweaking an offer or moving an image. They want to know everything about a buyer or a visitor.

While every click of a visit to these marketers' sites is analyzed with web analytics, it's much harder to know what people are doing elsewhere on the web. Modern marketers crave insight into two aspects of online consumers' behavior.

  1. They want insight into the "long funnel" -- what happened before someone got to their site that turned a stranger into a visitor.
  2. They want to measure engagement -- more than just knowing how many people a message might have reached, they want to know how many acted on it, regardless of where that link took them.

Web analytics is a huge industry, but the tools marketers rely on to understand visitors are breaking.

Web 2.0 Expo New York - 20% off with code RadarCookies, long the basis for tracking users, need web browsers to store them. In a world where we share URLs via email and social networks, those cookies get lost along the way, and with them the ability to track viral spread of a message. Invasive practices like toolbars and cross-site tracking cookies that try to tie users across websites have triggered huge consumer backlash (that hasn't stopped them from becoming common). Despite adoption, cross-site tracking cookies' days are numbered. This is one of the reasons companies like Tynt are finding other ways of following the spread of messages.

If you're a nosy marketer, it gets worse. We're moving from a browser-centric to an app-centric world. Every time you access the Internet through a particular app -- Facebook, Gowalla, Yelp, Foursquare, and so on -- you're surfing from within a walled garden. If you click on a link, all the marketer sees is a new visit. The referring URL is lost, and with it, the context of your visit.

This is why short URLs are so important. URLs survive the share. Because the interested reader is forced to go to the URL shortener to map the short URL to the real one, whoever owns the shortener sees the engagement between the audience and the content, no matter where it happens. That's why URLs are the new cookies.

Web analytics, marketing and points of control will be discussed at Web 2.0 Expo NY. Radar readers can save 20% on registration with the code "radar."

According to a Twitter email, will "wrap links in Tweets with a new, simplified link." There's good reason to believe this will become the dominant URL shortener. Here's why:

  • Twitter is adding malware detection to the links it shortens.
  • links will include a custom display that shows more of the destination before you click on the link.
  • The company has Twitter clients on most mobile devices, where it can make the default shortener if it wants.
  • The extremely short URL saves precious characters.

Back in late 2008, Twitter was looking for ways to monetize its platform. With, Twitter has found a product marketers will embrace if they want to understand how the world interacts with the messages they put out there.

By now, it's clear that Twitter is not just a site. It's a protocol for asymmetric follow. It's a message bus for human attention. It's able to force every Twitter user to let it know when an interaction happens, simply by changing URLs.

This is the real value of the company -- not just knowing what people are talking about, but knowing which things prompt an action, wherever that happens.


Older posts are this way If this message doesn't go away, click anywhere on the page to continue loading posts.
Could not load more posts
Maybe Soup is currently being updated? I'll try again automatically in a few seconds...
Just a second, loading more posts...
You've reached the end.

Don't be the product, buy the product!