Newer posts are loading.
You are at the newest post.
Click here to check if anything new just came in.

January 09 2014

Four short links: 9 January 2014

  1. Artificial Labour and Ubiquitous Interactive Machine Learning (Greg Borenstein) — in which design fiction, actual machine learning, legal discovery, and comics meet. One of the major themes to emerge in the 2H2K project is something we’ve taken to calling “artificial labor”. While we’re skeptical of the claims of artificial intelligence, we do imagine ever-more sophisticated forms of automation transforming the landscape of work and economics. Or, as John puts it, robots are Marxist.
  2. Clear Flexible Circuit on a Contact Lens (Smithsonian) — ends up about 1/60th as thick as a human hair, and is as flexible.
  3. Confide (GigaOm) — Enterprise SnapChat. A Sarbanes-Oxley Litigation Printer. It’s the Internet of Undiscoverable Things. Looking forward to Enterprise Omegle.
  4. FLIR One — thermal imaging in phone form factor, another sensor for your panopticon. (via DIY Drones)

September 03 2013

What is an enterprise, anyway?

This post was co-authored by Mike Loukides and Bill Higgins.

Bill Higgins of IBM and I have been working on an article about DevOps in the enterprise. DevOps is mostly closely associated with Internet giants and web startups, but increasingly we are observing companies we lump under the banner of “enterprises” trying — and often struggling — to adopt the sorts of DevOps culture and practices we see at places like Etsy. As we tried to catalog the success and failure patterns of DevOps adoption in the enterprise, we ran into an interesting problem: we couldn’t precisely define what makes a company an enterprise. Without a well understood context, it was hard to diagnose inhibitors or to prescribe any particular advice.

So, we decided to pause our article and turn our minds to the question “What is an enterprise, anyway?” We first tried to define an enterprise based on its attributes, but as you’ll see, these are problematic:

More then N employees
Definitions like this don’t interest us. What changes magically when you cross the line between 999 and 1,000 employees? Or 9,999 and 10,000? Wherever you put the line, it’s arbitrary. I’ll grant that 30-person companies work differently from 10,000 person companies, and that 100-person companies have often adopted the overhead and bureaucracy of 10,000 person companies (not a pretty sight). But drawing an arbitrary line in the sand isn’t helpful.

Not “born on the web”
So, Google isn’t an enterprise? Any definition of enterprise that omits some of the largest and richest companies in the world can’t be right. The contrast between web-native companies and companies that predate the web is interesting and important, but that hardly seems like the right way to define “enterprise.”
Dull, hierarchical, stuck in the past

The company your father (or grandfather, maybe) worked for? Sure, there are companies that fit this description, both large and small. These types of companies are not our audience. A company that is stuck in the past isn’t likely to adopt DevOps practices in any meaningful way. Enterprises are not monoliths, and we have found many cases of thoughtful, forward-looking individuals who are members of the DevOps community, learning from the Etsys and Netflixes of the world about how to adapt their cultures and practices to improve their delivery and operations.
Multiple lines of business
As companies grow beyond a single product line or market, they often form semi-autonomous divisions and centralize common functions like IT. However, certain companies — Apple is the exemplar — organize functionally rather than divisionally. Microsoft is following Apple’s lead to adopt a functional org structure. Does this mean Microsoft will soon cease be be an enterprise? Of course not.

What is an enterprise and why does it seemingly present unique challenges to adopting DevOps? After discussing this question with some industry colleagues, we realized our problem was that we were asking the wrong question. Horace Dediu of Asymco provided the key insight by suggesting that whether or not something is an “enterprise” is irrelevant. The key success criterion is an organization’s ability to learn and its willingness to change based on what it learns.

This struck home. Is an organization sufficiently agile to change its practices when the landscape changes? John Allspaw writes about corporate culture and the need to adapt in his article Blameless PostMortems and a Just Culture. The point of doing a post-mortem after a failure is to learn about what went wrong and figure out how to adapt in the future, not to establish blame. Corporations that need to establish blame never learn; they only put in place increasingly inflexible firewalls trying to make sure the same mistakes don’t happen again. The firewalls can’t prevent the next incident (because no two disasters are the same), but they limit the flexibility and freedom the organization needs to improve its operations. Allspaw concludes that you can only learn from your mistakes in a blame-free environment. Corporations can learn when they don’t behave as if everything is a zero-sum game where there are winners and losers.

Art Kleiner struck a similar theme at O’Reilly’s Foo Camp: companies that are successful over the long term need both cultural exceptionalism (we’re different; the rules don’t apply to us), and humility. Sometimes the rules do apply, and we have to go back to the drawing board and redesign our strategies to face the new situations. The ability to build distinctive capabilities is the key to long-term survival, but the distinctive capabilities can become traps unless they are linked to a culture with a discipline of challenging itself. And with this context, whether a corporation is an “enterprise” is surely much less important than whether it can learn. 30-person startups can learn, but they can also be as hidebound and traditionalistic as the largest corporate megalith.

Our original article talked about DevOps as a framework of cultural characteristics, supporting practices, and supporting tools. While this is technically correct, it obscures a key point: culture is the high-order bit — specifically, an organization’s culture as it affects its ability to learn and change. However one defines “enterprise,” what really matters is an organization’s culture — its values, norms of behavior, and reward systems determine whether or not it will be able to evolve, whether the challenge is adopting DevOps, or anything else. This is as true for massive international conglomerates as it is for small startups.

November 30 2012

To eat or be eaten?

One of Marc Andreessen’s many accomplishments was the seminal essay “Why Software is Eating the World.” In it, the creator of Mosaic and Netscape argues for his investment thesis: everything is becoming software. Music and movies led the way, Skype makes the phone company obsolete, and even companies like Fedex and Walmart are all about software: their core competitive advantage isn’t driving trucks or hiring part-time employees, it’s the software they’ve developed for managing their logistics.

I’m not going to argue (much) with Marc, because he’s mostly right. But I’ve also been wondering why, when I look at the software world, I get bored fairly quickly. Yeah, yeah, another language that compiles to the JVM. Yeah, yeah, the Javascript framework of the day. Yeah, yeah, another new component in the Hadoop ecosystem. Seen it. Been there. Done that. In the past 20 years, haven’t we gained more than the ability to use sophisticated JavaScript to display ads based on a real-time prediction of the user’s next purchase?

When I look at what excites me, I see a much bigger world than just software. I’ve already argued that biology is in the process of exploding, and the biological revolution could be even bigger than the computer revolution. I’m increasingly interested in hardware and gadgetry, which I used to ignore almost completely. And we’re following the “Internet of Things” (and in particular, the “Internet of Very Big Things”) very closely. I’m not saying that software is irrelevant or uninteresting. I firmly believe that software will be a component of every (well, almost every) important new technology. But what grabs me these days isn’t software as a thing in itself, but software as a component of some larger system. The software may be what makes it work, but it’s not about the software.

A dozen or so years ago, people were talking about Internet-enabled refrigerators, a trend which (perhaps fortunately) never caught on. But it led to an interesting exercise: thinking of the dumbest device in your home, and imagine what could happen if it was intelligent and network-enabled. My furnace, for example: shortly after buying our house, we had the furnace repairman over 7 times during the month of November. And rather than waiting for me to notice that the house was getting cold at 2AM, it would have been nice for a “smart furnace” to notify the repairman, say “I’m broken, and here’s what’s probably wrong.” (The Nest doesn’t do that, but with a software update it probably could.)

The combination of low-cost, small-footprint computing (the BeagleBone, Raspberry Pi, and the Arduino), along with simple manufacturing (3D printing and CNC machines), and inexpensive sensors (for $150, the Kinect packages a set of sensors that until recently would easily have cost $10,000) means that it’s possible to build smart devices that are much smaller and more capable than anything we could have built back when we were talking about smart refrigerators. We’ve seen Internet-enabled scales, robotic vacuum cleaners, and more is on the way.

At the other end of the scale, GE’s “Unleashing the Industrial Internet” event had a fully instrumented network-capable jet engine on stage, with dozens of sensors delivering realtime data about the engine’s performance. That data can be used for everything from performance optimization to detecting problems. In a panel, Tim O’Reilly asked Matt Reilly of Accenture “do you want more Silicon Valley on your turf?” and his immediate reply was “absolutely.”

Even in biology: synthetic biology is basically nothing more than programming with DNA, using a programming language that we don’t yet understand and for which there is still no “definitive guide.” We’re only beginning to get to the point where we can reliably program and build “living software,” but we are certainly going to get there. And the consequences will be profound, as George Church has pointed out.

I’m not convinced that software is going to eat everything. I don’t see us living in a completely virtual world, mediated completely by browsers and dashboards. But I do see everything eating software: software will be a component of everything we do or buy, from our clothing to our food. Why is the FitBit a separate device? Why not integrate it into your shoes? Can we imagine cookies that incorporate proteins that have been engineered to become unappealing when we’ve eaten too much? Yes, we can, though we may not be happy about that. Seriously, I’ve had discussions about genetically engineered food that would diagnose diseases and turn different colors in your digestive track to indicate cancer and other conditions. (You can guess how you read the results).

Andreessen is certainly right in his fundamental argument that software has disrupted, and will continue to disrupt, just about every industry on the planet. He pointed to health care and education as the next industries to be disrupted; and we’re certainly seeing that, with Coursera and Udacity in education, and conferences like StrataRx in health care. We just need to push his conclusion farther. Is a robotic car a case of software eating the driver, or of the automobile eating software? You tell me. At the Industrial Internet event, Andreessen was quoted as saying “We only invest in hardware/software hybrids that would collapse if you pulled the software out.” Is an autonomous car something that would collapse if you pulled the software out? The car is still drivable. In any case, my “what’s the dumbest device in the house” exercise is way too limiting. When are we going to build something that we can’t now imagine, that isn’t simply an upgrade of what we already have? What would it mean for our walls and floors, or our plumbing, to be intelligent? At the other extreme, when will we build devices where we don’t even notice that they’ve “eaten” software? Again, Matt Reilly: “It will be about flights that are on time, luggage that doesn’t get lost.”

In the last few months, I’ve seen a number of articles on the future of venture investment. Some argue that it’s too easy and inexpensive to look for “the next Netscape,” and as a consequence, big ambitious projects are being starved. It’s hard for me to accept that. Yes, there’s a certain amount of herd thinking in venture capital, but investors also know that when everyone is doing the same thing, they aren’t going to make any money. Fred Wilson has argued that momentum is moving from consumer Internet to enterprise software, certainly a market that is ripe for disruption. But as much as I’d like to see Oracle disrupted, that still isn’t ambitious enough.

Innovation will find the funds that it needs (and it isn’t supposed to be easy). With both SpaceX and Tesla Motors, Elon Musk has proven that it’s possible for the right entrepreneur to take insane risks and make headway. Of course, neither has “succeeded,” in the sense of a lucrative IPO or buyout. That’s not the point either, since being an entrepreneur is all about risking failure. Neither SpaceX nor Tesla are Facebook-like “consumer web” startups, nor even enterprise software startups or education startups. They’re not “software” at all, though they’ve both certainly eaten a lot of software to get where they are. And that leads to the most important question:

What’s the next big thing that’s going to eat software?


May 18 2012

Top Stories: May 14-18, 2012

Here's a look at the top stories published across O'Reilly sites this week.

A federal judge learned to code
The judge presiding over the Oracle/Google case learned Java, and that skill came in handy when coding specifics arose during the trial. It's proof that coding is a part of cultural competence, even if you never do it professionally.

The chicken and egg of big data solutions
So, here we are with all of this disruptive big data technology, but we seem to have lost the institutional wherewithal to do anything with it in a lot of large companies, at least until package solutions come along.

DIY learning: Schoolers, Edupunks, and Makers challenge education
Schoolers, Edupunks and Makers are showing us what's possible when learners, not institutions, own the education that will define their lives.

John Allspaw on DevOps
John Allspaw discusses DevOps in high-volume web companies and the importance of cooperation between development and operations.

JavaScript and Dart: Can we do better?
O'Reilly editor Simon St. Laurent talked with Google's Seth Ladd about the challenges of improving the web. How can we build on JavaScript's ubiquity while addressing performance, team, and scale issues?

Velocity 2012: Web Operations & Performance — The smartest minds in web operations and performance are coming together for the Velocity Conference, being held June 25-27 in Santa Clara, Calif. Save 20% on registration with the code RADAR20.

May 16 2012

The chicken and egg of big data solutions

Before I came to O'Reilly I was building the "big data and disruptive analytics practice" at a major systems integrator. It was a blast to spend every week talking to customers in different industries who were waking up to the possibilities that technologies like Hadoop offered their businesses. Many of these businesses are going to fundamentally change as they embrace this stuff (or be replaced by those that do). But there's a catch.

Twenty years or so ago large integrators made big business building applications on the then-new relational paradigm. They put in Oracle databases with custom code, wrote PowerBuilder apps on Sybase, and of course lots of businesses rolled their own with VB and SQL Server. It was an era of custom coding where Oracle, Sybase, SQL Server, Informix and etc. were thought of as platforms to build stuff on.

Then the market matured and shifted to package solution implementation. ERP, CRM, …, etc. The big guys focused on integrating again and told their clients there was no ROI in building custom stuff. ROI would come from integrating best-of-breed solutions. Databases became commodity back ends to the applications that were always the real focus.

Now along comes big data, NoSQL, data science, and all that stuff and it seems like we're starting the cycle over again. But this time clients, having been well trained over the last decade or so, aren't having any of that "build it from scratch" mentality. They know that Hadoop and other new technologies can be transformative to their business, but they want it packaged up and solution'ified like they are used to. I heard a lot of "let us know when you have a solution already built or available to buy that does X" in the last year.

Also, lots of the shops that do this stuff at scale are built and staffed around the package implementation model and have shed many of the skills they used to have for custom work. Everything from staffing models to methodologies are oriented toward package installation.

So, here we are with all of this disruptive technology, but we seem to have lost the institutional wherewithal to do anything with it in a lot of large companies. Of course that fact was hard on my numbers. I had a great pipeline of companies with pain to solve, and great technologies to solve it, but too much of the time it was hard to close it without readymade solutions.

Every week I talked to the companies building these new platforms to share leads and talk about their direction. After a while I started cutting them off when they wanted to talk about the features of their next release. I just got to the point where I didn't really care, it just wasn't all that relevant to my customers. I mean, it's important that they are making the platforms more manageable and building bridges to traditional BI, ETL, RDBMS, and the like. But the focus was too much on platforms and tools.

I wanted to know "What are you doing to encourage solution development? Are you staffing a support system for ISVs? What startups and/or established players are you aware of that are building solutions on this platform?" So when I saw this tweet I let out a little yelp. Awesome! The lack of ready-to-install solutions was getting attention, and from Mike Olsen.

You can watch the rest of what Mike Olson said here and you'll find he tells a similar story about the RDBMS historical parallel.

I talked to Mike a few weeks ago to find out what was behind his comment and explore what else they are doing to support solution development. It boils down to what he said — he will help connect you with money — plus a newly launched partner program designed to provide better support to ISVs among others. Also, the continued attention to APIs and tools like Pig and Hive should make it easier for the solution ecosystem to develop. It can only be good for his business to have lots of other companies directly solving business problems, and simply pulling in his platform.

Hortonworks also started a partner program in the fall and I think we'll see a lot more emphasis on this across the space this year. However, at the moment wherever I look (Hortonworks partners, Cloudera Partners, Accel big data portfolio) the focus today remains firmly on platform and tools or partnering with integrators. Tresata, a startup focused on financial risk management, pops up in in a lot of lists as the obvious odd one out — an actual domain-specific solution.

What about other people that could be building solutions? Is it the maturity level of the technology, the lack of penetration of Hadoop etc. into your customer's data centers, or some combination of other factors that is slowing things down?

Of course, during the RDBMS adoption it took a lot of years before the custom era was over and thoroughly replaced by the era of package implementation. The question I'm pondering is whether customer expectations and the pace of technology will make it happen faster this time? Or is the disruptive value of big data going to continue to accrue only to risk-taking early adopters for the foreseeable future?

If you are building a startup based on a solution or application that leverages big data technology, and you aren't being stealthy, I'd love to hear about it in the comments.


January 23 2012

Survey results: How businesses are adopting and dealing with data

On December 7, 2011, we held our fifth Strata Online Conference. This series of free web events brings together analysts, innovators and researchers from a variety of fields. Each conference, we look at a particular facet of the move to big data — from personal analytics, to disruptive startups, to enterprise adoption.

This time, we focused on how businesses are going to embrace big data, and where the challenges lie. It was a perfect opportunity to survey the attendees and get a glimpse into enterprise adoption of big data. Out of the roughly 350 attendees, approximately 100 agreed to give us their feedback on a number of questions we asked. Here are the results.

Some basic facts

While the attendees worked for a mix of commercial, educational, government, and non-profit companies, the vast majority (82%) worked for a commercial, for-profit company.

What kind of organization do you work for?
Click to enlarge.

Most of the attendees' organizations were also fairly large — more than half of them had 500 co-workers, and 22% of them had more than 10,000.

How big is your organization?
Click to enlarge.

We used this demographic information to segment and better analyze the other three questions we asked.

Big data adoption and challenges

We then asked attendees about their journey to big data. Fewer than 20% of them already have a big data solution in place — which we clarified to mean some kind of massive-scale, sharded, NoSQL, parallel data query system that may employ interactivity and machine-assisted data exploration. More than a quarter said they have no plans at this time.

How soon do you expect to implement a big data solution?
Click to enlarge.

While it's relatively early days for adoption, more than 60% of attendees said they were in the process of gathering information on big data and what it meant to them. This is a spurious result at best: we're of course selecting an audience that wants to be an audience. Nevertheless, the volume of attendees and their feedback suggests that deployment is ramping up: if you're a big data vendor, this is the time to be fighting for mindshare.

What's the biggest challenge you see with big data?
Click to enlarge.

When it comes to actually deploying big data, companies have plenty of challenges. The big ones seem to be:

  • Data privacy and governance.
  • Defining what big data actually is.
  • Integrating big data with legacy systems.
  • A lack of big data skills.
  • The cost of tools.

Analyzing a bit further

These results might be informative, but what we really want to know is how they correlate. After all, Strata is a data conference: we'd be remiss if we didn't crunch things a bit!

First, we wondered whether there's a relationship between the size of a company and the kinds of problems it's experiencing with big data.

Obstacles by company size
Click to enlarge.

Our results suggest that governance and skill shortages are problems for larger companies, and that smaller businesses worry much less about data privacy and integrating legacy systems. Cost concerns come largely from mid-sized businesses.

Then we wondered whether adoption is tied to company size.

Big data adoption progress by company size
Click to enlarge.

Among our attendees, smaller firms were ahead of the game: none of the companies larger than 500 employees said they had big data in place today.

We also found that educational, government, and NGO respondents didn't list cost as a top concern, suggesting that they may have a tolerance for open-source or home-grown approaches.

Obstacles by company type
Click to enlarge.

Of course, the number of responses from these segments isn't statistically significant, but it warrants further study, particularly for commercial offerings trying to sell outside the for-profit world.

Finally, we wondered whether the things a company worries about change as it goes from "just browsing" to "trying to build."

Obstacles by time to implement
Click to enlarge.

Concerns do seem to shift over the course of adoption and maturity. Early on, companies struggle to define what big data is and worry about staffing. As they get closer to implementation, their attention shifts to legacy system integration. Once they have a system, talent shortages and a variety of other, more specific concerns emerge.

While not a hard-core study — respondents weren't randomly selected, the number of responses within some segments isn't statistically significant, and so on — this feedback does suggest that there's a large demand for clear information on what big data is and how it'll change business, and that as enterprises move to adopt these technologies they'll face integration headaches and staffing issues.

The next free Strata Online Conference will be held on January 25. We'll be taking a look at what's in store for the upcoming Strata Conference (Feb 28-March 1 in Santa Clara, Calif).

Strata 2012 — The 2012 Strata Conference, being held Feb. 28-March 1 in Santa Clara, Calif., will offer three full days of hands-on data training and information-rich sessions. Strata brings together the people, tools, and technologies you need to make data work.

Save 20% on registration with the code RADAR20


November 30 2011

Big data goes to work

Companies that are slow to adopt data-driven practices don't need to worry about long-term plans — they'll be disrupted out of existence before those deadlines arrive. And even if your business is on the data bandwagon, you shouldn't get too comfortable. Shifts in consumer tolerances and expectations are quickly shaping how businesses apply big data.

Alistair Croll, Strata online program chair, explores these shifts and other data developments in the following interview. Many of these same topics will be discussed at "Moving to Big Data," a free Strata Online Conference being held Dec. 7.

How are consumer expectations about data influencing enterprises?

Alistair CrollAlistair Croll: There are two dimensions. First, consumer tolerance for sharing data has gone way up. I think there's a general realization that shared information isn't always bad: we can use it to understand trends or fight diseases. Recent rulings by the Supreme Court and legislation like the Genetic Information Nondiscrimination Act (GINA) offer some degree of protection. This means it's easier for companies to learn about their customers.

Second, consumers expect that if a company knows about them, it will treat them personally. We're incensed when a vendor that claims to have a personal connection with us treats us anonymously. The pact of sharing is that we demand personalization in return. That means marketers are scrambling to turn what they know about their customers into changes in how they interact with them.

What's the relationship between traditional business intelligence (BI) and big data? Are they adversaries?

Alistair Croll: Big data is a successor to traditional BI, and in that respect, there's bound to be some bloodshed. But both BI and big data are trying to do the same thing: answer questions. If big data gets businesses asking better questions, it's good for everyone.

Big data is different from BI in three main ways:

  1. It's about more data than BI, and this is certainly a traditional definition of big data.
  2. It's about faster data than BI, which means exploration and interactivity, and in some cases delivering results in less time than it takes to load a web page.
  3. It's about unstructured data, which we only decide how to use after we've collected it and need algorithms and interactivity in order to find the patterns it contains.

When traditional BI bumps up against the edges of big, fast, or unstructured, that's when big data takes over. So, it's likely that in a few years we'll ask a business question, and the tools themselves will decide if they can use traditional relational databases and data warehouses or if they should send the task to a different architecture based on its processing requirements.

What's obvious to anyone on either side of the BI/big data fence is that the importance of asking the right questions — and the business value of doing so — has gone way, way up.

How can businesses unlock their data? What's involved in that process?

Alistair Croll: The first step is to ask the right questions. Before, a leader was someone who could convince people to act in the absence of clear evidence. Today, it's someone who knows what questions to ask.

Acting in the absence of clear evidence mattered because we lived in a world of risk and reward. Uncertainty meant we didn't know which course of action to take — and that if we waited until it was obvious, all the profit would have evaporated.

But today, everyone has access to more data than they can handle. There are simply too many possible actions, so the spoils go to the organization that can choose among them. This is similar to the open-source movement: Goldcorp took its geological data on gold deposits — considered the "crown jewels" in the mining industry — and shared it with the world, creating a contest to find rich veins to mine. Today, they're one of the most successful mining companies in the world. That comes from sharing and opening up data, not hoarding it.

Finally, the value often isn't in the data itself; it's in building an organization that can act on it swiftly. Military strategist John Boyd developed the observe, orient, decide and act (OODA) loop, which is a cycle of collecting information and acting that fighter pilots could use to outwit their opponents. Pilots talk of "getting inside" the enemy's OODA loop; companies need to do the same thing.

So, businesses need to do three things:

  1. Learn how to ask the right questions instead of leading by gut feel and politics.
  2. Change how they think about data, opening it up to make the best use of it when appropriate and realizing that there's a risk in being too private.
  3. Tune the organization to iterate more quickly than competitors by collecting, interpreting, and testing information on its markets and customers.
Moving to Big Data: Free Strata Online Conference — In this free online event, being held Dec. 7, 2011, at 9AM Pacific, we'll look at how big data stacks and analytical approaches are gradually finding their way into organizations as well as the roadblocks that can thwart efforts to become more data driven. (This Strata Online Conference is sponsored by Microsoft.)

Register to attend this free Strata Online Conference

What are the most common data roadblocks in companies?

Alistair Croll: Everyone I talk to says privacy, governance, and compliance. But if you really dig in, it's culture. Employees like being smart, or convincing, or compelling. They've learned soft skills like negotiation, instinct, and so on.

Until now, that's been enough to win friends and influence people. But the harsh light of data threatens existing hierarchies. When you have numbers and tests, you don't need arguments. All those gut instincts are merely hypotheses ripe for testing, and that means the biggest obstacle is actually company culture.

Are most businesses still in the data acquisition phase? Or are you seeing companies shift into data application?

Alistair Croll: These aren't really phases. Companies have a cycle — call it a data supply chain — that consists of collection, interpretation, sharing, and measuring. They've been doing it for structured data for decades: sales by quarter, by region, by product. But they're now collecting more data, without being sure how they'll use it.

We're also seeing them asking questions that can't be answered by traditional means, either because there's too much data to analyze in a timely manner, or because the tools they have can't answer the questions they have. That's bringing them to platforms like Hadoop.

One of the catalysts for this adoption has been web analytics, which is, for many firms, their first taste of big data. And now, marketers are asking, "If I have this kind of insight into my online channels, why can't I get it elsewhere?" Tools once used for loyalty programs and database marketing are being repurposed for campaign management and customer insight.

How will big data shape businesses over the next few years?

Alistair Croll: I like to ask people, "Why do you know more about your friends' vacations (through Facebook or Twitter) than about whether you're going to make your numbers this quarter or where your trucks are?" The consumer web is writing big data checks that enterprise BI simply can't cash.

Where I think we'll see real disruption and adoption is in horizontal applications. The big data limelight is focused on vertical stuff today — genomics, algorithmic trading, and so on. But when it's used to detect employee fraud or to hire and fire the right people, or to optimize a supply chain, then the benefits will be irresistible.

In the last decade, web analytics, CRM, and other applications have found their way into enterprise IT through the side door, in spite of the CIO's allergies to outside tools. These applications are often built on "big data," scale-out architectures.

Which companies are doing data right?

Alistair Croll: Unfortunately, the easy answer is "the new ones." Despite having all the data, Blockbuster lost to Netflix; Barnes & Noble lost to Amazon. It may be that, just like the switch from circuits to packets or from procedural to object-oriented programming, running a data-driven business requires a fundamentally different skill set.

Big firms need to realize that they're sitting on a massive amount of information but are unable to act on it unless they loosen up and start asking the right questions. And they need to realize that big data is a massive disintermediator, from which no industry is safe.

This interview was edited and condensed.


August 31 2011

Four short links: 31 August 2011

  1. OSMdroid -- The OpenStreetMapView is a (almost) full/free replacement for Android's MapView class. Also see this tutorial. (via Simon Gianoutsos)
  2. 10 Immutable Laws of Security (Microsoft) -- an oldie but a goodie. Law #1: If a bad guy can persuade you to run his program on your computer, it's not your computer anymore.
  3. What's in The Trough? (BERG London) -- as a predictor or similar tool for action, the Gartner Hype Cycle is comically useless. As a tool for brainstorming, as BERG point out, it's fantastic.
  4. JP Rangaswami's Enterprise Gamification (Livestream) -- video of JP's "Enterprise Gamification" talk. As Kevin Slavin points out, the introduction is cheesily bad but the talk is pantswettingly good.

August 26 2011

Top Stories: August 22-26, 2011

Here's a look at the top stories published across O'Reilly sites this week.

Ruminations on the legacy of Steve Jobs
Apple, under Steve Jobs, has always had an unrelenting zeal to bring humanity to the center of the ring. Mark Sigal argues that it's this pursuit of humanity that may be Jobs' greatest innovation.
The nexus of data, art and science is where the interesting stuff happens
Jer Thorp, data artist in residence at the New York Times, discusses his work at the Times and how aesthetics shape our understanding of data.
Inside Google+: The virtuous circle of data and doing right by users
Data liberation and user experience emerged as core themes during a recent discussion between Tim O'Reilly and Google+ VP of Product Bradley Horowitz.
Five things Android needs to address on the enterprise side
Android has the foundation to support enterprise use, but there's a handful of missing pieces that need to be addressed if it's going to fully catch on in the corporate world.
The Daily Dot wants to tell the web's story with social data journalism
The newly launched Daily Dot is trying an experiment in community journalism, where the community is the Internet. To support their goal, they're applying the lens of data journalism to the social web.

Strata Conference New York 2011, being held Sept. 22-23, covers the latest and best tools and technologies for data science — from gathering, cleaning, analyzing, and storing data to communicating data intelligence effectively. Save 30% on registration with the code STN11RAD.

August 25 2011

Five things Android needs to address on the enterprise side

My lovely cubicle by ashley_dryden, on FlickrAndroid has the foundation to support enterprise use, but there's a handful of missing pieces that need to be addressed if it's going to fully catch on in the corporate world. Below I look at five enterprise areas that Google and third-party developers need to work on.

Managing the device fleet

A typical enterprise needs to have a way of managing a fleet of devices, whether personal or company owned. There are currently a number of vendors providing solutions to this problem, including 3LM, Good Technology, MobileIron, and Sybase.

What needs to happen: Google needs to help create a standard for a complete enterprise Android solution, or it must support one from a third party. Until recently, the closest candidates were the Motorola Droid Pro and Photon lines, but Google's planned acquisition of Motorola could yield a full enterprise option. Keep in mind that Motorola already owns 3LM, one of the leaders in Android security solutions.

Enforcing security policies

CIOs need to enforce their security policies, and they also want to be able to wipe a lost or stolen device. Android does provide the plumbing for most of this work and third-party vendors are starting to create solutions on top of it, such as Motorola's Enterprise Device Policy Management API and related MotoBlur solutions.

What needs to happen: This market is getting fragmented, and CIOs will need to do their own research for the right solution for their particular enterprise.

Securing connections to enterprise networks

Most corporate networks are secured with either SSL or VPN solutions. Android supports both, at least on paper. The problem is that corporate America typically uses proprietary VPN solutions from vendors like Cisco and Juniper. That means that most Android devices do not offer any useful VPN options to corporate users. This is a big issue that is slowly being addressed by device manufacturers. Companies like Samsung are entering into licensing agreements with the Ciscos of the world to make sure enterprise-grade VPN is part of their Android product lines.

What needs to happen: Carriers or OEMs need to bundle the right VPN solutions with their devices. We're starting to see this with certain Motorola models on the Verizon and Sprint networks.

Android Open, being held October 9-11 in San Francisco, is a big-tent meeting ground for app and game developers, carriers, chip manufacturers, content creators, OEMs, researchers, entrepreneurs, VCs, and business leaders.

Save 20% on registration with the code AN11RAD

Sandboxing apps

I often hear IT people say they want to control the types of applications and content users can download to their company phones. While it's possible to wall off a company-issued device, it's an expensive strategy that creates a false sense of security. A better approach may be to allow coexistence of both corporate and personal applications on the same device. Android already provides solid application sandboxing, which isolates data so each app has its own data privacy.

What needs to happen: IT departments need to provide enterprise-grade apps for enterprise data. Those departments must also get used to corporate apps coexisting on devices with consumer apps. A good example of enterprise apps is Google's Apps for Enterprise cloud solution and its mobile counterparts, such as GMail, GTalk, and Docs.

Trusted markets for business apps

Google's Android Market is based on reactive testing that basically crowd sources quality assurance. That model won't cut it for corporate clients. The rise of enterprise-friendly boutique markets, like Cisco AppHQ, could provide the needed alternatives for enterprise adoption.

What needs to happen: The free market needs to work its magic. Multiple app stores are a good thing, and eventually consumers will know which brand to trust for certain types of applications. Google could help the process by allowing other stores to list their apps on Google's Android Market. Carriers could also pre-load multiple store apps.

The future of Android in the enterprise

While Android doesn't come with all the enterprise bells and whistles, it's built on a strong and secure foundation. And while Google needs to do more to provide the missing pieces, the company has created the infrastructure for other companies to step in and fill out Android's enterprise offerings. The strategy appears to be working, as research has found Android to be gaining adoption within corporate IT departments. As more employees bring Android devices into their offices, and as Android's corporate offerings mature, I expect enterprise acceptance to accelerate in the years ahead.

Background Photo: My lovely cubicle by ashley_dryden, on Flickr


July 12 2011

Is the enterprise dead as a tablet strategy?

In "HP's Tortured WebOS Positioning," Jean-Louis Gassée makes the assertion that the consumerization of IT renders the "enterprise-only" pivot null and void.

I disagree, but first some clarity for those who aren't familiar with the term "consumerization of IT."

When the enterprise lost its mojo

Once upon a time, large enterprises (think: Fortune 2000 companies) were widely perceived to be the ideal customer, owing to their large size, well-defined and massive IT budgets, wide range of solution needs, and target-ability from a sales perspective.

All sorts of hardware, software, hosted services and consulting services companies — not to mention a significant chunk of the venture capital industry — fed off of this massive ecosystem, the impact of which meant that innovation began in the enterprise, and then trickled down to the consumer.

However, when the dotcom bubble blew up at the end of 2000, coinciding with the end of the over-hyped Y2K project "pig trough," enterprises lost the impetus to spend aggressively on IT.

In parallel, they began to (rightly) question the return on investment for the many projects they had funded. In broad terms, this led to a reclassification of IT from being a strategic asset, and core differentiator, to being a liability, and a necessary evil.

Basically, the CFO trumped the CIO going forward.

The neutering of the enterprise from an IT perspective coincided almost perfectly with the second coming of a consumer-focused Apple (which frankly, has never grokked the enterprise).

Now suddenly innovation began originating in the consumer realm, and then trickling to the enterprise after it was proven to be a safe investment.

Web 2.0 Summit, being held October 17-19 in San Francisco, will examine "The Data Frame" — focusing on the impact of data in today's networked economy.

Save $300 on registration with the code RADAR

Is the enterprise dead as a player in tablet devices?

Call me naive, but speaking from the perspective of an iOS developer (and someone who used to sell a bunch of hardware and software solutions to enterprises), I think an opportunity exists for some tablet maker to penetrate the enterprise.

Why? Apple's model of controlling both the value-add hardware channel and the software distribution channel is a decided anathema to enterprises, which typically prefer working with and through VAR channels (i.e., Value-Added Resellers) and System Integrators (SI).

Similarly to VARs and SIs, the Apple model heavily complicates the types of solutions that vendors can provide, the pricing that vendors can achieve, the ability to not broadcast key customers to competitors, and the like.

And don't even get me started on Android as an alternative. Here, Google's focus on free, loosely-coupled, "good enough" and ad-supported is a distinctly different set of sales and support assumptions than the enterprise, which is all about high-touch, custom and deeply integrated, has come to expect.

Thus, in the bigger picture the open questions are two-fold. One, is whether there exists a vendor in the tablet space other than Apple that is going to properly set and meet market expectations, as opposed to perennially over-promising, and under-delivering.

This, of course, requires an actual product discipline, inclusive of coherent evangelism, a clear roadmap and release strategy, and a culture of focused execution and iteration — especially on the software side of the equation.

Here, the litmus test is actually kind of easy. Until one of these manufacturers stops talking about Adobe's Flash as a feature (versus a bug, until it works caveat-free); or touting Snapdragon processors and their clock speeds (customers buy outcomes, not attributes), Apple is going to remain the only credible player in tablets.

There is just too much of a halo effect working to Apple's advantage, and the company has high execution credibility in this domain.

By contrast, RIM, the long-time enterprise leader of mobile devices, has clearly (so far) screwed the pooch with their confused PlayBook strategy. And Gassée's piece covers HP's early tablet missteps.

Attacking undefended hills

HP, IBM, Oracle and Microsoft all have logical entry points, from a solutions perspective, into the enterprise via tablet computing. But they must get focus and religion on attacking the "undefended hill" (to use an HP axiom) that is the enterprise tablet.

Simply put, no one is credibly focusing on this customer and its surrounding eco-channel.

Which brings me to the second question: Does there exist even one enterprise with sufficient vision to be "greedy" from an investment and innovation perspective while every one of their peers is acting scared (to use a Warren Buffett axiom)?

After all, it only take one serious proof point to ignite a market, and if HP, et al are really serious, proving out the enterprise should be a core focus.

Netting it out: It's too early to proclaim game over when the stakes are measured in the billions of devices, the budgets are measured in the billions of dollars, and a sleeping gorilla lies naked, unfed and uncared for in the enterprise.


March 16 2011

Knowledge management in the age of social media

Twitter and office buildingKnowledge management, which is broadly defined as the identification, retention, effective use, and retirement of institutional insight, has been an elusive goal for most large organizations. It is motivated by practical business intent, such as the distribution of knowledge to avoid relearning the same best practices over and over. However, in reality it is a requirement that is remarkably difficult to attain. Some of the smartest people I have worked with have been frustrated by their efforts, not through lack of trying or ability but by the inherent challenges it presents. The emergence and impact of social media in the enterprise forces us to rethink knowledge management and creates completely new challenges.

Today, some of the core issues with existing knowledge management approaches can be categorized as behavioral and technical (I recognize the complexity of the subject and acknowledge there are many more qualities to examine; using the following two should be sufficient to support the points in this blog).

1. Behavioral

In order for a knowledge management system (KMS) to have value, employees must enter insight on a regular basis and they must keep the knowledge current (we can all agree that out-of-date information, which has reference value, is much less useful as the general desired state of actionable knowledge). Seldom are either of these behaviors adequately incentivized. By sharing tacit knowledge, many employees believe they are reducing their own value to the organization. In addition, updating the information requires effort, which is rarely a priority against the core responsibilities of the employee.

Psychic income earned on your own time might provide incentive for Wikipedia updates, but it doesn't often translate to well spent effort on company time.

2. Technical

When presenting to audiences I often ask if it is easier for them to use a public search engine to find information about their organizations or use their own organizations' websites for a search. As you might guess, the majority of the room goes with the public engine.

It's remarkably difficult to organize information in the right manner, make it searchable, and then present it so the most relevant responses are at the top of the search results. In addition, organizational information is hardly the example of pristine structure. While public search engines benefit from counting the number of links between items (a good measure of popularity), internal systems have no such equivalent. Unstructured content is the king of the public web, whereas it is the bane of the enterprise. (Things will change in the future as new technologies, such as those that support the semantic web, are broadly adopted and implemented).

The situation is compounded when employees are disillusioned by the effectiveness and effort to use the KMS and resort to old habits, like asking colleagues or improvising in the absence of guidance (thus repeating mistakes or missing best practices). The system often fails to be adopted — or at best is used by a small proportion of the organization — and no amount of resuscitation is enough to bring it back to life.

Web 2.0 Expo San Francisco 2011, being held March 28-31, will examine key pieces of the digital economy and the ways you can use important ideas for your own success.

Save 20% on registration with the code WEBSF11RAD

Social media completely changes the existing knowledge management paradigm

It may be time to put down your tools in trying to make the old model of knowledge management work; social media is a completely new beast that changes many of the rules.

In the old world order, knowledge was usually created and stored as a point in time. In the future, organizational policy or insight may not be formed by an individual creating a document that goes through an approval process and is ultimately published. It will likely begin with an online conversation and it will be forever evolving as more people contribute and circumstances change.

Social media takes knowledge and makes it highly iterative. It creates content as a social object. That is, content is no longer a point in time, but something that is part of a social interaction, such as a discussion. It easily disassembles the pillars of structure as it evolves. As examples: content in a micro-blogging service can shift meaning as a discussion unfolds; conversations in enterprise social networks that link people and customer data can defy categorization; and internal blogs and their comments don't lend themselves to obvious taxonomy.

The days of the single, authoritative voice are coming to an end. The community has prevailed.

The shift to the adoption of enterprise social computing, greatly influenced by consumerization, points to one emergent observation: the future is about managing unstructured content.Let's consider the magnitude of this for a moment. Years of effort, best practices, and technologies for supporting organizational insight in the form of curated, structured insight has to be rethought. It's an enormous challenge, but it may in fact be the best thing that ever happened to knowledge management.

There is an important silver lining to this story

In the long run, social media in the enterprise will likely be a boon for knowledge management. It should mean that many of the benefits we experience in the consumer web space — effective searching, grouping of associated unstructured data sources, and ranking of relevance — will become basic features of enterprise solutions. It's likely we'll see the increasing overlap between public and private data to enhance the value of the private data.

For example: want to know more about a staff member? Internal corporate information will include role, start date, department etc., but we may get additional information pulled in from social networks, such as hobbies, photos or previous employment. Pull up client data and you'll get the information keyed in by other employees, but you might also get the history and values of the company, competitors, and a list of executives, gleaned from the broader repository of the public web. I'll leave the conversation about privacy for another day.

It's likely that social-media-driven knowledge management will require much less of the "management" component. Historically we've spent far too much time cleaning up the data, validating, and categorizing it. In the future, more of our time will be spent analyzing all the new knowledge that is being created through our social interactions. Smart analysis can result in new insight, and that has powerful value for organizations.

No doubt this is an enormously complex space and social media magnifies the challenges. The time is right to evaluate your knowledge management strategy. New value creation starts now.

Photo: Office, by ianmunroe on Flickr


March 09 2011

One foot in college, one foot in business

screenshot.png In a recent interview, Joe Hellerstein, a professor in the UC Berkeley computer science department, talked about the disconnect between open source innovation and development. The problem, he said, doesn't lie with funding, but with engineering and professional development:

As I was coming up as a student, really interesting open source was coming out of universities. I'm thinking of things like the Ingres and Postgres database projects at Berkeley and the Mach operating system at Carnegie Mellon. These are things that today are parts of commercial products, but they began as blue-sky research. What has changed now is there's more professionally done open source. It's professional, but it's further disconnected from research.

A lot of the open source that's very important is really "me-too" software — so Linux was a clone of Unix, and Hadoop is a clone of Google's MapReduce. There's a bit of a disconnect between the innovation side, which the universities are good at, and the professionalism of open source that we expect today, which the companies are good at. The question is, can we put those back together through some sort of industrial-academic partnership? I'm hopeful that can be done, but we need to change our way of business.

Hellerstein pointed to the MADlib project being conducted between his group at Berkeley and the project sponsor EMC Greenplum as an example of a new partnership model that could close the gap between innovation and development.

Our sponsor would have been happy to donate money to my research funds, but I said, "You know, what I really need is engineering time."

The thing I cannot do on campus is run a professional engineering shop. There are no career incentives for people to be programmers at the university. But a company has processes and expertise, and they can hire really good people who have a career path in the company. Can we find an arrangement where those people are working on open source code in collaboration with the people at the university?

It's a different way of doing research funding. The company's contributions are not financial. The contributions are in engineering sweat. It's an interesting experiment, and it's going well so far.

In the interview Hellerstein also discusses MAD data analysis and where we are in the industrial revolution of data. The full interview is available in the following video:


January 19 2011

Four short links: 19 January 2010

  1. Implementing REST -- This is a place for exploring aspects of implementing applications using the REST architectural style. This may include statements about existing frameworks and libraries, general discussions about the nature of the style and how it may be expressed and/or encouraged via a programming framework, etc.
  2. When Teaching Restrains Discovery -- read about this research (short story: the more specific the skills taught, the less exploratory students were) and think about how we teach people to program, how we teach them the company culture, how we teach them to succeed.
  3. The Maker Generation in the Enterprise (JP Rangaswami) -- We have to get away from the idea that knowledge work is smooth and stable and uniform and assembly-line in structure and characteristic. Knowledge work is lumpy. Period. There will be peaks. And there will be troughs. The current thinking appears to go something like this: “If we have troughs it will look like we don’t have enough work to do, so we need to pretend to work. Let’s fill our days up in advance with things that don’t depend on market or customer stimulus, things we can plan well in advance. And let’s call these things meetings. Then we can look busy all the time.” Such thinking has produced some unworthwhile consequences.
  4. i.materialise 3D Printing in Titanium -- Titanium’s high heat resistance, high accuracy and unparalleled strength lets designers now make things that before now could only be made by the research and development departments of only the largest corporations in the world. By putting this technology in the public’s hands were democratizing manufacturing and giving you the opportunity to, design and order something this is exactly as you want it to be. (via Chris Anderson on Twitter)

December 10 2010

Four short links: 10 December 2010

  1. Let it Snow -- bookmarklet from David Flanagan that makes Javascript snowflakes fall. Awww. (via Mike Loukides)
  2. You Can Work on Great Technology at Startups -- There are more innovative database startups at various stages in their life than I can remember right now. So true--waiting for the inevitable amalgamation, thinning out, etc. (via Nat Friedman on Twitter)
  3. Dropbox for Teams -- an interesting package of features from a very innovative company. (via Hacker News)
  4. Cloud Computing Checklist -- Comparison and Analysis of the Terms and Conditions of Cloud Computing Services. What to look out for when signing a cloud contract. (via Rick Shera in email)

November 15 2010

Four short links: 15 November 2010

  1. Between the Bars -- snail-mail-to-blogs transcription service for prisoners, to make visible stories that would otherwise be missed. there is a religous program here called Kairo's in the program inmates are given letters and drawings made by small children not one in that program did not cry, after reading the words of incouragement from those kids. An unmissable reminder of the complexity of human stories, suffering, and situations, the posts range from the banal to the riveting. (via Benjamin Mako Hill)
  2. Kinect Opensource News -- a roundup of open source Kinect hacks. I like memo's gestural interface the best. Impressive stuff for just a few days' access to the open source drivers. (via Andy Baio)
  3. You Fix The Budget (NY Times) -- a simpler version of Budget Hero, which lets you choose policies and see their effect on the deficit. Unlike Budget Hero, the NYT app doesn't discuss non-deficit consequences of the actions (social consequences, ripple-on economic effects). Like Budget Hero, you can't add your own policies: you're forced to choose from the ones presented. Real life is more complex than this simulation, but even something this simple is powerful: by interacting with this, you understand the magnitude of (say) education vs healthcare, and you realize how much of the current debate is froth.
  4. Meet the New Enterprise Customer, a Lot like the Old Enterprise Customer -- Ben Horowitz nails the difficulty of selling to the enterprise, and drives a stake through the "they'll buy our service with their credit cards, like consumers do" myth. xcellent enterprise sales reps will guide a company through their own purchasing processes. Without an enterprise sales rep, many companies literally do not know how to buy new technology products. (via Mike Olson on Twitter)

September 08 2010

Better, faster, cheaper ... emergent

Gov 2.0 Summit, 2010Carl Malamud gave the opening keynote at the Government 2.0 Summit yesterday and greatly magnified my disappointment at having missed the event. If you've seen Carl speak, you know he is one person on the agenda that won't give a "presentation" or a "talk." Carl is an orator from a different era. He gives speeches. Rousing, moving, elevating speeches that turn our shared history into a kind of sermon; that inform us and inspire our better angels in equal measure. This is the kind of speech whose stirring coda lifts an audience to its feet and leaves them hitting replay in their heads -- not just to pilfer the richest sound bites for their tweet streams -- but to gather it all in.

I'm sorely disappointed that I missed it. I had to make do with reading it and watching it later on YouTube.

I absolutely agree with Carl that the failure in government IT is a failure to govern. We simply can't do without these things that we are trying so hard to build. Information technology is the infrastructure of governance, yet we fail to deliver it over and over again, and at great cost.

Carl's prescriptions for open data, open systems, and government copyright reform are also right on. However, after years of observing the system from the inside, I have come to believe they aren't enough.

While much can be said about the business practices of the "beltway bandits," they are not exactly the modern analogs to those turn-of-the-century pharma companies Carl describes. In the world before the FDA those exploiters waged an asymmetric and undefended war on American consumers. The beltway bandits are waging a war too, but one with more conventional symmetry -- where the government on the other side is heavily armed with a bureaucracy of its own.

We are the witnesses to an arms race of growing bureaucratic complexity, where absent market forces are inadequately replaced with the Federal Acquisition Rules and binders full of subordinate regulation. Where each additional regulation, intended to de-risk the process of building software, instead adds risk by delaying the delivery of anything actually useful.

The end result is hardly a lion preying on sheep. A better analogy might be Mothra and Godzilla locked in a death embrace. If your view is from the side of the government it is easy to place the blame with the beltway bandits. However, get inside their world a little bit and you may see them as a bit less rapacious and a lot more hogtied. When the customer is more interested in earned value reports and paperwork than working software, well, that's what they get. The beltway bandits are without a doubt a Frankenstein's monster, but it took a Dr. Frankenstein to build them.

Really though, I don't care one whit who is to blame. The real issue is that all of this complexity of product and process keeps out participants, innovation, and success. Open access systems can experiment, innovate, and deliver things while closed or limited access systems evolve to deliver rents -- in the economic sense.

In attempting to regulate a market that has inadequate competition, the government has inadvertently erected a bureaucracy that burdens market entry and facilitates the taking of uneconomic rents by those same beltway bandits they are trying to regulate. We should not be surprised that our creation, essentially a regulated quadopoly, is neither efficient or innovative.

Carl's focus on open data, open systems, and misuse of intellectual property is important and relevant, as they will all contribute to moving government IT back into open access territory. But we will have to deal with the market and incentive factors as well -- and they are probably bigger. In other words, demonizing the bandits without addressing the root cause -- the lock-in incentives inherent in a single-customer market -- will just lead to new ways to lock them in.

Shifting gears a bit, there is another area I would like to parse a bit further. The other major problem with government IT is the problem of enterprise IT in general, but at even larger scale. Some of this stuff is just really friggin' complicated. And I'm just not convinced we know how to build some of these systems, at this scale, inside this rate of technological change. Eliminate the added complexity of working with the government for a moment and ask yourself, do repeatable best practices even exist for specifying, planning, delivering and operating systems at the scale of the Navy Marine Corps Internet (400,000 nodes and thousands of applications)? Do they even exist for the example Carl used, the National Archives system?

Before you jump all over me, notice exactly how I worded the question: specify, plan, deliver, and operate. This is the classic systems engineering approach to developing IT. It is reductionist, assumes a reasonably stable technology ecosystem and problem space, and relies on complex planning and execution to deliver.

I am beginning to be of the opinion that this approach fundamentally won't work for systems at the scale that government often finds itself building, especially as they interconnect into ever larger and more complex wholes -- even if bureaucracy wasn't an impediment.

Consider this analogy: Our economy consists of countless businesses, each one to a large degree planned, hierarchical, and reductionist in outlook. And while they often acquire each other and grow very large, so far no single company has grown to swallow our entire economy. And if there is a lesson in the Soviet's five-year plans, none will. These planned entities have natural limits (based on current management science and systems) to how large they can become before they become unwieldy, and beyond that we rely on market mechanisms to coordinate them in an emergent way.

So, if large scale software systems are like that, what do you do if you want one that is bigger or more complex than our plans-oriented methodologies can deliver? Well, our government has been busily demonstrating that you don't do it by planning harder and de-risking more aggressively.

What Carl intrinsically grasps when he suggests open data and open systems, even if he doesn't say it outright, is that government IT must recognize that it is entering a different realm, one where we need to abandon planning beyond a certain scale and adopt an approach that intentionally facilitates emergence. Intentional emergence isn't planning faster or harder, it's about structuring incentives, policies, and eco-systems to encourage the complex to emerge from the simple. This may not mean a fully atomized simplicity, but may come to look like our economy where pockets of planning coexist in an emergent ecosystem.

Like on the web, it means that software/systems engineering will still exist in the nodes, but the coordinating signals among the whole must be economic. Systems engineering simply isn't equipped to operate at that scale and complexity. To cope we have to make a cognitive shift from planning, reductionism, and hierarchy to flattened networks and emergence, and put specifics such as open source, open systems and intellectual property policy into this broader framework.

I'm not exactly certain how to replace systems engineering as the basis for large system emergence, but I have some ideas. They draw inspiration from the transition of a planned to market economy. Reforms to government IT should look less like a more comprehensive CMMI and more like China's market reforms of the 1980s -- less about systems engineering and more about ecosystem engineering focused on incentives and policy.

As a starting point, we might ask the question, how might a NARA emerge in bits and pieces if a decentralized meta organization of government entities and citizens had budgetary and cultural incentives to contribute along a least resistant path that encouraged interoperability? And what should the FARS and other policy say to encourage rather than prohibit such an outcome?


September 06 2010

Four short links: 6 September 2010

  1. Akihabara (Github) -- open source (GPL2 and MIT dual-licensed) HTML5/Javascript engine for classic arcade games. (via chadfowler on Twitter)
  2. Eureka Streams -- open sourced Java app for enterprise Twitter-like activity: build a profile, join groups, post updates, subscribe to updates from individuals or groups. (via dlpeters on Twitter)
  3. Open Microbiome -- hoping to build open tools, standard samples, data, and metadata for analysis of the microbiome (all the microorganisms that live in, on, and with macroorganisms like us). Early days, but glad to see people are already thinking of building this research open from the ground up. And if you think sequencing the human genome gave us a lot of data we struggle to find patterns in, wait until you start including microorganisms: we have 10x as many bacteria in us as we have cells and the species variety is vast. (via phylogenomics on Twitter)
  4. Habits of Mathematical Minds -- fantastic list of skills and approaches that are hallmarks of many successful minds, not just in mathematics. (via ddmeyer on Twitter)

August 27 2010

Applying the lessons of Enterprise 2.0 to Gov 2.0

Last year, MIT professor Andrew McAfee published a landmark book on the business use and impact of social software platforms titled Enterprise 2.0: New Collaborative Tools for Your Organization’s Toughest Challenges. The book is a collection of McAfee's research since the spring of 2006 when he coined the phrase Enterprise 2.0. Shorthand for enterprise social software, Enterprise 2.0 is the strategic integration of Web 2.0 technologies into an organization's intranet, extranet, and business processes. Those technologies, including wikis, blogs, prediction markets, social networks, microblogging, and RSS, have in turn been adopted by government agencies, a phenomenon that falls under the mantle of Gov 2.0. As the use of such technology has grown, Congress is now considering the risks and rewards of Web 2.0 for federal agencies.

Gov 2.0 Summit, 2010The insights McAfee has gained from years of research into the use of social software by large organizations have broad application to understanding how and where technology will change government, and it's the basis for his talk, New Collaborative Tools for Government's Toughest Challenges, at the Gov 2.0 Summit in Washington D.C. I spoke in detail with Andrew, and anyone interested in understanding how social software is being used in large organizations will find the full half-hour audio interview of great interest.

Below are the questions I asked, and timestamps for the audio of where they start if readers want to jump ahead.


How is Enterprise 2.0 different from Web 2.0? And how does it apply to so-called Government 2.0? What do rules and regulations mean for the growth of social software? What does this mean for open government?
(Answer begins at 4:55)

Does automated filtering hold promise for government or the enterprise to prevent sensitive information from leaking? (Answer begins at 7:13)

Do reports of exfiltration of data from intelligence agencies mean collaborative software is a risk? (Answer begins at 8:35)

One of the examples in Enterprise 2.0 is Intellipedia. What lessons does its creation and evolution hold for the intelligence agencies? What about other government entities? (Answer begins at 9:52)

My interview with Sean Dennehy and Don Burke, the two CIA officers who have spearheaded the Intellipedia effort since its inception, is embedded below:

One of the most interesting parts of the book, for me, was the discussion of ideation platforms and collective intelligence. Government agencies are really running with the concept, including the upcoming launch of Innocentive shows another model. But does crowdsourcing really work? When, and under what conditions? What are the lessons from the private sector and academia in that regard? (Answer begins at 15:00)

You can read more about how game mechanics and crowdsourcing were combined to solve a complex challenge at Professor McAfee's blog.

What are the most common mistakes in implementations of social software, or ESSPs as you call them? Specifically, how do you set up effective crowdsourcing platforms? (Answer begins at 19:10)

What did the MIT "balloon team" that won the DARPA Network Challenge do right? (Answer begins at 21:09)

What challenges - and opportunities does the incoming millennial workforce hold for government and business with respect to IT? What does research show about how boomers, Gen Xers, and millennials interact, collaborate and work? Are there some myths to bust with respect to entrepreneurship and innovation? (Answer begins at 23:29)

What are the cultural issues around adoption of Enterprise 2.0 and Gov 2.0? (Answer begins at 27:07)

What does your new research on the strategic implementation of IT in large enterprises show to date? Why does government lag the private sector in this area, in the so-called "IT gap?" What could be done about it? (Answer begins at 30:03)

March 17 2010

Google's New Marketplace Has over a Thousand Apps

One week into its public launch, the Google Apps Marketplace has just under 1,500 (enterprise) apps. Combined with's app exchange (also with over a thousand apps), enterprises interested in moving to cloud apps have an increasing number of software tools to choose from.


Popular apps (measured in terms of # of installs) includes graphic design and office integration apps (aviary design suite and offisync), a collaboration and project management tool (manymoon), a free travel planner (tripit), a basic ERP app (, and a CRM application (Zoho CRM).

The typical supplier has about 2 offerings in the Google Apps Marketplace. Below are the suppliers with the most number of unique apps:


(†) Data for this post was through 2/16/2010.

Older posts are this way If this message doesn't go away, click anywhere on the page to continue loading posts.
Could not load more posts
Maybe Soup is currently being updated? I'll try again automatically in a few seconds...
Just a second, loading more posts...
You've reached the end.

Don't be the product, buy the product!