Newer posts are loading.
You are at the newest post.
Click here to check if anything new just came in.

January 02 2014

Four short links: 3 January 2014

  1. Commotion — open source mesh networks.
  2. WriteLaTeX — online collaborative LaTeX editor. No, really. This exists. In 2014.
  3. Distributed Systems — free book for download, goal is to bring together the ideas behind many of the more recent distributed systems – systems such as Amazon’s Dynamo, Google’s BigTable and MapReduce, Apache’s Hadoop etc.
  4. How Netflix Reverse-Engineered Hollywood (The Atlantic) — Using large teams of people specially trained to watch movies, Netflix deconstructed Hollywood. They paid people to watch films and tag them with all kinds of metadata. This process is so sophisticated and precise that taggers receive a 36-page training document that teaches them how to rate movies on their sexually suggestive content, goriness, romance levels, and even narrative elements like plot conclusiveness.

December 25 2013

Four short links: 25 December 2013

  1. Inside Netflix’s HR (HBR) — Which idea in the culture deck was the hardest sell with employees? “Adequate performance gets a generous severance package.” It’s a pretty blunt statement of our hunger for excellence. They talk about how those conversations play out in practice.
  2. Top Science Longreads for 2013 (Ed Yong) — for your Christmas reading.
  3. CocoaSPDY — open source library for SPDY (fast HTTP replacement, supported in Chrome) for iOS and OS X.
  4. The Internet of Things Will Replace the Web — invisible buttons loaded with anticipatory actions keyed from mined sensor data. And we’ll complain it’s slow and doesn’t know that I don’t like The Beatles before my coffee and who wrote this crap anyway?

February 14 2013

Four short links: 14 February 2013

  1. Welcome to the Malware-Industrial Complex (MIT) — brilliant phrase, sound analysis.
  2. Stupid Stupid xBoxThe hardcore/soft-tv transition and any lead they feel they have is simply not defensible by licensing other industries’ generic video or music content because those industries will gladly sell and license the same content to all other players. A single custom studio of 150 employees also can not generate enough content to defensibly satisfy 76M+ customers. Only with quality primary software content from thousands of independent developers can you defend the brand and the product. Only by making the user experience simple, quick, and seamless can you defend the brand and the product. Never seen a better put statement of why an ecosystem of indies is essential.
  3. Data Feedback Loops for TV (Salon) — Netflix’s data indicated that the same subscribers who loved the original BBC production also gobbled down movies starring Kevin Spacey or directed by David Fincher. Therefore, concluded Netflix executives, a remake of the BBC drama with Spacey and Fincher attached was a no-brainer, to the point that the company committed $100 million for two 13-episode seasons.
  4. wrka modern HTTP benchmarking tool capable of generating significant load when run on a single multi-core CPU. It combines a multithreaded design with scalable event notification systems such as epoll and kqueue.

February 05 2013

Four short links: 5 February 2013

  1. toolbar — tooltips in jQuery, cf hint.css which is tooltips in CSS.
  2. Security Engineering — 2ed now available online for free. (via /r/netsec)
  3. Economics of Netflix’s $100M New Show (The Atlantic) — Up until now, Netflix’s strategy has involved paying content makers and distributors, like Disney and Epix, for streaming rights to their movies and TV shows. It turns out, however, the company is overpaying on a lot of those deals. [...] [T]hese deals cost Netflix billions.
  4. Inceptiona FireWire physical memory manipulation and hacking tool exploiting IEEE 1394 SBP-2 DMA. The tool can unlock (any password accepted) and escalate privileges to Administrator/root on almost* any powered on machine you have physical access to. The tool can attack over FireWire, Thunderbolt, ExpressCard, PC Card and any other PCI/PCIe interfaces. (via BoingBoing)

April 12 2012

Strata Week: Add structured data, lose local flavor?

Here are a few of the data stories that caught my attention this week:

A possible downside to Wikidata

Wikidata data model diagram
Screenshot from the Wikidata Data Model page.

The Wikimedia Foundation — the good folks behind Wikipedia — recently proposed a Wikidata initiative. It's a new project that would build out a free secondary database to collect structured data that could provide support in turn for Wikipedia and other Wikimedia projects. According to the proposal:

"Many Wikipedia articles contain facts and connections to other articles that are not easily understood by a computer, like the population of a country or the place of birth of an actor. In Wikidata, you will be able to enter that information in a way that makes it processable by the computer. This means that the machine can provide it in different languages, use it to create overviews of such data, like lists or charts, or answer questions that can hardly be answered automatically today."

But in The Atlantic this week, Mark Graham, a research fellow at the Oxford Research Institute, takes a look at the proposal, calling these "changes that have worrying connotations for the diversity of knowledge in the world's sixth most popular website." Graham points to the different language editions of Wikipedia, noting that the encyclopedic knowledge contained therein is always highly diverse. "Not only does each language edition include different sets of topics, but when several editions do cover the same topic, they often put their own, unique spin on the topic. In particular, the ability of each language edition to exist independently has allowed each language community to contextualize knowledge for its audience."

Graham fears that emphasizing a standardized, machine-readable, semantic-oriented Wikipedia will lose this local flavor:

"The reason that Wikidata marks such a significant moment in Wikipedia's history is the fact that it eliminates some of the scope for culturally contingent representations of places, processes, people, and events. However, even more concerning is that fact that this sort of congealed and structured knowledge is unlikely to reflect the opinions and beliefs of traditionally marginalized groups."

His arguments raise questions about the perceived universality of data, when in fact what we might find instead is terribly nuanced and localized, particularly when that data is contributed by humans who are distributed globally.

The intricacies of Netflix personalization

Netflix suggestion buttonNetflix's recommendation engine is often cited as a premier example of how user data can be mined and analyzed to build a better service. This week, Netflix's Xavier Amatriain and Justin Basilico penned a blog post offering insights into the challenges that the company — and thanks to the Netflix Prize, the data mining and machine learning communities — have faced in improving the accuracy of movie recommendation engines.

The Netflix post raises some interesting questions about how the means of content delivery have changed recommendations. In other words, when Netflix refocused on its streaming product, viewing interests changed (and not just because the selection changed). The same holds true for the multitude of ways in which we can now watch movies via Netflix (there are hundreds of different device options for accessing and viewing content from the service).

Amatriain and Basilico write:

"Now it is clear that the Netflix Prize objective, accurate prediction of a movie's rating, is just one of the many components of an effective recommendation system that optimizes our members' enjoyment. We also need to take into account factors such as context, title popularity, interest, evidence, novelty, diversity, and freshness. Supporting all the different contexts in which we want to make recommendations requires a range of algorithms that are tuned to the needs of those contexts."

Fluent Conference: JavaScript & Beyond — Explore the changing worlds of JavaScript & HTML5 at the O'Reilly Fluent Conference (May 29 - 31 in San Francisco, Calif.).

Save 20% on registration with the code RADAR20

Got data news?

Feel free to email me.


Reposted byRK RK

March 01 2012

Commerce Weekly: Small banks lagging in mobile

Here are some of the commerce stories that caught my attention this week.

Smaller banks lagging in mobile channel

Smaller financial institutions, which depend on a higher level of customer service to compete with the giants, are falling behind in the increasingly important mobile channel, according to a report by Javelin Strategy & Research. Javelin says about 37% of customers at big banks use mobile banking, compared with only 21% at regional and community banks and only 15% at credit unions. Javelin's report suggests two reasons for this. First, community bank customers tend to be older, less well off, and less tech-savvy than customers at big banks. Second, big banks can invest more in online and mobile development and marketing, resulting in a better banking experience through those channels. (That's certainly been my experience: my attempts to switch to a smaller bank were thwarted by a virtually unusable online banking system, which drove me back into the warm and fuzzy interface of a cold financial giant.)

Some smaller financial institutions say they benefitted from the anti-big-bank sentiment of the past year, epitomized by Bank Transfer Day on Nov. 5, 2011. Redwood Credit Union in Santa Rosa, for example, says its new membership was three times the normal rate last fall. But to keep that momentum going, Javelin suggests, financial institutions like Redwood will need to funnel some of their new income into development of these channels.

The report also found that mobile usage is beginning to surpass non-mobile online usage, even if those customers tap their accounts through a mobile browser. Most customers reach banks' mobile sites through a browser on their phone. However, at the largest banks, which tend to offer a "triple play," more customers use apps and SMS text instead of the browser.

X.commerce harnesses the technologies of eBay, PayPal and Magento to create the first end-to-end multi-channel commerce technology platform. Our vision is to enable merchants of every size, service providers and developers to thrive in a marketplace where in-store, online, mobile and social selling are all mission critical to business success. Learn more at

How Netflix improves its recommendations

One of the interesting presentations at O'Reilly's Strata conference this week was about how Netflix looks at its data to present recommendations of other shows members might like. Netflix streams 30 million shows a day. It has 5 billion ratings on those shows and collects another 4 million every day. Data scientist Xavier Amatriain discussed how Netflix uses the data from those ratings and other, more implicit data (including what people watch, which listings they mouse over to read, whether or not they finish programs) to offer recommendations that members will like enough to keep their accounts active, month after month.

Netflix gained a lot of attention a few years back with a broad open innovation initiative: it offered $1 million to anyone who could improve the Netflix recommendation engine by at least 10%. Amatriain said two teams tied for the prize with plans that improved the probability that Netflix could recommend shows that members would like based on their previous activities (though, he added, the cost of integrating those new recommendation engines into Netflix' system may have exceeded their value). Even so, since 75% of shows watched on Netflix's streaming service are based on recommendations, it's more important than ever to offer something that will draw viewers' interest.

Netflix queue example

The clues from all this data allow Netflix to present an array of recommendations to its members. First, there's a row of "top ten" most likely shows. Of course, as Amatriain pointed out, these recommendations are based on viewing history and clues of the entire membership household, not just one viewer. For example, when I log on, along with the thrillers and comedies that Netflix recommends to me, there's a fair amount of "Pretty Little Liars" and other teen dramas that my daughters might like. I used to wonder if this bizarre mix confused Netflix, but Amatriain's talk has reassured me that the company understands what's going on. Then, at a finer-grained level, there are "hyper genres" that Netflix can offer based on your track record: not just Kids Shows, but Goofy Kids Shows; not just Family Movies but Feel-good Father-Daughter Movies. Slicing the offerings narrowly improves the chances of a hit, and it's no accident that the single most likely recommendation is the first one in each row.

Of course, the main complaint Netflix receives (other than its new price structure, I would imagine) is, "why don't you have the show I want to watch?" Amatriain said the company also looks at implicit data to decide what new content to license. So when you search for a show that Netflix doesn't offer for streaming, it gets noted. I guess if you really want it to show up, keep searching for it.

Opera enters the payment fray, PayPal and Home Depot go nationwide

Mobile World Congress, the humongous European conference on all things mobile, is happening this week and everyone loosely connected to mobile payments seemed to time an announcement around it. Here are some of the more interesting announcements that have come down the PR wire from Barcelona:

  • Opera, whose Opera Mini browser has more than 160 million downloads, launched the Opera Payment Exchange (OPX). Opera says it wants to "democratize" the payment space by building a payment platform that works on more platforms and devices than Android and iOS smartphones. It says the OPX platform provides APIs that developers can use to integrate payment systems with the Opera Mini mobile browser.
  • PayPal and Home Depot said they would roll out nationwide the payment program they have been piloting in a handful of Bay Area stores over the past six weeks. The program is a significant step for PayPal, bringing its payment system offline and into the physical retail world. Customers can buy hardware and other stuff on their PayPal account, with a PayPal card or with a mobile number and PIN — no NFC required.
  • Isis, the mobile payments joint venture between AT&T, T-Mobile, and Verizon Wireless, announced more partners in its effort to build a payments ecosystem. Customers of Chase, CapitalOne, and BarclayCard will be able to load their payment information into Isis-compatible phones when they're ready. Isis secured deals with the top four credit card companies (or "payment networks" to use the parlance) last July; now it's making agreements with the banks ("issuers"). Isis is planning two pilots in 2012, in Austin and Salt Lake City, though it's not clear what phones the technology will be in by then.

Tip us off

News tips and suggestions are always welcome, so please send them along.

If you're interested in learning more about the commerce space, check out DevZone on, a collaboration between O'Reilly and X.commerce.

Bank photo: Old Bank in Sunbury Village by Maxwell Hamilton, on Flickr


December 21 2011

December 09 2011

Top Stories: December 5-9, 2011

Here's a look at the top stories published across O'Reilly sites this week.

The end of social
Mike Loukides: "If you want to tell me what you listen to, I care. But if sharing is nothing more than a social application feed that's constantly updated without your volition, then it's just another form of spam."

Why cloud services are a tempting target for attackers
Jeffrey Carr says before organizations embrace the efficiencies and cost savings of cloud services, they should also closely consider the security repercussions and liabilities attached to the cloud.

White House to open source as open government data platform
The new " in a box" could empower countries to build their own platforms. With this step forward, the prospects are brighter for stimulating economic activity, civic utility and accountability under a global open-government partnership.

Stickers as sensors
Put a GreenGoose sticker on an object, and just like that, you'll have an Internet-connected sensor. In this interview, GreenGoose founder Brian Krejcarek discusses stickers as sensors and the data that can be gathered from everyday activities.

What publishers can learn from Netflix's problems writer Tim Carmody examines the recent missteps of Netflix and takes a broad look at how technology shapes the reading experience.

Tools of Change for Publishing, being held February 13-15 in New York, is where the publishing and tech industries converge. Register to attend TOC 2012.

December 08 2011

What publishers can learn from Netflix's problems

In a wide-ranging interview, Tim Carmody (@tcarmody), a writer for, Snarkmarket, The Idler, et al., looked at the lessons publishers and others can take from Netflix' recent troubles, and he examined the ways in which technology shapes the reading experience. (Carmody will be a keynote speaker at TOC 2012.)

Specific highlights from the interview (below) include:

  • Inevitability isn't inevitable, just ask Netflix — For a while Netflix's continued ascendance appeared "inevitable." That's a fantasy, said Carmody, and the best lesson publishers can take is that "anything that looks inevitable now might not look so inevitable in six months." Carmody said it's important to disrupt your business — something Netflix has done well — but you must tread lightly because consumers are fickle. [Discussed at the 3:50 mark.]
  • Reading experiences are not confined to a specific form — If you spend your days crunching numbers on a screen, you're likely "primed" to make a database of friends on Facebook. Play Angry Birds on your iPad? Carmody said you might gravitate toward game-like publications. Publishers need to understand that the context of all content influences what we read and how we read it. "We're always making generalizations based on the broadest set of technologies that we're reading," Carmody said. "It's never just within the medium or within the format. It's everything. The way we look at street signs changes the way we read books, the way we read the newspaper changes the way we read magazines. All of these things are always operative." [Discussed at 1:22.]
  • Kickstarter's tier model can work for publishing — Bundling content and offering levels or tiers of content (if you buy tier three, you also get tiers one and two) is a powerful retail model that could work well in book publishing. [Discussed at 6:22.]

You can view the entire interview in the following video.

TOC NY 2012 — O'Reilly's TOC Conference, being held Feb. 13-15, 2012, in New York City, is where the publishing and tech industries converge. Practitioners and executives from both camps will share what they've learned and join together to navigate publishing's ongoing transformation.

Register to attend TOC 2012


  • Ebooks and the threat from "internal constituencies"
  • The problem with Amazon's Kindle Owners' Lending Library

  • December 02 2011

    Four short links: 2 December 2011

    1. Challenges in Teaching Biology -- everything that Alison says about teaching biology is true of teaching computer science. Read, learn, evolve.
    2. First Open Source Netflix Projects Released -- Curator makes Apache Zookeeper easier to use. (via Ian Kallen)
    3. LLVM3 Released -- these are key tools for reliable development of fast systems. I think of it as JVM without the bloat, though undoubtedly that's unfair to both Java and LLVM. (via Hacker News)
    4. Scribe -- Zooniverse tool for crowdsourcing transcriptions. (via Tim Sherratt)

    August 01 2011

    Four short links: 1 August 2011

    1. The Flashed Face Effect Video -- your brain is not perfect, and it reduces faces to key details. When they flash by in the periphery of your vision, you perceive them as gross and freakish. I like to start the week by reminding myself how fallible I am. Good preparation for the rest of the week... (via BERG London)
    2. The Newsonomics of Netflix and the Digital Shift -- Netflix changed prices, tilting people toward digital and away from physical. This post argues that the same will happen in newspapers. Imagine 2020, and the always-out-there-question: Will we still have print newspapers? Well, maybe, but imagine how much they’ll cost — $3 for a local daily? — and consumers will compare that to the “cheap” tablet pricing, and decide, just as they doing now are with Netflix, which product to take and which to let go. The print world ends not with a bang, but with price increase after price increase. (via Tim O'Reilly)
    3. Phonegap -- just shipped 1.0 of an HTML5 app platform that allows you to author native applications with web technologies and get access to APIs and app stores.
    4. UnQL -- query language for document store databases, from the creators of CouchDB and SQLite. (via Francisco Reyes)

    July 29 2011

    Visualization of the Week: A map of regional movie tastes

    Netflix knows a lot about our movie tastes, as the company can track what we rent via DVDs or watch via streaming video, how we rent and how we rate videos. Some of that data about our viewing and renting habits is open to the public in the form of the "Local Favorites" section. Here you can see what's popular in and around your city.

    Slacktory has taken this information to create a visualization: "The United States of Local Favorites."

    Netflix Favorites mapped out regionally

    Some of the regional favorites aren't too surprising: Southern California seems to like "L.A. Story." Some of the favorites are quite amusing: "Beverly Hill Chihuahua" is apparently much loved in Mississippi.

    Other than pointing to the source of the data, Slacktory doesn't offer any insight into how the visualization was built. Are our movie tastes really that geographically diverse?

    Found a great visualization? Tell us about it

    This post is part of an ongoing series exploring visualizations. We're always looking for leads, so please drop a line if there's a visualization you think we should know about.

    Strata Conference New York 2011, being held Sept. 22-23, covers the latest and best tools and technologies for data science -- from gathering, cleaning, analyzing, and storing data to communicating data intelligence effectively.

    Save 20% on registration with the code STN11RAD


    July 22 2011

    Open question: Which streaming services do you use?

    Netflix Watch InstantlyIn a New York Times article on Netflix's new plans and pricing, a customer quote toward the end of the piece jumped out at me:

    "Netflix’s streaming video selection is horrible," [Shelia] Haupt said. "What I can get on demand from my cable company is so much better."

    I found this notable for a couple reasons. First — and less important — I disagree about Netflix's selection. The company's streaming catalog was horrible when the service first launched in 2007, but it's improved to the point where I'm often surprised at what's available. More often than not I can find something interesting to watch. (And while I have an unabashed appreciation of low-grade pop culture, I also like "quality" content.)

    Second — and relevant to this open question — Haupt's comment about using her cable company's on-demand selection hints at a usage pattern I'd like to explore. Specifically, are folks putting their time and money behind particular streaming services? Or are they sampling from a streaming buffet? A Hulu show here, a Netflix movie there, and perhaps an acoustic Pandora station for Sunday mornings?

    So with that in mind, here are the questions I'd like to dig into:

    • Which streaming services do you use most often?
    • Do you use services for specific things (Hulu for TV, Netflix for movies, etc.)?
    • Do you pay for any streaming services?
    • Would you prefer to pay for multiple services from different providers or one super-service from a single provider?

    Please weigh in through the comments.

    Web 2.0 Expo New York 2011, being held Oct. 10-13, showcases the latest Web 2.0 business models, development tools and design strategies for the builders of the next-generation web.

    Save 20% on registration with code WEBNY11RAD


    June 30 2011

    How Netflix handles all those devices

    Netflix's shift to streaming delivery has made quite an impression on Internet traffic. According to Sandvine's latest report, Netflix now claims almost 30% of peak downstream traffic in North America.

    That traffic occurs, in no small part, because Netflix can run on so many devices — PCs, tablets, gaming consoles, phones, and so on. In the following interview, Netflix's Matt McCarthy (@dnl2ba) shares a few lessons from building across those varied platforms. McCarthy and co-presenter Kimberly Trott will expand on many of these same topics during their session at next month's OSCON.

    What are some of the user interface (UI) challenges that Netflix faces when working across devices?

    Matt McCarthyMatt McCarthy: Scaling UI performance to run well on a low-cost Blu-ray player and still take advantage of a PlayStation 3's muscle has required consulting WebKit and hardware experts, rewriting components that looked perfectly good a week before, and patiently tuning cache sizes and animations. There's no silver bullet.

    Since we've standardized on WebKit, we don't have to support multiple disparate rendering engines, DOM API variants, or script engines. However, there are lots of complex rendering scenarios that are difficult to anticipate and test, especially now that we're starting to take advantage of WebKit accelerated compositing. There are WebKit test suites, but none that are both comprehensive and well documented, so we're working on our own test suite that we can use to validate partners' ports of our platform.

    OSCON JavaScript and HTML5 Track — Discover the new power offered by HTML5, and understand JavaScript's imminent colonization of server-side technology.

    Save 20% on registration with the code OS11RAD

    How do the platform lessons Netflix has learned apply to other developers?

    Matt McCarthy: The challenges we face may be familiar to many large-scale AJAX application developers. In addition, mobile developers need to make similar trade-offs between memory usage and performance, other sophisticated user interfaces need to handle UI state, and most large code bases can benefit from good abstraction, encapsulation, and reuse.

    The urgency and difficulty of solving those challenges may differ for different applications, of course. If your application is very simple, it would be silly for you to use the level of abstraction we've implemented to support A/B testing in Netflix device UIs. But if you're innovating heavily on user experience, your performance isn't always what you'd like, and your UI is an endless font of race conditions and application state bugs, then maybe you'd like to learn about our successes and mistakes.

    There were reports last year that some Netflix PS3 users were seeing several different UIs. What are the benefits and challenges with this kind of A/B testing?

    Matt McCarthy: Netflix is a subscriber service, so ultimately what we care about is customer retention. But retention, by definition, takes a long time to measure. We use proxy metrics that correlate well with retention. Some of our most closely watched metrics have to do with how many hours of content customers stream per month. Personally, I find it gratifying to have business interests that are aligned closely with our customers' interests.

    The challenges grow as the A/B test matrix grows, since the number of test cell combinations scales geometrically with the number of tests. Our quality assurance team has been working on automated tests to detect regressions so a fancy new feature doesn't inadvertently break another feature that launched last month. Our engineers adhere to a number of best practices, e.g. defining, documenting, and adhering to interfaces so we don't find nasty surprises when we replace a UI component in a test cell.

    A/B testing user interfaces obviously takes a lot more effort than developing our "best bet" UI and calling it a day, but it's been well worth the cost. We've already been surprised a few times by TV UI test results, and it's changed the direction we've taken in new UI tests for both TV devices and our website. Every surprise validates our approach, and it shows us a new way to delight and retain more customers.

    This interview was edited and condensed.


    May 11 2011

    How the cloud helps Netflix

    NetflixAs Internet-based companies outgrow their data centers, they're looking at larger cloud-based infrastructures such as those offered by Microsoft, Google, and Amazon. Last year, Netflix made such a transition when it moved some of its services into Amazon's cloud.

    In a recent interview, Adrian Cockcroft, (@adrianco) cloud architect at Netflix and a speaker at Velocity 2011, talked about what it took to move Netflix to the cloud, why they chose Amazon's platform, and how the company is accommodating the increasing demands of streaming.

    Our interview follows.

    Why did Netflix choose to migrate to Amazon's cloud?

    AdrianCockcroft.jpg Adrian Cockcroft: We couldn't build our own data centers fast enough to track our growth rate and global roll out, so we leveraged Amazon's ability to build and run large-scale infrastructure. In doing that, we got extreme agility. For example, when we decided to test world-wide deployment of services, our developers were immediately able to launch large-scale deployments and tests on another continent, with no planning delay.

    What architectural changes were required to move from a conventional data center to a cloud environment?

    Adrian Cockcroft: We took the opportunity to re-work our apps to a fine-grain SOA-style architecture, where each developer pushes his own auto-scaled service. We made a clean separation of stateful services and stateless business logic, and designed with the assumption that large numbers of systems would fail and that we should keep running without intervention. This was largely about paying down our technical debt and building a scalable web-based product using current best practices.

    Velocity 2011, being held June 14-16 in Santa Clara, Calif., offers the skills and tools you need to master web performance and operations.

    Save 20% on registration with the code VEL11RAD

    What issues are you facing as streaming demand increases?

    Adrian Cockcroft: We work with all three "terabit-scale" content delivery networks — Level 3, Limelight, and Akamai. They stream our movies to the end customer, and if there is a problem with one of them, traffic automatically switches to another. We don't see any limits on how much traffic we can stream. We aren't trying to feed everyone in the world from a single central point — it's widely distributed.

    Netflix doesn't ask customers to change much on their side (browsers, speeds, etc.) — how do you achieve this level of inclusivity, and do you see it continuing?

    Adrian Cockcroft: We have very wide ranging support for streaming devices and expect this to continue. We are working on the HTML5 video tag standards, which may eventually allow DRM-protected playback of movies on any browser with no plugin. We currently depend on Silverlight for Windows and Mac OS, and we don't have a supported DRM mechanism for playback on Linux browsers.

    For hardware devices, we work with the chip manufacturers to build Netflix-ready versions of the chipsets used to build TV sets and Blu-ray players. That way we are included in almost all new Internet-connected TV devices.

    This interview was edited and condensed.


    December 16 2010

    Strata Week: Shop 'til you drop

    Need a break from the holiday madness? You're not alone. Check out these items of interest from the land of data and see why even the big consumers face tough choices.

    Does this place accept returns?

    On Monday, Stack Overflow announced that they have moved the Stack Exchange Data Explorer (SEDE) off of the Windows Azure platform and onto in-house hardware.


    SEDE is an open source, web-based tool for querying the monthly data dump of Creative Commons data from its four main Q&A sites (Stack Overflow, Server Fault, Super User, and Meta) as well as other sites in the Stack Exchange family. The primary reason given (within a polite write-up by Jeff Atwood and SEDE lead Sam Saffron), was the desire to have fine-tuned control over the platform.

    When you are using a [Platform-as-a-Service] you are giving up a lot of control to the service provider. The service provider chooses which applications you can run and imposes a series of restrictions. ... It was disorienting moving to a platform where we had no idea what kind of hardware was running our app. Giving up control of basic tools and processes we use to tune our environment was extremely painful.

    While the support that comes with Platform-as-a-Service was acknowledged, it seems that the ability to better automate, adjust, and perpetuate processes and systems with more fine-grained control won out as a bigger convenience.

    Where did you get that lovely platform?

    Strata 2011Of course, one company's headache is another's dream. Netflix, a company known for playing with big data and crowdsourcing solutions "before it was cool," posted on Tuesday the four reasons they've chosen to use Amazon Web Services (AWS) as their platform and have moved onto it over the last year.

    Laudably, the company states that it viewed its tremendous recent growth (in terms of both members and streaming devices) as a license to question everything in the necessary process of re-architecting. Instead of building out their own data centers, etc., they decided to answer that set of questions by paying someone else to worry about it.

    Also to their credit, Netflix has enough self-awareness to know what they are and aren't good at. Building top-notch recommendation systems and providing entertainment? You betcha. Predicting customer growth and device engagement? Not so much.

    How many subscribers would you guess used our Wii application the week it launched? How many would you guess will use it next month? We have to ask ourselves these questions for each device we launch because our software systems need to scale to the size of the business, every time.

    Self-awareness is in fact the primary lesson in both Netflix's and Stack Exchange's platform decisions. If you feel your attention is better spent elsewhere, write a check. If you've got the time and expertise to hone your hardware, roll your own.

    [Of course, Netflix doesn't go for the pre-packaged solutions every time. They also posted recently about why they love open source software, and listed among the projects they make use of and contribute back to: Hadoop, Hive, HBase, Honu, Ant, Tomcat, Hudson, Ivy, Cassandra, etc.]

    With what shall we shop?

    The New York Times this week released a cool group of interactive maps based on data collected in the Census Bureau's American Community Survey (ACS) from 2005 to 2009. Data is compared against the 2000 census to uncover rates of change.

    [While similar to the census, the ACS is conducted every year instead of every 10 years. The ACS includes only a sampling of addresses instead of a comprehensive inventory. It covers much of the same ground on population (age, race, disability status, family relationships), but it also asks for information that is used to help make funding distribution decisions about community services and institutions.]

    The Times maps explore education levels; rent, mortgage rates, and home values; household income; and racial distribution. Viewers can select among 22 maps in these four categories, and then pan and zoom to view national, state, or local trends down to the level of individual census tracts.

    Above is the national view of the map that looks at change in median household income. The ACS website itself provides some maps displaying the survey numbers from the 2000 census and the 2005-2009 survey, as well as a listing of data tables.

    The Times map shows the uneven way in which these numbers have gone up or down in various parts of the country, with some surprising results that are worth exploring. Note that the blue regions are places where income has dropped, and the yellow regions are places where it has increased. (No wonder a lot of us are getting creative with holiday shopping.)

    If this kind of research floats your boat, check out Social Explorer, the mapping tool used to create the New York Times maps.

    Even markets like to buy things

    The emerging landscape of custom data markets is already shifting as Infochimps recently announced the acquisition of Data Marketplace, a start-up incubated at Y Combinator.

    While Stewart Brand may be right in thinking information wants to be free, there's also enormous value to be added by aggregating, structuring, and packaging data, as well as in matching up buyers with sellers. That's the main service Data Marketplace aims to provide, particularly in the field of financial data.

    At Infochimps, information is offered a la carte, and many of the site's datasets are offered for free. These include sets as diverse as "Word List - 100,000+ official crossword words (Excel readable)", "Measuring Worth: Interest Rates - US & UK 1790-2000", and "Retrosheet: Game Logs (play-by-play) for Major League Baseball Games." Data Marketplace is a bit different, in that it allows users to enter requests for data (with a deadline and budget, if desired) and then matches up would-be buyers with data providers.

    Infochimps has said that Data Marketplace, which is less than a year old, will continue to operate as a standalone site, although its founders Steve DeWald and Matt Hodan will depart for new projects.

    If you're interested in the burgeoning business of aggregated datasets, be sure to check out the Data Marketplaces panel I'll be moderating at Strata in February.

    Not yet signed up for Strata? Register now and save 30% with the code STR11RAD.

    November 05 2009

    Three Paradoxes of the Internet Age - Part Two

    Individual perception of increased choice can occur while the overall choice pool is getting smaller

    This gem from Whimsley makes the point - with extensive statistical modeling supporting the argument - that our algorithm-obsessed, long tail merchants are actually depleting the overall choice pool despite the fact that as individuals we may be experiencing a sense of more choice through recommendations engines...

    Online merchants such as Amazon, iTunes and Netflix may stock more items than your local book, CD, or video store, but they are no friend to "niche culture". Internet sharing mechanisms such as YouTube and Google PageRank, which distil the clicks of millions of people into recommendations, may also be promoting an online monoculture. Even word of mouth recommendations such as blogging links may exert a homogenizing pressure and lead to an online culture that is less democratic and less equitable, than offline culture.

    In short, the long tail has gangrene at its extremity - the niche. More disarming is the conclusion that it isn't just the output of our recommendation algorithms that is leading to what the author calls "monopoly populism"and the end of niche culture:
    "The recommender "system" could be anything that tends to build on its own popularity, including word of mouth...Our online experiences are heavily correlated, and we end up with monopoly populism...A "niche", remember, is a protected and hidden recess or cranny, not just another row in a big database. Ecological niches need protection from the surrounding harsh environment if they are to thrive. Simply putting lots of music into a single online iTunes store is no recipe for a broad, niche-friendly culture.

    The network effects that so characterize Internet services are a positive feedback loop where the winners take all (or most). The issue isn't what they bring to the table, it is what they are leaving behind.

    here is a link to yesterday's post: More access to information doesn’t bring people together, often it isolates us.

    Tomorrow: The myth of personal empowerment takes root amidst a massive loss of personal control.

    Reposted byjagas jagas
    Older posts are this way If this message doesn't go away, click anywhere on the page to continue loading posts.
    Could not load more posts
    Maybe Soup is currently being updated? I'll try again automatically in a few seconds...
    Just a second, loading more posts...
    You've reached the end.

    Don't be the product, buy the product!