Newer posts are loading.
You are at the newest post.
Click here to check if anything new just came in.

August 08 2011

Open minds and open source community

The last week of July was OSCON. Now that I’ve had a week off to think about all the amazing conversations I overheard and took part in, I want to share some of the ideas that I was hearing.

I took Steve Yegge’s call to “work on important things” to heart in last week in Portland. I sought out hard conversations with the collection of open source users, developers and luminaries that attended OSCON. I started most of my hallway track sessions asking and thinking very broadly about our tag line, "from disruption to default."

Growing the community

If open source is now the default, if the audience is now much broader, then are we talking enough about the primary audience for open source tools, projects and products? Now our constituency is not all developer peers. We must challenge the idea that if someone really wants to use a piece of software, he or she will be willing to slog through half-written documentation, the actual code base and an unkind user interface.

Gabe Zichermann made a fantastic point in his keynote about the introduction of new players to a game: In the first screens of a game, there is no way to do the wrong thing. In fact, there is only one thing to do, which is the first step in learning how to interface with that product. Likewise, lowering the barrier to entry is paramount to continuing to promote open source as a culture and a global good.

There was a time in open source when, if you used a product, you also were a contributor to the code. Now, we are experiencing a redefinition of contributor and community member. Users who make good bug reports are fantastic contributors, though they may not code. Designers who rework the user experience for broader appeal are also important contributors. An entire culture based on finding people to relate to as outcasts is now learning to deal with popularity. The label "geek" is no longer a pejorative, but instead is becoming a badge of honor.

These adolescent growing pains are causing tension within the greater open source community. Change is happening: ‘Open’ has gone from elitist anti-crowd to global buzzword, experiencing dramatic growth and shifts in business models. Hackers aren’t doing this simply for love but often looking for corporate sponsorship. There are more subtle changes, too, like the redefinition of ‘community’ to include not only developers but also end users, corporate sponsors, people of color and women, designers and UX engineers.

Boy, it's hard to be cool. The early projects which had success in broadening the reach of a project or in securing corporate sponsorship, funding or a business model were decried as sellouts. Think RedHat, circa 2002. Who would ever pay for Linux support?

In more recent times, Jono Bacon has navigated that line well. He is not only the community face of Ubuntu and Canonical, but an actual rock star. He gave a keynote at OSCON last week about the future of communities. Then, in an incredibly authentic blog post, Jono reflected on his presentation and noted what he learned. Transparency and openness to alternate perspectives are what allow him to rally an eclectic community of developers, users, designers, and the curious who in turn promote and host events to add a three dimensional component for the community around Ubuntu.

Android Open, being held October 9-11 in San Francisco, is a big-tent meeting ground for app and game developers, carriers, chip manufacturers, content creators, OEMs, researchers, entrepreneurs, VCs, and business leaders.

Save 20% on registration with the code AN11RAD

Using data and open source better

In some quiet time at OSCON, Dave Eaves and Edd Dumbill had a fantastic conversation about making use of data to better understand the ebb and flow of an open source community’s engagement.

Dave has been working with Mozilla to seek out and understand what is quantitatively happening in the community to inform constant improvement. Ask “why are bugs in this section of the code taking twice as long to be reviewed?” Ask “who has contributed consistently over the last 18 months, but not in the last 30 days?” Take the information, both qualitative and quantitative, and then use it to continue to improve our communities.

In another example, when OpenStack was announced on the OSCON keynote stage last year, it gave rise to conversations with the thesis that there are "real open source" projects (and, presumably, "faux open source" projects). Under scrutiny were corporate sponsorships and business models which have funded a proliferation of newer open source projects like Ubuntu or Hadoop. Many argue that companies still don't know how to take a product and "open source it."

The fact that “open source” is a verb suggests we have made it, but the community's limited engagement with OpenStack means there's still a long way to go. At Nebula’s launch keynote, Chris Kemp asked who had worked with OpenStack; the response was anemic. There is hope. OpenStack has increased from a dozen contributors (primarily funded by Rackspace) to more than 250 in the current release and is seeking to engage more independent developers as well. (There are currently 1200 developers in the development tree.)

What does open mean?

Some arguments for open source are about transparency or safety. The more eyes see code, the more refined it becomes, and the more security exploits are found before impacting an end user. In her keynote at OSCON, Karen Sandler answered how open source will benefit consumers of medical devices. A corporate sponsor, however will be considering the talent pool of theoretically unlimited volunteer workers. The dogmatic among us want "free-libre" for the independence from corporate overlords.

I believe open is a mindset: accepting and respecting the views of others and encouraging more perspectives in our communities. Not every company will have goals or methods that align with the open source philosophy. There are many companies which will. The first step to hacking is to frame the problem and understand the biases. We have the opportunity here to share experience and find where we can work together. Instead of conversations about if a corporate sponsored project is really open, why not welcome the corporate sponsors to the table and work to find where our interests align. Nurture those ideas, then start discussing the harder edge cases.

We lack common motivations for a set of complicated goals loosely collected and anointed “open source." When we address this underlying problem, the other symptoms we’ve been treating will ease. There is a culture behind the idea of open source. To move that vision forward, there will have to be more public successes.

Jim Zemlin suggested in his keynote that we need to brand the vision behind open source, much like "Made in the USA" or "Fair Trade." This means working with and embracing our old rivals and finding where our interests align. In the least marketing-y keynote I've ever seen from Microsoft, Gianugo Rabellino spoke of a place for both open and closed source products and advocates embracing their interoperability. While I don't see Microsoft being a corporate sponsor for Linux, they are kernel contributors. Alignment of interests give rise to opportunities where everyone benefits. We will see more in the future.

Have we succeeded in disrupting? Yes. Is open source the default? In some places. How do we do it again? We embrace the change we birthed and work to evolve and grow this adolescent into a robust, healthy and curious philosophy.

July 29 2011

Open source alchemy: Health care and Alembic at OSCON

At last year's OSCON I spoke with David Riley, Brian Behlendorf and Arien Malec about how open source solutions can help improve our health care system. A lot has happened in the past year, both with the Direct Project and the efforts to build a Nationwide Health Information Network.

This year David Riley gave an update on Aurion (developed from the CONNECT codebase), which is a major project of the Alembic Foundation. Alembic was founded by David, formerly the CONNECT initiative lead for the Federal Health Architecture (FHA), and Brian Behlendorf, the chief technology officer for theWorld Economic Forum. David's presentation focused on the Aurion Project's relationship to CONNECT, and it gave us a sense of where the Project is heading in the future. I spoke with Brian and David during OSCON, and in this first clip they discuss the mission and goals of the Alembic Foundation:

In this next clip they speak about how their efforts are related to the broader work on health information exchange, and specifically how Aurion will support the Direct Project:

In this final clip, brought on by an earlier question from Fred Trotter, they explore possibilities for Alembic to work on open source electronic health records like the Veterans Health Information Systems and Technology Architecture (VistA):

Strata Conference New York 2011, being held Sept. 22-23, covers the latest and best tools and technologies for data science -- from gathering, cleaning, analyzing, and storing data to communicating data intelligence effectively.

Save 20% on registration with the code STN11RAD


Sponsored post

July 28 2011

The future of community

This morning Jono Bacon from Canonical kicked off OSCON by talking about "The Future of Community", which he admitted was a vague and dangerous title to choose. People who try to predict the future tend to fail, but that didn't stop him. After musing about being chosen to give a keynote at OSCON he dove into his main point about how he feels that community management is at the beginning of a renaissance.

Historically, the first communities were human tribes. A lot of challenges faced early tribes: How do you feed tribe members? How do you keep tribes healthy? People didn't have community managers, they simply tried things and learned from what worked and what didn't work. Tribe members didn't set out to be community leaders, much like open source leaders didn't set out to be leaders. Open source hackers that succeed in creating valuable open source projects become leaders because of their efforts and the same was true for early tribes.

Jono went on not to predict the future, but to share an informed trend: Community management is at a renaissance. The renaissance connected the dark ages with the enlightenment and people started to educate themselves. People created a repeatable science: If this happens and then you take that action, you can expect a certain repeatable outcome. For instance, the concept of people getting sick, then taking pills and then getting well is a great example of repeatability.

We're seeing this happening with community management. We're seeing a profession of community management come about. More people than ever before are working professionally as community managers; these people are the connection points between communities and companies. When we have repeatable community experiences, we want others to be able to repeat these community developments!

Jono underscored the key lesson in his keynote: We're at the beginning of this community renaissance. We're going to see a repeatable body of knowledge that will allow us to push communities forward.


July 27 2011

Science Hack Day goes global

Science Hack DayGet a bunch of scientists and geeks in a room together for 48 hours straight and you're bound to get something interesting (Cold fusion? Laser-guided sharks?). Exactly what that might be and how it would be used remain open questions, but the compression of time, location and intensity — a sort of pressure cooker for creativity — is what Science Hack Day aims to create. It's an interesting model.

The Alfred P. Sloan Foundation thought so, too, and that's why they've given Science Hack Day a grant for expansion. Ariel Waldman, Institute for the Future research affiliate and Science Hack Day SF creator, will announce the expansion today at OSCON (we got her blessing to share the news a little early). Here's how Waldman says the new money will be put to use:

10 people from around the world will be selected to win a scholarship for a trip to Science Hack Day San Francisco 2011 where they'll experience first-hand how Science Hack Day works and connect with a global community of organizers. This Science Hack Day Ambassador Program will award individuals who are motivated and planning to organize a Science Hack Day in their city.

Further details on the Ambassador program are available at the Science Hack Day website.

July 25 2011

Late breaking OSCON changes

There are always last minute changes at OSCON. In the associated podcast, Edd Dumbill and I talk about what's changed. We highlight:

July 24 2011

What's happening at OSCON Java ?

This year at OSCON we're highlighting Java. Edd Dumbill and I talk about his Seven reasons you should use Java again Radar post and the upcoming sessions at OSCON Java.

The post received quite a few comments and we share our thoughts on them as well as the the OSCON Java program in the associated podcast.

OSCON Java 2011, being held July 25-27 in Portland, Ore., is focused on open source technologies that make up the Java ecosystem. (This event is co-located with OSCON.)

Save 20% on registration with the code OS11RAD

July 18 2011

Emerging languages show off programming's experimental side

The emerging languages space is programming's work bench. It's where novel theories are tested and new ideas are tweaked — and the underlying motivations are often less important than the work being done.

Characterized this way, it's easy to understand why last year's OSCON emerging languages camp proved to be popular. It offered an unusual opportunity to focus on the essence of the field rather than strict practical applications. This year, to meet the demand and amplify the work being done, emerging languages has been promoted to a full-fledged conference track at OSCON 2011.

I got in touch with the track's organizer, Alex Payne (@al3x), to discuss the particulars of this year's schedule as well as the broader state of emerging languages. Our interview follows.

Which languages in this year's emerging languages track intrigue you most?

Alex PayneAlex Payne: One of the things that's neat this year is that there's a couple of very experimental languages in the track. One of the most interesting to me is Wheeler, which describes itself as a language with no objects, functions or variables. It's difficult to describe how that works in practice.

Ola Bini, author of the Ioke language, has a new language called Seph, which is sort of an experiment in using Java 7 as a platform for building new languages. It builds on some of the ideas in Ioke and it borrows from Clojure a bit. Seph could be really interesting.

The FAUST language was a late entry. It's a programming language for real-time signal processing and synthesis. I like that it's so domain specific. That's nice to see because a lot of the languages we got last year were general purpose, but this one is very focused.

OSCON Emerging Languages Track — Learn and exchange ideas with the inventors of tomorrow’s computing future in OSCON's Emerging Languages track.

Save 20% on registration with the code OS11RAD

Do the languages you're seeing plug in to broader trends, like the popularity of Node.js?

Alex Payne: I'm not the biggest Node fan, and I kind of think it's the other way around. Node applies research and techniques that have been around for a long time. It's bringing those ideas to a broader group of programmers who know JavaScript as a common language.

The thing that Node, and Ruby before it, have done well is remind language developers that usability and community really matter. Node, in a very short period of time, has developed an enthusiastic community. They've developed good tools and tutorials around it. It's friendly, it's readable, and there's lots of examples. Developers working on the cutting edge can forget to do that stuff, and then they wonder a year or two down the line why no one is interested in their project.

A comment attached to our 2010 interview questioned the need for emerging languages — "evolution is better than revolution" was the exact phrase. What's your take on that?

Alex Payne: Most of these languages are evolutionary and not revolutionary. Clojure is a synthesis of stuff going on in concurrent programming, and it borrows heavily from Lisp. Go is certainly an evolutionary language, building on the enduring qualities of C and attempting to correct what some might see as the missteps of C++, Objective-C, and Java.

I don't think that a lot of these languages come along and say everything we're doing in the realm of programming is broken and we have to rethink it. The only person who came to last year's camp with that kind of agenda was Jonathan Edwards, who presented on his Coherence language. But he had a lot of push-back at the end of his presentation, and this was in a room full of people who were interested in emerging languages.

Your perspective on this depends on how close you are to these emerging language conversations. If you're the sort of programmer who only picks up a new language once or twice a decade, then yes, seeing people talk about the need for new languages is probably revolutionary and probably threatening. If you're involved in discussions about programming languages day-to-day on mailing lists or on sites like Lambda the Ultimate, Hacker News, or Reddit, then you can watch these new languages slowly take shape. With that social context and perspective, new languages look much more evolutionary.

Are there categories or problems this year's crop of languages are trying to address? I'm thinking of things like mobile or big data.

Alex Payne: Most of the languages are not going after the problems the industry as a whole is thinking about right now. The submissions that we got for this year's track are focused on experimenting with the fundamentals of the languages themselves. For example, we have a language called Plaid that's an experiment in a concept called typestate. We also have a talk that's about object-functional languages like Scala and how you resolve some of the problems in them. The group of people submitting talks for this track seem to be more interested in ironing out issues in the language world rather than addressing industrial problems.

Are you surprised at the popularity of any current languages?

Alex Payne: I'm constantly surprised at the popularity and success of Objective-C. Almost everyone I know tells the same story about Objective-C: they started learning it and they hated it. They thought it was the worst of C and the worst of dynamic languages. And then eventually, they learned to love it.

Most of the time, that software works pretty darn well, so who am I to judge? I'm pleasantly surprised by the continued success of Objective-C, and I think it should be taken as a lesson for the language designers out there.

This interview was edited and condensed.


To get things done, be "reasonably unreasonable"

Turing_Plaque.pngIn a recent interview, John Graham-Cumming (@jgrahamc), VP of engineering at Causata, Inc. and a speaker at OSCON 2011, said to change the world, it's often necessary to act in a "reasonably unreasonable" manner — to go just enough against the norm to effectively rock the boat.

He knows what he's talking about. In 2009, using a blend of new and old media tools and a bit of geek expertise, Graham-Cumming got the UK government to apologize for its treatment of mathematician and computer scientist Alan Turing in the 1950s. Below he discusses the techniques that produced that apology.

Your OSCON session description says people need to be "reasonably unreasonable" to change the world. What does that mean?

JohnGraham_Cumming.jpgJohn Graham-Cumming: You have to be "unreasonable" to get things done. By that, I mean that you have to go against the norm. If you are reasonable and go in the direction of society, then you don't contribute greatly. Just look at people like Richard Stallman and Linus Torvalds. Stallman's ideas were pretty "unreasonable" at a time when there was a large move to proprietary software. Torvalds was "unreasonable" in thinking that he could build his own kernel and in telling Tanenbaum where to go.

I say "reasonably" because you shouldn't take being unreasonable so far that people don't listen to you.

How did you apply that philosophy to your campaign to make the British Government apologize for the mistreatment of Alan Turing?

John Graham-Cumming: The important thing about the Turing campaign was that I felt that people were celebrating Turing without acknowledging the harm done to him by Britain's laws at the time. I didn't want people to be able to sweep this under the rug, so I decided to just tell everyone about what happened to him and ask for an apology from the UK government. That was pretty "unreasonable" in the sense that the UK government doesn't apologize for much, and I thought they would ignore my request.

Geek Lifestyle at OSCON 2011 — From fine-tuning your setup to taking the geek approach to growing your own food, we'll celebrate and explore hacker culture in all its richness in the Geek Lifestyle track at OSCON (July 25-29 in Portland, Ore.)

Save 20% on registration with the code OS11RAD

What tools did you use in your Turing campaign?

John Graham-Cumming: Twitter was very effective at getting people to hear about the campaign, but it's an echo chamber and builds pretty slowly. You tend to get the same circle of people mentioning an issue because they care about it, and it's hard to get it to a large audience.

What really works is having a definitive source of information that people can point to on Twitter. So, when the BBC wrote about the Turing campaign on its website, Twitter was able to amplify that with thousands tweeting — and ultimately signing the petition. In some ways, what Twitter needs is a Wikipedia-style "[citation needed]" so that people take what's written on it seriously.

Facebook also was helpful, but I think Twitter was much more effective.

I also appeared on countless radio programs, on TV, in print and anywhere else I could. I made myself the focus of the campaign initially, and then when celebrities started signing, I used them to get the media to talk about the campaign.

The important thing to realize with the media is that there needs to be a hook or peg onto which the story they are telling can be hung. So, initially there was some press about the campaign starting, and then I'd badger people in the press when there was a suitable hook — for example, when Richard Dawkins signed and publicly stated his support. You have to look for things to tell the press so they know what to write about.

What kind of code did you use in your campaign, and how did you use it?

John Graham-Cumming: I used a custom Perl script that downloaded the names of the signatories every hour, looked them up on Wikipedia and, using some simple techniques, figured out if they were celebrities of any kind. If they were, then I was sent an email by the script and would try to get in contact with the celebrity to see if I could use his or her name.

How do you translate technical and historical concepts into calls to action?

John Graham-Cumming: You just have to tell a human story. In the case of Turing, this was easy: he was clearly a genius, a war hero, and then he was prosecuted and he committed suicide.

This interview was edited and condensed.

Photo: Turing Plaque by Joseph Birr-Pixton, on Wikimedia Commons


  • Lessig on Culture and Change
  • Creating cultural change
  • John Graham-Cumming's project to build Babbage's Analytical Engine

  • July 13 2011

    Who are the OSCON data geeks?

    This podcast highlights some of the sessions in OSCON Data and who might be interested in them.

    Edd Dumbill, Bradford Stephens and I took the liberty of making irreverent monikers for several of the types of attendees we expect at OSCON Data. These include:

    DBA Dude
  • Data Scientist
  • NOSQL Nerd
  • Scaling Geek
  • Real-time Traveler
  • (Podcast production by Rich Goyette Audio.)

    OSCON Data 2011, being held July 25-27 in Portland, Ore., is a gathering for developers who are hands-on, doing the systems work and evolving architectures and tools to manage data.

    Save 20% on registration with the code OS11RAD

    July 12 2011

    Make your nomination for the OSCON Data Innovation Award

    The OSCON Data Innovation Award, to be announced at OSCON Data (July 25-27 in Portland, Ore), will be given to an organization, open source project, or individual who has made a significant contribution to the burgeoning data management field.

    Nominations are open to the entire open source community. Al entries will be judged by the award committee, which includes: Sarah Novotny (Program Chair, OSCON and OSCON Data), Bradford Stephens (Program Chair, OSCON Data), Edd Dumbill (Program Chair, OSCON), and Mike Olson (CEO, Cloudera).

    You can submit a nomination here. Nominations should include the name of the recipient, any associated project or organization, and a reason of why you're nominating the company, project or individual. The nomination window closes on July 18, 2011.

    (Note: O'Reilly employees cannot be nominated.)

    OSCON Data 2011, being held July 25-27 in Portland, Ore., is a gathering for developers who are hands-on, doing the systems work and evolving architectures and tools to manage data.

    Save 20% on registration with the code OS11RAD

    July 11 2011

    JavaFX 2.0: Making RIA with Java

    JavaFXAs far as Rich Internet Applications (RIA) are concerned, the major player for the last decade has been Flash, and its successor Flex. Silverlight has had its supporters as well, and HTML5 is moving the ball forward from AJAX and CSS. And then there's Java.

    After a brief stint of popularity, Java applets fell out of style, and even sites that used Java in the back-end rarely used it on the client side. JavaFX, now in its second generation, is an attempt to bring Java back onto the client side, and Jim Weaver is a big fan. Founder of JMentor, Weaver thinks that JavaFX brings a rich programming environment to the client, he'll be talking about that at OSCON later this month. He recently clued us in on why he thinks JavaFX is such a strong contender.

    Can you give us a short history and description of JavaFX?

    Jim WeaverJim Weaver: JavaFX came out in 2007. It was created by a guy named Chris Oliver, and it was introduced as a way to be able to quickly and intuitively put together rich client Java user interfaces. Developers in general, and the industry in general, rejected JavaFX primarily because it was a new scripting language. They didn't want to learn a language, so it just didn't get the traction that it needed.

    After Oracle acquired Sun, they took a different direction with JavaFX. They abandoned JavaFX Script, the scripting language for it, and made JavaFX a set of libraries, APIs and run time. You code it in pure Java or any Java domain-specific language. That's the focus now. The new version just got out of early access and into a very early beta in June, and it's starting to get some traction with developers trying it and using it for smaller applications.

    OSCON Java 2011, being held July 25-27 in Portland, Ore., is focused on open source technologies that make up the Java ecosystem. (This event is co-located with OSCON.)

    Save 20% on registration with the code OS11RAD

    The argument could be made that with Flash/Flex so ubiquitous on browsers now, and HTML5 gathering steam, another RIA platform is redundant.

    Jim Weaver: I have nothing bad to say about Flex. I think Flex has been a great rich-client environment, and they've done a great job with deployment of the run times, except for iPhone and iPad.

    The reason why I don't use Flex is because it's not Java. It doesn't have the millions of Java classes that are available out there and the richness of the native Java APIs. You find yourself having to use some bridging technologies. So that's why I'm pushing for Java. But there's nothing wrong with Flex. Flex is great.

    I do see a lot of advances in HTML5. I think there's a lot of misconceptions and expectations that need to be calibrated around what HTML5 is: rate of adoption, rate of compliance on different browsers, and that kind of thing. Fundamentally, you're still talking about trying to make browsers an application execution platform. You're still working with JavaScript and you're still trying to shoehorn some technologies.

    The mix that I'm shooting for would combine HTML5 and rich-client Java in a number of ways: it would be rich-client Java, web-started, that contains JavaFX APIs for its richness on the client, with a scene graph that contains UI controls, effects, animations and transformations. JavaFX has an embedded web browser component that you can use as part of the scene graph, and it's adopting the HTML5 standard.

    The other way of doing it would be the exact opposite, where you have a browser and it's got an app that has JavaFX. I don't like that as well because then you depend on the browser for your JVM and your Java execution environment.


    July 07 2011

    Seven reasons you should use Java again

    O'Reilly is celebrating the release of Java 7, and our inaugural OSCON Java conference: July 25-27 in Portland, Ore.

    Java's back, baby. Of course, lots of us never stopped using Java in the first place, or perhaps we weren't allowed to. But where platform selection was elective, especially among startups and web developers, alternative platforms have offered more agility and expressiveness over the last 10 years. The likes of PHP and Ruby on Rails enable developers to do a whole lot more with less.

    Times change, and 16 years on, this ain't your father's Java. Here's seven reasons you should take another look at Java.

    1. You can actually hire engineers

    Not to be sniffed at when your Node.js Ninjas and Rails Rockstars get headhunted into the latest impossibly well-funded startup.

    2. IDEs take the pain away

    Eclipse and NetBeans are astonishingly powerful tools, and can mask some of the admitted horror that is Java API soup. Bend like a reed, admit you can't know everything, and float downstream on the good ship autocomplete.

    3. Language support

    You don't have to write Java to use Java. You can get all the benefits of the portable JVM runtime but scribble away in the familiar environs of Ruby or Python. It can be faster too. And that's not to mention the newer languages aimed at modern programming such as Scala, Groovy or Clojure.

    OSCON Java 2011
    To celebrate the release of Java 7, the first 77 people registering today for OSCON Java with code JAVA7 will get the pass at a discounted price of $700 (applicable to OSCON Java package only).

    Register now with code JAVA7

    4. Android

    Who can resist the little green robot? Google made a sensible choice when they chose the Java language to power what's becoming the world's dominant mobile phone platform. So, maybe Oracle has a billion-dollar beef with this, but programmers — new and seasoned alike — are picking up Java as mobile becomes the future of consumer software.

    5. Everybody else does

    Java's ecosystem is a goldmine. Pretty much everything you want to do, there's a library for it. Actually, there's probably an Apache project for it. Very often the lowest impedance way to speak to the rest of the world is via a Java API.

    6. It changes slowly

    We're celebrating the release of Java 7 right now. It's been two years in the making, and frankly, there's not a whole lot that's changed for most people. If you can take your finger off the refresh button at Hacker News for more than an hour, any software that lives longer than a year becomes a real pain to maintain when the underlying platform keeps changing. Yes, looking at you, Rails.

    7. You'll end up using it anyway

    At a certain point you'll need performance, predictability and a ready supply of engineers. Scaling, deploying and programming to the cloud are places where Java excels. Twitter found this out, and will be talking about it at OSCON Java.

    There's still time left to register for OSCON Java. See you in Portland, 25-27 July!


    A rough guide to JVM languages

    O'Reilly is celebrating the release of Java 7, and our inaugural OSCON Java conference: July 25-27 in Portland, Ore.

    The possibility of using alternative languages on the JVM has always been an appealing side story to Java. Jython and JRuby were early pioneers in implementing a dynamic language on top of a statically typed VM. Now that Java 7 directly includes support in the JVM for dynamic languages, the coexistence is official.

    Pick any of the languages here and you can expect the support of the robust JVM threading and garbage collection, plus access to a broad array of application libraries.

    Up and coming


    The pragmatist among JVM languages, Scala is a general-purpose programming language. It was conceived and is generally viewed as a "better Java," and of all alternative JVM languages has the best acceptance in an enterprise setting.

    Scala combines the familiar feel of object-oriented Java with strong language support for concurrency, XML and functional programming features: many of the tools that contemporary complex and scalable systems require. Scala also takes a lot of the awkwardness out of Java code, through features such as type inference and traits.

    object HelloWorld {
        def main(args: Array[String]) {
          println("Hello, World!")

    Scala-related content at OSCON Java:

    OSCON Java 2011
    To celebrate the release of Java 7, the first 77 people registering today for OSCON Java with code JAVA7 will get the pass at a discounted price of $700 (applicable to OSCON Java package only).

    Register now with code JAVA7


    Clojure is a functional programming language based on Lisp. Through careful design, Clojure is simpler to read and use than Lisp, and it interacts cleanly with the Java world. Its functional nature makes programs very concise and composable.

    In common with Scala, Clojure is designed with concurrency in mind. Its variables are immutable, and the use of software transactional memory and agents help manage shared mutable state in a more sustainable way than locking.

    The experience of Clojure's inventor Rich Hickey is reason enough alone to give it a whirl. I once happily spent an hour listening to him describe the design and implementation of sequences in the language.

    A "Lisp that could," Clojure is finding increasing (and surprising, to most observers) traction and acceptance. One contributing factor to this is an advanced build and package management infrastructure in leiningen and Clojars. Salesforce-owned hosted platform provider Heroku recently added Clojure as its third supported environment, following Ruby and Node/JavaScript.

    A small but important omission from the Clojure ecosystem is a port of the book The Little Schemer.

    (println "Hello, World!")

    Clojure-related content at OSCON Java:

    Tried and tested


    A mashup of ideas from Python, Ruby and Smalltalk, Groovy gained early traction in the world of JVM languages, reaching its stable release in December 2007. Having been around a while, Groovy strongly acknowledges the Java world into which it was born, and is a good choice for Java developers wishing to use a more agile and dynamic language.

    One of Groovy's jewels is Grails, a high-productivity web development environment inspired by Ruby on Rails. The Groovy world has also spawned Gradle, a modern project automation system for Java and JVM languages, providing an alternative to the established Ant and Maven projects.

    println "Hello, World!"

    Groovy-related content at OSCON Java:


    Another veteran of the JVM language scene, Rhino is an implementation of the JavaScript programming language in Java. Part of the Mozilla project, it is typically used by developers to add user scriptability to their applications using JavaScript, though it is also used in situations where a system is predominantly implemented in JavaScript.

    First released in 1999, Rhino has been around the block a few times. As a JavaScript engine, it is facing increasing competition from the C++-based V8, but retains the immense advantage of Java interoperability.

    print('Hello, World!');

    Remastered classics


    A port of the Python language to the JVM, Jython offers some advantages over and above using Python, including Java's multi-threading and the ability to statically compile into Java classes. As a Python terse object-oriented language, programmers can quickly prototype in Jython, creating hybrid Java-Jython systems. Jython also offers developers an alternative to Rhino's JavaScript for embedding scriptability in their applications.

    Jython also lets web developers bring popular Django web application framework into a Java setting.

    print "Hello, World!"


    The Java implementation of Ruby is no second-class citizen. JRuby has been recorded as better performing than Ruby 1.8, and strives for C-Ruby compatibility. Users of JRuby can benefit from using the Ruby on Rails web framework, or add Ruby scriptability to their applications.

    One appealing aspect of building a system in JRuby or Jython is that it gives developers the option of reimplementing performance-critical code in Java without having to switch platforms as a project matures. Twitter's migration from Ruby to the JVM is a case in point.

    puts "Hello, World!"

    JRuby-related content at OSCON Java:

    The Wild West

    The JVM remains a fertile ground for language experimentation, as several of the languages featured in OSCON's Emerging Languages track demonstrate, including Seph and Gosu. As a platform, the JVM bootstraps language inventors and experimenters today much as lex and yacc did two decades ago. At OSCON Java, aspiring language creators should pop along to Charles Nutter's JVM Bytecode for Dummies.

    There's still time left to register for OSCON Java. See you in Portland, 25-27 July!


    Seven Java projects that changed the world

    O'Reilly is celebrating the release of Java 7, and our inaugural OSCON Java conference: July 25-27 in Portland, Ore.

    Java's open source ecosystem is strong and healthy, one of the primary reasons for our creation of OSCON Java. Over the last decade, several projects have traveled beyond mere adoption and had effects dominating the Java world, into software development in general, and some even further into the daily lives of users.


    Ported to Java by Kent Beck and Erich Gamma from Beck's work in unit testing in Smalltalk, JUnit has been largely responsible for popularizing test-driven development over the last decade. Many implementations have been created, in .NET, C, Python, Perl and just about every language in popular use.


    As Java and its APIs matured in the early 2000s, the Eclipse IDE provided a way for programmers to be productive and negotiate the growing Java ecosystem. Eclipse was also the first major project to use the SWT UI toolkit, providing important competition to Sun's Swing and showing that Java programs can provide a rich native interface. Eclipse has evolved toward a goal of being a universal IDE, and it now provides a rich foundation for platform vendors to integrate with.


    The Spring Framework has played an important role in enabling Java developers to be productive, managing a balance between simplicity and features. Spring gives Java developers a set of services providing commonly used application functionality such as data access and transaction management. As a competitor to Sun's Enterprise Java Beans system, Spring enabled an alternative and simpler path for Java applications, as well as ensuring a healthy competition of ideas in the way Java applications are constructed.

    OSCON Java 2011
    To celebrate the release of Java 7, the first 77 people registering today for OSCON Java with code JAVA7 will get the pass at a discounted price of $700 (applicable to OSCON Java package only).

    Register now with code JAVA7


    The Solr server, and the Lucene search engine it encapsulates, has been for many years a simple and practical solution to providing search capabilities to web and enterprise applications. Solr's genius is in providing HTTP access to the powerful and fast Lucene search library, enabling it to become a part of any system, regardless of whether it is implemented in Java or not. More than any other project, Solr has ensured that good search is a checkbox item for modern web applications.

    Hudson and Jenkins

    Originally developed as Hudson, and now also as Jenkins, this continuous integration tool is a key part of a Java development setup. Jenkins provides automated build and testing of a software project, continuing in the footsteps of JUnit in enabling agile development on the Java platform. While both Hudson and Jenkins persist for now as forks of each other, it doesn't detract from the work of Kohsuke Kawaguchi in creating a world-class continuous integration platform and so enhancing the quality of much Java development.


    This Java implementation of the famous MapReduce model is the powerhouse that has enabled most "big data" systems. By lowering the cost of extracting value from large data sets, Hadoop has made practical the personalization and advertising businesses of Facebook and Yahoo, and many other companies. In the same way that Linux enabled large websites to be built on cheap hardware, Hadoop enables large-scale distributed computing by handling failure at the software level. Both Hadoop and the above-mentioned Lucene are the product of the work of Doug Cutting.


    Controversy is never that far from Java, whether the custodian be Sun or Oracle. Google's choice to use Java as the programming language for its massively popular Android mobile operating system has led to a renewed interest in the language from whole new communities of software developers. Android programs undergo a further step to convert JVM bytecode to Dalvik bytecode — Dalvik being a virtual machine optimized for mobile devices. Google has been able to leverage Eclipse to provide software developers with a mature development environment for creating Android applications.

    Oracle and Google are currently engaged in a lawsuit over a claim that Android infringes on multiple patents held by Oracle. The results of that suit notwithstanding, Android has done much to recruit developers and perpetuate the use of Java in client-side software development.

    There's still time left to register for OSCON Java. See you in Portland, 25-27 July!


    Into the wild and back again

    The psychological wear and tear of office life leads many to fantasize about leaving it all behind. Ryo Chijiiwa knows this feeling well, but unlike most people he actually did something about it. In 2009, Chijiiwa quit his job at Google, packed up and moved off the grid.

    In the following interview, Chijiiwa, who will speak in-depth about his experiences at OSCON, talks about how solitude and nature have shaped his perspective.

    What possessed you to do an about-face from working for high-profile tech companies and ditch the grid?

    RyoChijiiwa.jpgRyo Chijiiwa: Part of it was that I was simply burnt out. I had spent the better part of 10 years either studying computer science in college or working as a professional software engineer (or both), and I suddenly decided I wanted to experience life outside the cubicle. Living in the woods was a childhood dream of mine, and it seemed like a good time to realize that dream.

    But, the other part was that I started to see some fundamental issues with the way the industry and our society are structured. At a personal level, I realized that striving for success and accomplishment didn't bring me any closer to happiness. And at a societal level, it occurred to me that a system predicated on infinite growth simply was not sustainable. So, I decided to step back, slow down, and rethink my life and my priorities.

    Can you describe a typical day in the wilderness?

    Ryo Chijiiwa: There's really no "typical" day in the wilderness. For a long time, there was actually a fair amount of work to do because I was trying to turn a patch of completely undeveloped land into something habitable. I built my cabin mostly on my own, and that alone took several months. The work that needs to happen also varies depending on the season. In the spring, I might be tending to the garden, or clearing brush to lower the risks of a forest fire. In the autumn, I might spend a lot of time gathering firewood. I also dedicate a fair chunk of time to cooking because food is important. There's also ample time for reading, writing, reflection and contemplation, too, which is one of the benefits of a slower lifestyle. Of course, if I can't find anything better to do, I can always step out of my cabin and go wander the woods.

    Geek Lifestyle at OSCON 2011 — From fine-tuning your setup to taking the geek approach to growing your own food, we'll celebrate and explore hacker culture in all its richness in the Geek Lifestyle track at OSCON (July 25-29 in Portland, Ore.)

    Save 20% on registration with the code OS11RAD

    As far as communications go, I was completely cut off for a while. I had an iPhone, but AT&T had absolutely no coverage on my property, and I actually enjoyed being disconnected. There's a certain peace of mind you can get only by switching off completely. I eventually found out that Verizon had coverage, so I got a feature phone on a pre-paid plan for emergencies and for those few occasions when I needed access to the outside world.

    After much deliberation, I got a MiFi earlier this year so I could go online, though I can't say getting "wired" was unequivocally better for my quality of life. With Internet access, I spend more time and electricity on my laptop, uselessly browsing the web when I could be doing something else. I think this is a common problem people have these days, but the shift that happened when I suddenly got Internet access really made it noticeable.

    Electricity is another constraint. This past winter, when sunlight was scarce and my solar panels were covered in snow, I once had to tell my mom, who lives in Japan, that I couldn't Skype with her unless the sun shined. While the lack of power was something of an inconvenience, it was also reassuring to know that I could have power as long as the sun shines, which isn't something you can say when you're dependent on the grid and the power goes out.

    What has solitude taught you?

    Ryo Chijiiwa's Hut 1.0
    Ryo Chijiiwa's Hut 1.0. He's currently working on Hut 2.1.
    Ryo Chijiiwa: I've learned a ton. I've learned some carpentry and architecture from designing and building my own cabin. I've also learned a lot about off-grid electricity, about the importance of water, gardening, wildlife, and self-reliance, to name a few things. But the fact that I gained knowledge and skills is hardly surprising.

    What I think made this experience uniquely valuable for me, though, is that I've learned so much about myself. In many ancient cultures, venturing out into the wilderness alone was a rite of passage, a necessary step toward adulthood. In our society, on the other hand, isolation is feared and even stigmatized. Yet, there's a lot about yourself that you can learn only through isolation and solitude. Sometimes, you can't hear yourself unless you put yourself in an environment where there's nobody else — no parents, no bosses, no peers. And knowing who I am, what my strengths and weaknesses are, and knowing what's really important to me is invaluable because the truly difficult decisions in life can only be solved if you know who you are.

    Do you expect to bring that new knowledge back into the grid at some point?

    Ryo Chijiiwa: Absolutely. I've been a fan of the open source model for a long time now, and I think a big part of it has to do with my desire to share and contribute things, whether it's code or knowledge. My entire journey, since the day I left Google, has been chronicled in my blog, Laptop and a Rifle, where I tried to make the whole experience pretty transparent. I'm also working on a book that's filled with practical knowledge, which will hopefully be published as an ebook sometime later this year.

    What are the benefits of alternative lifestyles? What do they allow people to do?

    Ryo Chijiiwa: Alternative lifestyles can have a number of advantages. The major one, I think, is that it helps us strike a better work-life balance. For example, I don't have to choose between working and living in a cabin in the woods because I can do both! One doesn't necessarily have to choose between working and traveling — you can do both! At the very least, there's so much more you can do when you're not spending 60 hours at the office.

    I think there are some benefits to society at large, too. Living in a 120-square-foot cabin in the woods and living purely off of solar energy probably helped me reduce my carbon footprint. After the March 11 earthquake in Japan, I was able to jet off and volunteer in the tsunami disaster area for two months, which would have been difficult to pull off if I had been tethered to a job and a mortgage. Also, by sharing what I've learned, I'm hoping that that information will help others realize their own dreams, and live healthier, happier, and more sustainable lives.

    Do you believe it's possible to find balance between always on and completely off?

    Ryo Chijiiwa: It's very, very difficult. When I'm in my cabin, I've accomplished something of a middle ground, simply because I have a limited supply of electricity and my MiFi account has a 3GB per month data limit. The trend is clearly going in the other direction. Everything is going into the cloud, which means you'll need an "always on" connection to have access to not just your email and social life, but your photos, favorite TV shows and your music.

    Living in the woods, and going from being completely disconnected to being mostly connected, made me realize how difficult it actually is to incorporate technologies into our lives in a healthy way. The pace of technological change is so blindingly fast that we're doing a poor job of adapting, not just at an individual level, but even as a society and as a species.

    The environmental impact of technological changes that started two centuries ago only became apparent a few decades ago. It concerns me somewhat to think about how, two centuries from now, our descendants will look back on today's technologies and the impacts they have on people, societies and our environment. There will likely be unforeseen consequences, some of which may prove to be undesirable. Unfortunately, only time will tell.

    This interview was edited and condensed.


    July 01 2011

    Music and lyrics and code

    If the popularity of Geek Choir at various tech-related conferences is any indication, there's a substantial correlation between producing music — whether vocally or with an instrument — and coding.

    Michael Brewer (@operatic), application programmer specialist at the University of Georgia and a speaker at OSCON 2011, got the official Geek Choir sessions started at the Open Source Bridge and O'Reilly OSCON conferences. In a recent interview, he discussed how the choir came about and how music and coding complement each other.

    How do music and technical aptitude intersect?

    MichaelBrewer.jpgMichael Brewer: Since Geek Choir got accepted, I've been hearing a lot of anecdotal evidence of a high crossover between music and geek aptitude. Of course, people have been talking about the math-music connection since "Gödel, Escher, Bach." Recent studies have again shown connections between early exposure to music and math ability, although it's not exactly what we think of as the "Mozart Effect."

    I tend to view it as a combination of pattern recognition and the ability to organize and reproduce thoughts about larger, more abstract concepts and their executions. Also, the production of sound using tools at hand — including vocal chords — is similar, in a sense, to producing code with software or hacking other projects.

    We are a species that bonds with our tools in unusual ways. Correlation doesn't prove causation, though — there's a bit of a chicken-and-egg problem in figuring out if geeks are good at music or if musicians are good at being geeks.

    Geek Lifestyle at OSCON 2011 — From fine-tuning your setup to taking the geek approach to growing your own food, we'll celebrate and explore hacker culture in all its richness in the Geek Lifestyle track at OSCON (July 25-29 in Portland, Ore.)

    Save 20% on registration with the code OS11RAD

    What is a "Geek Choir," and how did it get started?

    Michael Brewer: I attended several OSCONs in the early-to-mid 2000s. I noticed several music jam sessions, as well as the popularity of the Gibson guitar booth in the exhibit hall one year. Folks were jamming on those guitars for hours.

    At the first Open Source Bridge Conference, I suggested a Geek Choir session for the "unconference" on the last day. We started with fairly few people, but once we started singing — and people in the halls heard us and "voted with their feet" — we more than tripled our attendance in 15 minutes. The next year, Geek Choir made it into the OSBridge main conference. We had a very successful and enjoyable session, mixing experienced singers with absolute newbies.

    In your OSCON session description, it says there's no advanced prep for the session choirs — why did you decide to go that route, and what benefits does a no-prep environment create?

    Michael Brewer: It makes it easier on the newbies if everyone is getting introduced to the music at the same time. Also, it means that I have to be sure in my preparation to select music that is both accessible for inexperienced singers and worthwhile for experienced musicians. It's a good engineering challenge.

    What are some tips for putting together a Geek Choir?

    Michael Brewer: Here's a few:

    • Be welcoming and respectful. Everyone can contribute, even if they have never sung in public before or don't read music.
    • Choose — or compose — music that can be done by a mix of voices, both in terms of range and skill level. Parts can be done, but they have to be fairly straightforward to pick up. Shape note songs are good for this, as they were specifically engineered to (a) be easily learned by the (somewhat) untrained American choirs of the late 1700s and early 1800s, and (b) be performed with mixed genders on the various lines — sopranos and tenors would both sing the melody, for example, and altos and basses the bass parts, at comfortable octaves for each.
    • Stay in the public domain when you can. There are some tremendous repositories of music, including the Choral Public Domain Library and the International Music Score Library Project.
    • Have a great time!

    There are a variety of open source tools for arranging music. Which do you recommend?

    Michael Brewer: In general, there are two types of music composition software: music sequencers, which work with manipulating and combining blocks of sounds (sequences) into larger musical works, and notation software, which deals primarily in written or printed music.

    Wonderful music is created with either. I generally work with notation software, so I'm much more familiar with notation editors. In this arena, everyone is chasing the main two commercial products — Finale (which I use) and Sibelius. For a long time, the open source tools weren't really comparable, in terms of ease of use, but MuseScore has really closed the gap. There's also LilyPond. I haven't worked with it yet, but I've heard good things about it.

    What similarities, if any, do you see in the communal qualities of music and the communities that grow around open source projects?

    Michael Brewer: There are several:

    • Both are groups of people coming together to create something, be it software or music.
    • There is artistry in the finished product for both. Code is most definitely art.
    • People vote with their feet for both, in terms of joining and leaving.
    • Coming together to work on common tasks builds connections and solidarity among the members. They tend to view themselves as a collective, giving themselves an identity as part of a larger whole.


    June 30 2011

    How Netflix handles all those devices

    Netflix's shift to streaming delivery has made quite an impression on Internet traffic. According to Sandvine's latest report, Netflix now claims almost 30% of peak downstream traffic in North America.

    That traffic occurs, in no small part, because Netflix can run on so many devices — PCs, tablets, gaming consoles, phones, and so on. In the following interview, Netflix's Matt McCarthy (@dnl2ba) shares a few lessons from building across those varied platforms. McCarthy and co-presenter Kimberly Trott will expand on many of these same topics during their session at next month's OSCON.

    What are some of the user interface (UI) challenges that Netflix faces when working across devices?

    Matt McCarthyMatt McCarthy: Scaling UI performance to run well on a low-cost Blu-ray player and still take advantage of a PlayStation 3's muscle has required consulting WebKit and hardware experts, rewriting components that looked perfectly good a week before, and patiently tuning cache sizes and animations. There's no silver bullet.

    Since we've standardized on WebKit, we don't have to support multiple disparate rendering engines, DOM API variants, or script engines. However, there are lots of complex rendering scenarios that are difficult to anticipate and test, especially now that we're starting to take advantage of WebKit accelerated compositing. There are WebKit test suites, but none that are both comprehensive and well documented, so we're working on our own test suite that we can use to validate partners' ports of our platform.

    OSCON JavaScript and HTML5 Track — Discover the new power offered by HTML5, and understand JavaScript's imminent colonization of server-side technology.

    Save 20% on registration with the code OS11RAD

    How do the platform lessons Netflix has learned apply to other developers?

    Matt McCarthy: The challenges we face may be familiar to many large-scale AJAX application developers. In addition, mobile developers need to make similar trade-offs between memory usage and performance, other sophisticated user interfaces need to handle UI state, and most large code bases can benefit from good abstraction, encapsulation, and reuse.

    The urgency and difficulty of solving those challenges may differ for different applications, of course. If your application is very simple, it would be silly for you to use the level of abstraction we've implemented to support A/B testing in Netflix device UIs. But if you're innovating heavily on user experience, your performance isn't always what you'd like, and your UI is an endless font of race conditions and application state bugs, then maybe you'd like to learn about our successes and mistakes.

    There were reports last year that some Netflix PS3 users were seeing several different UIs. What are the benefits and challenges with this kind of A/B testing?

    Matt McCarthy: Netflix is a subscriber service, so ultimately what we care about is customer retention. But retention, by definition, takes a long time to measure. We use proxy metrics that correlate well with retention. Some of our most closely watched metrics have to do with how many hours of content customers stream per month. Personally, I find it gratifying to have business interests that are aligned closely with our customers' interests.

    The challenges grow as the A/B test matrix grows, since the number of test cell combinations scales geometrically with the number of tests. Our quality assurance team has been working on automated tests to detect regressions so a fancy new feature doesn't inadvertently break another feature that launched last month. Our engineers adhere to a number of best practices, e.g. defining, documenting, and adhering to interfaces so we don't find nasty surprises when we replace a UI component in a test cell.

    A/B testing user interfaces obviously takes a lot more effort than developing our "best bet" UI and calling it a day, but it's been well worth the cost. We've already been surprised a few times by TV UI test results, and it's changed the direction we've taken in new UI tests for both TV devices and our website. Every surprise validates our approach, and it shows us a new way to delight and retain more customers.

    This interview was edited and condensed.


    June 29 2011

    What CouchDB can do for HTML5, web apps and mobile

    CouchDBCouchApps are JavaScript and HTML5 applications served directly from the document-oriented database CouchDB. In the following interview, Found Line co-founder and OSCON speaker Bradley Holt (@BradleyHolt) talks about the utility of CouchApps, what CouchDB offers web developers, and how the database works with HTML5.

    How do CouchApps work?

    Bradley Holt: CouchApps are web applications built using CouchDB, JavaScript, and HTML5. They skip the middle tier and allow a web application to talk directly to the database — the CouchDB database could even be running on the end-user's machine or Android / iOS device.

    What are the benefits of building CouchApps?

    Bradley Holt: Streamlining of your codebase (no middle tier), replication, the ability to deploy/replicate an application along with its data, and the side benefits that come with going "with the grain" of how the web works are some of the benefits of building CouchApps.

    To be perfectly honest though, I don't think CouchApps are quite ready for widespread developer adoption yet. The biggest impediment is tooling. The current set of development tools need refinement, and the process of building a CouchApp can be a bit difficult at times. The term "CouchApp" can also have many different meanings. That said, the benefits of CouchApps are compelling and the tools will catch up soon.

    OSCON JavaScript and HTML5 Track — Discover the new power offered by HTML5, and understand JavaScript's imminent colonization of server-side technology.

    Save 20% on registration with the code OS11RAD

    HTML5 addresses a lot of storage issues. Where does CouchDB fit in?

    Bradley HoltBradley Holt: The HTML5 Web Storage specification describes an API for persistent storage of key/value pairs locally within a user's web browser. Unlike previous attempts at browser local storage specifications, the HTML5 storage specification has achieved significant cross-browser support.

    One thing that the HTML5 Web Storage API lacks, however, is a means of querying for values by anything other than a specific key. You can't query across a set of keys or values. IndexedDB addresses this and allows for indexed database queries, but IndexedDB is not currently part of the HTML5 specification and is only implemented in a limited number of browsers.

    If you need more than just key/value storage, then you have to look outside of the HTML5 specification. Like HTML5 Web Storage, CouchDB stores key/value pairs. In CouchDB, the key part of the key/value pair is a document ID and the value is a JSON object representing a single document. Unlike HTML5 Web Storage, CouchDB provides a means of indexing and querying data using MapReduce "views." Since CouchDB is accessed using a RESTful HTTP API and stores documents as JSON objects, it is easy to work with CouchDB directly from an HTML5/JavaScript web application.

    How does CouchDB's replication feature work with HTML5?

    Bradley Holt: Again, CouchDB is not directly related to the HTML5 specification, but CouchDB's replication feature creates unique opportunities for CouchApps built using JavaScript and HTML5 (or any application built using CouchDB, for that matter).

    I've heard J. Chris Anderson use the term "ground computing" as a counterpoint to "cloud computing." The idea is to store a user's data as close to that user as possible — and you can't get any closer than a user's own computer or mobile device! CouchDB's replication feature makes this possible. Data that is relevant to a particular user can be copied to and from that user's own computer or mobile device using CouchDB's incremental replication. This allows for faster access for the user (since his or her application is hitting a local database), offline access, data portability, and potentially more control over his or her own data.

    Now that CouchDB runs on mobile devices, how do you see it shaping mobile app development?

    Bradley Holt: While Android is a great platform, the biggest channel for mobile applications is Apple's iOS. CouchDB has been available on the Android for a while now, but it is relatively new to iOS. Now that CouchDB can be used to build iPhone/iPad applications, we will most certainly see many more mobile applications built using CouchDB in order to take advantage of CouchDB's unique features — especially replication.

    The big question is, will these applications be built as native applications or will they be built as CouchApps? I don't know the answer, but I'd like to see more of these applications built on the CouchApps side. With CouchApps, developers can more easily port their applications across platforms, and they can use existing HTML5, JavaScript, and CSS skill sets.

    This interview was edited and condensed.


    June 28 2011

    Clojure: Lisp meets Java, with a side of Erlang

    ClojureClojure seems to be the hot new language of the moment, attracting both Java developers and hackers who remember Lisp with fond memories. To see what all the buzz was about, I talked to Stuart Sierra (@stuartsierra), a Clojure developer and author of "Practical Clojure." Sierra will be speaking about Clojure in next month's OSCON Java track.

    What is Clojure? How does it differ from functional languages like Erlang?

    Stuart SierraStuart Sierra: Clojure is a new programming language that runs on the Java Virtual Machine. It's in the Lisp family of languages and it's a general purpose language, but it emphasizes a functional programming style with tools to manage shared state in large multi-threaded programs.

    It's similar to Erlang with its emphasis on functional programming, but whereas Erlang is designed for systems distributed across many machines, Clojure focuses on many threads within a single machine. Erlang is trying to solve the very difficult problem of making everything parellelizable across multiple machines. Clojure is doing something different. It's trying to make sure that many threads in the same machine can safely share state.

    How similar is Clojure to the syntax of traditional Lisp-like languages?

    Stuart Sierra: Clojure is probably a little bit closer to Common Lisp than Scheme in style. It would not be particularly difficult for someone with Lisp experience to pick it up, although they would have to learn about some of the differences in the syntax. Clojure has a little bit more syntax than Common Lisp or Scheme, so it's actually a little easier to read and it has fewer parentheses than Lisp-like languages typically do. We've done examples where we put Java code and the equivalent Clojure code side-by-side and there are the same number or fewer parenthesis in the Clojure code because it's so much shorter.

    OSCON Java 2011, being held July 25-27 in Portland, Ore., is focused on open source technologies that make up the Java ecosystem. (This event is co-located with OSCON.)

    Save 20% on registration with the code OS11RAD

    Some developers have a visceral negative reaction to Lisp. Why do you think that is?

    Stuart Sierra: I think the problem people often have with Lisp is based on first encountering it in an academic setting, and the particular Lisp language they were using was designed for academic purposes. Scheme, in particular, came out of academia, and that focus has often made it more difficult to use for real-world applications. Clojure, on the other hand, is a very pragmatic language. It was designed by a career programmer, Rich Hickey, who wanted to solve real problems that he had with other programming languages. So if you can get over the initial similarities with Lisps that you might have seen before, I think you'll find that Clojure is a very productive language to work with.

    Is Clojure being used in large-scale production?

    Stuart Sierra: Yes, there are many "success stories" listed on the Clojure website and more on our wiki. I work for a company, Relevance, that does consulting for enterprise software, and we've had several client projects written in Clojure that have been quite successful. It's also been used in production by Akamai, financial companies, big data companies that are doing large data processing, and in a variety of other places.

    The Java Virtual Machine (JVM) seems to be the language platform of choice for developing languages. What makes the JVM so attractive, and how does Clojure leverage it?

    Stuart Sierra: The JVM makes an excellent platform for language development, and that's why we've seen so many new languages show up. It's actually much better as a target for languages than some lower-level assembly-type languages for the simple reason that it provides a way for all of those different languages to interoperate. So now you can have Clojure code, JRuby code, Scala code and Java code all making use of the same libraries and all calling back and forth to one another. So a Java method call in Clojure looks like a Clojure function call, but you're calling a Java method call by name. That compiles to a normal Java method call that has all of the performance benefits that the JVM can provide. You can define classes in Clojure that will be available to Java programs and accessible to Java programmers through the standard calling conventions that they're used to. All of Clojure's built-in data types implement the standard Java collection interfaces, so they can be passed directly into Java code without changes.

    What's the IDE situation for Clojure?

    Stuart Sierra: IDE support is evolving. There are plug-ins available now in varying states of sophistication for all of the major IDEs: Eclipse, IntelliJ, NetBeans. A lot of people are also using Emacs or another sophisticated text editor with good support for writing Lisp-like languages.


    June 24 2011

    Big data and open source unlock genetic secrets

    Replicating Nanomachines by jurvetson, on FlickrThe world is experiencing an unprecedented data deluge, a reality that my colleague Edd Dumbill described as another "industrial revolution" at February's Strata Conference. Many sectors of the global economy are waking up to the need to use data as a strategic resource, whether in media, medicine, or moving trucks. Open data has been a major focus of Gov 2.0, as federal and state governments move forward with creating new online platforms for open government data.

    The explosion of data requires new tools and management strategies. These new approaches include more than technical evolution, as a recent conversation with Charlie Quinn, director of data integration technologies at the Benaroya Research Institute, revealed: they involve cultural changes that create greater value by sharing data between institutions. In Quinn's field, genomics, big data is far from a buzzword, with scanned sequences now rating on the terabyte scale.

    In the interview below, Quinn shares insights about applying open source to data management and combining public data with experimental data. You can hear more about open data and open source in advancing personalized medicine from Quinn at the upcoming OSCON Conference.

    How did you become involved in data science?

    Charlie QuinnCharlie Quinn: I got into the field through a friend of mine. I had been doing data mining for fraud on credit cards and the principal investigator, who I work with now, was going to work in Texas. We had a novel idea that to build the tools for researchers, we should hire software people. What had happened in the past was you had bioinformaticians writing scripts. They found the programs that they needed did about 80% of what they wanted, and they had a hard time gaining the last 20%. So we had had a talk way back when saying, "if you really want proper software tools, you ought to hire software people to build them for you." He called my boss to come on down and take a look. I did, and the rest is history.

    You've said that there's a "data explosion" in genomics research. What do you mean? What does this mean for your field?

    Charlie Quinn: It's like the difference between analog and digital technology. The amount of data you'd have with analog is still substantial, but as we move toward digital, it grows exponentially. If we're looking at technology in gene expression values, which is what we've been focusing on in genomics, it's about a gigabyte per scan. As we move into doing targeted RNA sequencing, or even high frequency sequencing, if you take the raw output from the sequence, you're looking at terabytes per scan. It's orders of magnitude more data.

    What that means from a practical perspective is there's more data being generated than just for your request. There's more data being generated than a single researcher could possibly ever hope to get their head wrapped around. Where the data explosion becomes interesting is how we engage researchers to take data they're generating and share it with others, so that we can reuse data, and other people might be able to find something interesting in it.

    Health IT at OSCON 2011 — The conjunction of open source and open data with health technology promises to improve creaking infrastructure and give greater control and engagement for patients. These topics will be explored in the healthcare track at OSCON (July 25-29 in Portland, Ore.)

    Save 20% on registration with the code OS11RAD

    What are the tools you're using to organize and make sense of all that data?

    Charlie Quinn: A lot of it's been homegrown so far, which is a bit of an issue as you start to integrate with other organizations because everybody seems to have their own homegrown system. There's an open source group in Seattle called Lab Key, which a lot of people have started to use. We're taking another look at them to see if we might be able to use some of their technology to help us move forward in organizing the backend. A lot of this is so new. It's hard to keep up with where we're at and quite often, we're outpacing it. It's a question of homegrown and integrating with other applications as we can.

    How does open source relate to that work?

    Charlie Quinn: We try and use open source as much as we can. We try and contribute back where we can. We haven't been contributing back anywhere near as much as we'd like to, but we're going to try and get into that more.

    We're huge proponents not only of open source, but of open data. What we've been doing is going around and trying to convince people that we understand they have to keep data private up to a certain point, but let's try and release as much data as we can as early as we can.

    When we go back to talking about the explosion of data, if we're looking at Gene X and we happen to see something that might be interesting on Y or Z, we can post a quick discovery note or a short blurb. In that way, you're trying to push ideas out and take the data behind those ideas and make it public. That's where I think we're going to get traction: trying to share data earlier rather than later.

    At OSCON, you'll talk about how experimental data combines with public data. When did you start folding the two together?

    Charlie Quinn: We've been playing with it for a while. What we're hoping to do is make more of it public, now that we're getting the institutional support for it. Years ago, we went and indexed all of the abstracts at Pubnet by gene so that when people went to a text engine, you could type in your query and you would get a list of genes, as opposed to a list of articles. That helped researchers find what they were looking for — and that's just leveraging openly available data. Now, with NIH's mandate for more people to publish their results back into repositories, we're downloading that data and combining it with the data we have internally. Now, as we go across a project or across a disease trying to find how a gene is acting or how a protein is acting, it's just giving us a bigger dataset to work with.

    What are some of the challenges you've encountered in your work?

    Charlie Quinn: The issues we've had are with the quality of the datasets in the public repositories. You need to hire a curator to validate if the data is going to be usable or not, to make sure it's comparable to the data that we want to use it with.

    What's the future of open data in research and personalized medicine?

    Charlie Quinn: We're going to be seeing multiple tiers of data sharing. In the long run, you've going to have very well curated public repositories of data. We're a fair ways away from there in reality because there's still a lot of inertia against doing that within the research community. The half-step to get there will be large project consortiums where we start sharing data inter-institutionally. As people get more comfortable with that, we'll be able to open it up to a wider audience.

    This interview was edited and condensed.

    Photo: Replicating Nanomachines by jurvetson, on Flickr


    Older posts are this way If this message doesn't go away, click anywhere on the page to continue loading posts.
    Could not load more posts
    Maybe Soup is currently being updated? I'll try again automatically in a few seconds...
    Just a second, loading more posts...
    You've reached the end.
    No Soup for you

    Don't be the product, buy the product!

    YES, I want to SOUP ●UP for ...