Newer posts are loading.
You are at the newest post.
Click here to check if anything new just came in.

September 08 2010

Sensor networks and the future of forecasting

We can't control the climate, but are there ways to mitigate and avoid the negative effects extreme weather brings? I believe the starting point for potential solutions lies in data acquisition and environmental sensor networks (ESN).

Current technologies and sensors, ranging from cell phones to satellites, allow a "global environmental cyberinfrastructure" to be more than a topic for discussion at academic conferences. Researchers have studied connections and system interactions for some time, but now a broader segment of society is becoming aware of the precarious relationship between weather, climate and humanity. This awareness is sometimes motivated by the need to help. Other times there's a profit incentive. The reason doesn't matter if the result is a better low-cost global sensor network that can be tapped by anyone with a signal.

A systems approach to identifying natural hazards, coupled with a communications framework that can easily make data available to the public, is the crucial cornerstone of a functional environmental sensor network. The global monitoring of short- and long-range weather patterns and the linking of sensor-network data could allow forecasters to identify potential problems before they manifest.

The weather link between Russia and Pakistan

Weather has caused great disruption to many lives in both Russia and Pakistan in recent months. While these are separate circumstances, they share common physical factors. The following is a look at how events in one part of the world influence weather elsewhere.

The Russian heat wave

Global wheat prices spiked in early August. Much of that activity stemmed from potential crop losses in Russia, and it was helped by ubiquitous stories of parched fields and decimated crops. To be fair, part of the price spike came on the heels of a Russian export ban. Nonetheless, this story's origin is tied to weather.

The two maps below show the monthly year-over-year (2010 vs. 2009) changes in maximum temperature and precipitation for July. This weather was known to people in the agricultural and weather communities. The stage was set long ago for potential problems in western Russia.

(Click to enlarge)

(Click to enlarge)

A persistent high-pressure system centered over eastern Europe has combined with changes in the jet stream's pattern to keep temperatures high. When this high is as entrenched as it has been, it serves to do two things:

  1. It diverts a jet stream that would normally steer cooler air into parts of central Russia and northeastern Africa.
  2. It blocks moist air from the southeast, which exacerbates the dryness. This is a reason why parts of central Africa are seeing better moisture in recent months.

  3. There's an active west-to-east jet stream that travels above western Russia. This stream typically exhibits a seasonal shift to the east, and in the process allows moister air from the west to migrate into the region. The jet did not shift in July, and the result was a prolonged period of moisture-free air. When this combines with a strong high, the region experiences weather like it's seen over the last month. And when there is a high-pressure system in one region, there is often a corresponding low elsewhere.

    Flooding in Pakistan

    The low in this case has been over the mountainous region of northern Pakistan. This cold low has been the catalyst for a good portion of the excess rains. So, while located in distinct climate zones, the heat in Russia has a connection to the floods in Pakistan.

    But there's more to this puzzle. Every year, the annual Indian monsoon is anticipated throughout India and Pakistan, as much of the commercial activity that takes place in both countries is agrarian in nature. The monsoon was deficient in 2009, leading to short crops in many sectors. The arrival of the rainy season this year carried a heightened importance.

    The onset of the 2010 monsoon was healthy and most regions have been receiving beneficial moisture totals. But the placement of another area of high pressure over northeast India has, thus far, kept India's northern states dry. In the process, this high has been diverting even more moisture, which flows from southeast to northwest into central/western India and along into Pakistan.

    The first map below from NOAA's Earth System Research Laboratory depicts the storm tracks from the last week (as viewed via anomalies in outgoing longwave radiation), where the excess moisture is visible directly over Pakistan. The second map below, from the NOAA Climate Prediction Center, highlights wind anomalies over the last week. We see from this map that the stronger winds originating from the southeast were actively driving the moisture into areas that needed it the least.

    (Click to enlarge)

    (Click to enlarge)

    Pakistan's drainage infrastructure, which is silt-laden, has made the rainfall situation worse. When excess rains fall, it takes much longer to drain than necessary.

    Unfortunately, any excess rains in the coming weeks will likely be met with more problems for civilians. As of mid-August, estimates put flood-related deaths above 1,300, and more than 15 million people have been affected by the floods. Both of these figures are expected to rise.

    Satellites and risk management

    Could these separate-but-related crises have been foreseen? If so, what measures could have been initiated to mitigate some of the fallout?

    The map below is an indicator of vegetation health as derived via satellite for the wheat regions to the north of the Caspian and Black Seas. This particular graphic depicts the Normalized Difference Vegetation Index (NDVI) for the region, shown as an anomaly vs. the five-year average (for mid-to-late July). The index assigns a value for crop health, and based on the color scale shown in the legend, it is clear that the region has been exhibiting severe vegetation stress.

    The important thing to note here is that wheat prices started to increase in July, then exhibited a violent spike in early August. As this map is from mid-to-late July, we can see that by using tools such as satellite indices in conjunction with a long-range weather forecast, the current impact in wheat prices and ensuing financial turmoil could have been anticipated. To a certain degree, it could have been mitigated through a proactive physically-based risk management strategy. In addition, the same-satellite derived images that were capturing the movement of monsoon rains across India's agricultural regions could have been used to view the excess moisture in regions where the Pakistan floods originated.

    We can, of course, explore the questions surrounding how much lead time is necessary to avoid a crisis (remember Katrina). Nonetheless, it is clear that many were not aware of these disastrous systems until it was too late.

    (Click to enlarge)

    More coming soon

    This column is a starting point for discussions that examine climate, weather, sensors, networks, and their influence on society. The last couple of years have seen a heightened interest in this area from the research side, evidenced by new topics and sessions presented at the annual conferences of the American Geophysical Union, the American Meteorological Society, and others. I hope to bring some of these discussions to a broader audience while also helping readers understand how closely related their lives and decisions are to weather and climate.


September 07 2010

Four short links: 7 September 2010

  1. GalaxyZoo for Climate Science? -- GalaxyZoo is the crowdsourced physics research. A group of climate scientists want the same, to help predict "weather events". See also the Guardian article. (via adw_tweets on Twitter)
  2. Crispian's Science Map -- gorgeous Underground-style map showing scientists and their contributions. (via arjenlentz on Twitter)
  3. Programming Things I Wish I Knew Earlier (Ted Dziuba) -- opinionated piece, but boils down to "keep it simple until you can't", and "the more you know about the actual hardware, the better you can code". With EC2, when Amazon says "I/O performance: High", what does that even mean? Is that suitable for a heavy random read scenario? (via Hacker News)
  4. The Molecular Biology Carnival, 2ed -- collection of excellent blog writing about molecular biology. (via BioinfoTools on Twitter)

August 10 2010

Data as a climate change agent

Amidst varied hopes for open data and open government, enabling better data-driven decisions in both the private and public sector rank high. One of the existential challenges for humanity will be addressing climate change, particularly in countries where scientific resources are scant or even non-existent.

In February, the Obama administration NOAA) provides weather information. Earlier this summer, the Center for Strategic and International Studies (CSIS) published new research, "Earth Observation for Climate Change," and hosted a forum on leveraging climate data services to manage climate change. The video from the forum is embedded below:

"The vision is an informed society anticipating and responding to climate and its impacts," said Thomas R. Karl, director of the National Climatic Data Center at NOAA.

The dawn of climate data services

Gov 2.0 Summit, 2010According to the CSIS report, the September 2009 meeting of the World Climate Conference agreed to establish the Global Framework for Climate Services (GFCS) to connect research to policy making. The framework has four components:

  1. Observation and modeling
  2. Research and modeling
  3. Climate services information system
  4. A user interface (UI) program

When combined, that information system and UI would constitute a "World Climate Service System." According to the report:

"The goal of the new service will be to better inform decision makers (particularly in less-developed nations) by supplying data and analyses on climate change. When it is finally implemented, the World Climate Service System will provide climate and earth observations, models and forecasts to provide critical climate data to governments and other users around the world."

For instance, Karl pointed to Devil's Lake in North Dakota's expansion over recent decades. "The governor and mayor have asked for help," he said. "They have to make investments in roads and bridges. How should we do that? What can we say about the causes?"

Climate data services could serve a number of societal needs, said Karl, including better understanding of coastal inundation, changes in storm intensity, wave heights, drought conditions, or how extreme events might change as climate warms. There are also ancillary benefits to addressing societal challenges, said Karl, including identifying infrastructure issues or gaps in core capabilities, which would benefit the energy, transportation, agriculture and health sectors. The NOAA Office of Programming Planning and Implementation is soliciting input on its proposals, including an Ideascale instance.

Policy makers will need several categories of data to make better decisions, according to the report:

  • Trend data, like changes in forest size, gases in the atmosphere, or ocean currents.
  • Regional data, to identify specific issues in a smaller geographic area.
  • Effects assessment data, to measure the efficacy of mitigation or adaptation policies.
  • Compliance data, to monitor progress in a given agreement or treaty.
  • Planning data, to provide information that insurance companies, urban planners, corporations and others need to reduce risk or uncertainty.

Space policy and carbon-sensing satellites

The CSIS report also put a focus on space policy, specifically a "shortage of satellites actually designed and in orbit to measure climate change." Satellites have been used to measure pollution, or in the case of NASA's ICESat satellite, provide laser altimetry to measure the rate of melting in the Arctic ice. The crash of NASA's Orbiting Carbon Observation (OCO) satellite in 2009 "left the world essentially bereft of the ability to make precise measurements to assess emissions reductions effort," according to the CSIS report. That means climate scientists are relying on Japan's GOSAT, the European Space Agency's SCIAMACHY and Canada's CanX-2 satellites until OCO 2 gets up into orbit in February 2013. All of these current systems lack the advanced sensors or monitoring capabilities scientists desire to assess changes in the carbon cycle. The number of earth observation instruments has actually declined in recent years, as shown on the figure below:


Visualizing Climate Data

Many citizens will not find the NOAA Climate Services Interactive Map to be a terrific interface to gain insight, although scientists can find datasets relatively easily. Fortunately, NOAA has launched a prototype of a climate services portal,, which is a vast improvement on more traditional .gov websites. The site includes an online magazine, access to climate data and services, a section on understanding climate science, education and news.


The portal appears to be a rare beast in government IT: a public prototype. Some govies might even call it a government 2.0 beta.

Opening data for innovation

Will open climate data be available to civic developers and commercial concerns to build businesses upon?

"We invest a lot in terms of making data available with an open data policy, so everyone can see what everybody is doing" said Dr. Jack Kaye, associate director for research and analysis in NASA's Earth Science Division. "The sheer volume of data and complexity of it makes it a challenge for less sophisticated users. One of the challenges is to create tools that will facilitate the less knowledgeable user."

That's one area where is currently succeeding, in terms of providing a clear "climate dashboard," pictured to the right, that shows the progression of climate change since the late 19th century. As any IT executive knows, however, a dashboard is only as useful as the data feeding it. That's a concern that's been highlighted in the Climategate controversy over the past year. Despite that concern, there is reason for optimism behind seeing open climate data published online, where it can be exposed to more transparent vetting.

The story of how weather data provisions information and news outlets may be well worn in the Gov 2.0 dialogue, though most citizens don't think about NOAA data underpinning serious decisions on business, travel or recreation.

Thanks to "infovegan" Clay Johnson, the history of how weather data was opened is clearer today. The history of GPS shows the innovation and value spawned by the release of global positioning system data. As Time reported last year, a market-research firm estimated the global GPS< market will total $75 billion by 2013.

Earlier this spring, the United States released community health information to provision healthcare apps and drive better policy. Now, scientists and policy makers will explore the potential for climate data services to inform citizens and government, enabling both to make better decisions for communities and businesses alike.


The Gov 2.0 Summit will be held Sept. 7-8 in Washington, D.C. Learn more and request an invitation.

Older posts are this way If this message doesn't go away, click anywhere on the page to continue loading posts.
Could not load more posts
Maybe Soup is currently being updated? I'll try again automatically in a few seconds...
Just a second, loading more posts...
You've reached the end.
No Soup for you

Don't be the product, buy the product!

YES, I want to SOUP ●UP for ...