Newer posts are loading.
You are at the newest post.
Click here to check if anything new just came in.

December 27 2012

Four short links: 27 December 2012

  1. Improving the Security Posture of Industrial Control Systems (NSA) — common-sense that owners of ICS should already be doing, but which (because it comes from the NSA) hopefully they’ll listen to. See also Wired article on NSA targeting domestic SCADA systems.
  2. Geographic Pricing Online (Wall Street) — Staples, Discover Financial Services, Rosetta Stone, and Home Depot offer discounts if you’re close to a competitor, higher prices otherwise. [U]sing geography as a pricing tool can also reinforce patterns that e-commerce had promised to erase: prices that are higher in areas with less competition, including rural or poor areas. It diminishes the Internet’s role as an equalizer.
  3. Hacker Scouting (NPR) — teaching kids to be safe and competent in the world of technology, just as traditional scouting teaches them to be safe and competent in the world of nature.
  4. pressureNET Data Visualization — open source barometric data-gathering software which runs on Android devices. Source is on GitHub.

July 25 2012

Share your art: readers' works about weather

Perhaps unsurprisingly, rain makes a big splash in this month's readers' Share your art gallery. But you can also expect windy patches, the weather's effect on nature and the promise of warmth

June 13 2012

Share your art: weather

Rain, wind, sunshine – this spring has had it all, which makes weather the perfect theme for this month's Share your art

It may be raining outside, and we may still need our winter coats, but believe it or not, it's June! Welcome to this month's Share your art project.

I'd like to thank everyone who submitted artworks to May's project. The theme, picked by Share your art – let's shorten this to SYA – community member Elizabeth Barthory, who suggested Britain as our theme. You shared an array of wonderful works, including a workers digging up a road, closed down shops and picturesque landscapes. We created a gallery of some of our favourites, but you can scroll through the complete set of SYA artworks from all our projects in our online readers' gallery.

On to this month's project. After putting out a call out on Twitter for topic suggestions, @HollyBrodieArt replied:

Good choice. Weather it is. As always, your interpretation of the theme can be as literal or abstract as you like. It can be created in any medium but photography. If you are a photographer and would like to share your work, please visit Camera club.

This is a new series and I'm keen to hear your suggestions about how it can be improved or ideas for future themes. Please leave your comments in the thread below.

How to send us your artworks

Send an email to with "Share your art" in the subject line. Please include your full name so we can credit you properly, and the title of your work. Attach a high resolution .jpeg or .tiff photo file of your art to the email (maximum file size 20mb). A selection of your pictures will displayed in our online gallery.

We'll feature some of our favourite pictures from the group on By sending us your pictures you a) acknowledge that you have created the pictures or have permission to do so; and b) grant us a non-exclusive, worldwide, free licence to publish your pictures as described. Copyright resides with you and you may reuse your pictures however you wish. Full terms and conditions. © 2012 Guardian News and Media Limited or its affiliated companies. All rights reserved. | Use of this content is subject to our Terms & Conditions | More Feeds

November 27 2011

Weatherwatch: Original Crystal Palace viewed as a Tower of Babel

The Crystal Palace, built in Hyde Park to house the Great Exhibition of 1851, was six times larger than St Paul's Cathedral. Invitations to the crowned heads of Europe to visit were initially refused on the grounds of safety. The king of Hanover wrote to the king of Prussia that "countless hordes of desperate proletarians" were travelling to London to assassinate him and also that the structure would crash to the ground "cutting the crowds to ribbons." Prince Albert used sarcasm to reply "That mathematicians have calculated that the Crystal Palace will blow down in the first gale ... doctors that owing to so many races coming together the Black Death would make its appearance, theologians that this second Tower of Babel would draw on it the vengeance of an offended God", but assured the two kings that he and Queen Victoria would be going anyway.

In fact there were safety fears. The Astronomer Royal predicted that as had happened with some bridges the large crowd inside would set up a resonance and the whole building would vibrate itself to pieces. To settle the matter 300 workmen, and then the army were called in to march up and down and even jump in unison. Nothing moved. After the exhibition the palace was rebuilt in Sydenham, lasting until 30 November, 1936. Ironically it was the strong wind fanning a small fire that destroyed it. The blaze spread so fast it defeated 80 fire crews; the glow could be seen from Brighton. © 2011 Guardian News and Media Limited or its affiliated companies. All rights reserved. | Use of this content is subject to our Terms & Conditions | More Feeds

September 13 2011

A new look for weather data

WeatherSpark is tapping into a variety of datasets to deliver a different level of weather engagement. The new website, which provides data from more than 4,000 weather stations, lets you interact with full-screen weather graphs to investigate current forecasts and historical weather patterns.

In the interview below, I talk with WeatherSpark co-founder Jacob Norda (@jacobnorda) about his company's approach to weather data and visualizations.

WeatherSpark full screen forecast and historical trends
In addition to forecast information (left), WeatherSpark also offers access to historical trend data (right). (Click to enlarge.)

What problems with traditional weather information are you trying to solve?

Jacob Norda: Most weather websites present weather data using tables with numbers and icons, and they show maps in very small viewports. This makes it hard to get an overview, and it typically requires a lot of page views to find the relevant information.

We wanted to address these shortcomings by using an integrated dashboard with a powerful map and graph view. This allows for an overall weather impression — "it's raining nearby," "today will be a hot/cold day" — the ability to look for very specific information — "It'll be 70F at 2pm" — as well as an intuitive way to get to historical weather and average weather information, which is not readily available.

We decided to show the historical information because we think it's interesting. People oftentimes say things like, "last summer was unusually hot/cold/wet," and we wanted to provide a way to actually look that up. That information also powers the averages, which we had to have, so it would have been a missed opportunity to not make it available.

In addition, technical restrictions make it so other websites can only show radar animations over very short periods of time, typically two hours. We've solved these technical issues, and that means we can offer full-screen radar playback spanning several days. That allows for things like radar animations of hurricanes.

Where does WeatherSpark get its data?

Jacob Norda: We get the forecasts from a number of sources, including the National Oceanic and Atmospheric Administration (NOAA), the Norwegian Meteorological Institute, World Weather Online, and Weather Central. The sources are noted on the about page. The historical data comes from a number of governmental and non-governmental data sources, primarily NOAA.

How can weather data be improved?

Jacob Norda: There's a wild variety in file formats, both for historical and forecast information. The information would be more easily used if this data were somehow unified. However, removing old formats or APIs would break legacy systems, so we don't envision the current sources doing that. We're considering offering a unified API, but it's in the pre-roadmap stage at this point.

This interview was edited and condensed.

Strata Conference New York 2011, being held Sept. 22-23, covers the latest and best tools and technologies for data science — from gathering, cleaning, analyzing, and storing data to communicating data intelligence effectively.

Save 30% on registration with the code ORM30


August 31 2011

New tools and techniques for applying climate data

Upon entering the New York Academy of Sciences (NYAS) foyer, guests are greeted by a huge bust of Darwin, along with wonderful preservations and replicas of samples of his early works adorning the walls. While Darwin revolutionized science with curiosity and the powers of observation, who knows what he could have accomplished with the informatics and computational resources that are available to scientists today?

It was fitting last Friday that the NYAS held their First International Workshop on Climate Informatics at their downtown offices on a beautiful day when everyone seemed to be dodging the city in advance of Hurricane Irene. Aside from being a wonderful venue to hold a workshop — I enjoyed reading the pages of Darwin's "Descent of Man" writings on the wall — the discussions gave me much food for thought.

As with any small conference in a single-speaker setting, the majority of talks were good, covering the range of climate data and statistical methods and applications. And as is often the case, I was more impressed with the talks that addressed topics outside of my disciplines, particularly the machine learning discussion provided by Arindam Banerjee of the University of Minnesota.

But the highlight came during the breakout sessions, which provided in-depth discussions surrounding the challenges and opportunities in applying new methods to climate data management and analysis. Topics ranged from multiple-petabyte data management issues faced by paleoclimatologists to management and manipulation of large datasets associated with global climate modeling and Earth Observation (EO) technologies.

Overall, the workshop showed that we're seeing the early confluence of two communities: climate scientists looking for new tools and techniques are on one side, data scientists and statisticians looking for new problems to tackle are on the other.

Data poor to data rich

One of the event's more interesting side notes came from a breakout session where we explored the transition from being a data-poor field to a data-rich field. As an applied scientist, I certainly would say that climate researchers have been blessed with more data, both spatially and temporally. While the days of stitching various datasets together to test an idea may be behind us, the main issues tend to come down to scale. Is global coverage at 4KM resolution good enough for satellite observations? Can we build a robust model with data at this scale? Do interpolation methods for precipitation and temperature work across various physiographic environments?

While more data helps alleviate some of the scientific challenges we have faced in the past, it also raises more questions. Further, each year of global observations builds the database of reanalysis data — as an example, look at the reanalysis data that's part of the MERRA maintained at NASA's Goddard Space Flight Center.

That said, I'll default to the position that too much data is a good problem to have.

Path forward for the data community

The timing of this event was also useful for another reason. The upcoming Strata Summit in New York will bring together data scientists and others in the data domain to address the challenges and strategies this growing community faces. I'll be giving a talk on new ways to collect, generate and apply atmospheric and oceanic data in a decision-making context under the rubric of atmospheric analytics. In addition to the talk, I'm eager to learn how I can better utilize the data I'm working with as well as bring back some new tools to share with my colleagues in other fields who may face similar big data challenges.

Strata Conference New York 2011, being held Sept. 22-23, covers the latest and best tools and technologies for data science — from gathering, cleaning, analyzing, and storing data to communicating data intelligence effectively.

Save 30% on registration with the code ORM30


April 28 2011

Interest in renewable energy could benefit data services

Google noted that a recent investment of $100 million will assist in enabling the Shepherds Flat Wind Farm to become the world's largest wind farm by 2012. This follows Google's $168 million solar investment into the BrightSource Energy tower project. The company has now invested nearly $350 million into clean energy to date.

Such investments from non-traditional cleantech investors are starting to receive more attention. The sustainable increase of large-scale infrastructure investments in the alternative energy sector will likely be accompanied by a rise in the demand for data-driven services that can help optimize efficiency of the related operational costs.

Enter the growing need for timely and accurate weather data. Last month I touched upon the potential for the weather services sector to contribute their expertise to the smart grid arena. Demand anticipation, efficient raw material utilization, baseload and peak usage forecasting, logistics planning — these are just a few of the many areas where atmospheric analytics can contribute to this growing global market. More frequent and more precise weather data can help utilities anticipate demand surges, and in the process reduce both unnecessary expenditures and unnecessary emissions. Such supporting weather data is not just limited to the network of government-maintained observation stations — cheap ubiquitous sensors can be placed just about anywhere, and granular data that can help make a decision more efficient translates to more streamlined raw material procurement and utilization, not to mention lower costs passed on to the consumer.

At many energy conferences of late containing the cleantech theme (look at the Green:Net event event sponsored by GigaOm), there has been a lot of talk around the benefits of "smart" meters and "smart" algorithms, which will in part be used to help transform the energy infrastructure. These tools and techniques can only truly be considered smart if they are embedded with ambient data feeds that can supply accurate data streams, which can be developed into weather-driven efficiency algorithms (largely based upon persistence). The resultant algorithms can then help to enable the energy management systems to operate in sync with their surroundings — in essence, becoming smarter.

As short-range demand anticipation models are largely based on a set of standard assumptions, there will be limited human involvement once a system is constructed (as long as good input data is available), and this will fit in nicely with the functionality associated with automated demand response systems. WeatherTrends360 provides examples of granular hourly weather data feeds and displays that can be embedded into such systems (see chart below). While the cornerstone of applied modeling (garbage in = garbage out) is implicit, it should be noted that a smart algorithm will only be as good as the data upon which it has been trained.

Weather chart

As the shift toward increasing the share that renewables make up in the total energy-generating matrix gathers momentum, the need for data services providing temperature, wind, and solar analytics will strengthen. Look for innovative ways in which the weather industry, including both data providers and forecasters, can generate new sources of returns in this space, as the industry evolves.

January 11 2011

La Nina and global commodities

In the weather and climate community, 2010 will be remembered as a year where the strong La Nina pattern exerted a significant influence on global agricultural production, with weather extremes hitting key commercial producing regions across a number of sectors.

Picture 1.png

The Southern Oscillation index (SOI), a measure of El Nino/La Nina strength and duration, was strongly positive over the last half of the year, and in fact this may be the strongest La Nina that we have seen since the 1973/74 event. The figure below highlights the intensification of the La Nina over the course of the year. The numbers on the perimeter show the day of the year, and the SOI is represented by the solid line; we can see that at the start of 2010, the SOI was in negative phase (-10.1, -14.5 & -10.6 for Jan, February, and Mar 2010 respectively), but then a strong shift occurred during the El Nino - La Nina transition, and the year finished at nearly +27. Further, as the map above shows, the current equatorial Pacific Ocean Sea Surface Temperature (SST) anomalies are still negative, and while some La Nina indicators seem to be approaching a peak and then a return to neutral phase, the current event is still certainly not over.

Picture 3.png

Statistically, there are certain types of patterns that we can associate with a given La Nina or El Nino year. However, there is no typical event where all expected seasonal weather outcomes manifest themselves, so using this approach, or relying on analog years when attempting to identify potential seasonal impacts, can be dangerous. Disclaimer notwithstanding, there are some general relationships that tend to hold up that are highlighted in this map from NOAA's National Climatic Data Center. Some of seasonal relationships did verify, and also led to some of the food stories that are getting an increased amount of attention by analysts and traders at the start of the year. A few examples include: dryness in Brazil & Argentina (wheat), increased precpitation and flooding in eastern Australia (sugarcane), and a wetter summer pattern across the Indian subcontinent (sugarcane, pulses).

Picture 4.png

In recent months, we have seen agriculture commodity prices exhibit a sharp rise in everything from coffee to natural rubber, and these higher prices have started to impact the margins of the producers, who more often than not, pass this along to the consumer. In an interesting recent article in the Financial Times (paid registration required), the author noted the following price changes (in %) of agriculture and energy commodities since January 1, 2010:

Picture 6.png

Remember that weather, particularly excess precipitation, will also affect the production cycle in the natural resources sector. As such, while impacts to agricultural interests are usually the first associated with an acute or extended adverse weather pattern, the energy and mineral sectors have weather exposure also. Further, with saturated fields and subsurface conditions present across much of the Queensland mining territories, there will not be a quick return to a normal production schedule after the heavy rains subside. With the potential financial implications (and opportunities) abound as the mining sector deals with the effects of this pattern, we are seeing this concern reflected by the types of inquiries received within our own customer base at Weather Trends. Most of our clients in the commodity sector aim to act upon the relationship between weather and commodities in the "traditional" sectors of energy and agriculture, but in recent months we have seen a marked increase in inquiries for forecasts and models relating the pattern to the broader natural resource extraction industries.

In a recent statement by the UN Food and Agriculture Organization, officials hinted at the potential for high prices to serve as a catalyst for food riots. As recently as 2007/08, riots broke out in three dozen countries across Asia, Africa and Latin America as a response to a strong and sustained spike in prices for staples including wheat, rice and oilseeds. However, for many food categories, the relative prices in this current regime have not equaled the highs seen three years ago. Also, while the negative impacts seem to get the most attention, growers in many origins are seeing better production and yields than they were during the season leading up to the riots.

So at the beginning of 2011, with our forecast in place, we are now developing and refining our commodity supply expectations for the year. Will the La Nina pattern remain in place to affect spring plantings and emergence for North America? If the pattern shifts back toward El Nino later in the year, what can we expect for Australia/AsiaPac? And how might a transition effect the onset of the Indian Monsoon? Finally, what do all of these factors mean for commodity prices in 2011? These are only a few of the many questions that we are attempting to quantify and provide to those with a weather risk to their operations or their portfolio.

For those who will be attending the American Meteorological Society meeting in Seattle in a couple of weeks, I will be discussing some of these topics in greater detail.

December 20 2010

Light fantastic

As we approached the shortest day of the year, the weak midday light mimicked the colours of Caspar David Friedrich's palette

Today the sun shone, a burnished wintry gold in the middle of the day. So close to the shortest day of the year, even at noon there was a poetic and almost twilit quality to it that would have been beautiful under any circumstances. Hitting the crystalline white snow on trees and in parks, this weak light took on a glorious richness. Innumerable winter shades of blue, silver and bronze illuminated the city.

Snow truly is a magical sight, however harsh, however inconvenient this week's conditions. If art criticism has a value in this world surely it is to draw attention to visual glories that are easy to take for granted. Actually, snow is not usually taken for granted. It is in everyone's dreams of winter wonderland. But the relentless hard weather this winter and the trials facing travellers this week kind of grind you down. Magic? More like a pain ... So let's take just one moment among the news of bleak weather to acknowledge how grand and sublime snow can be.

So as not to offend anyone suffering as a result of the snow, I will take as my image a painting that is anything but merry. Caspar David Friedrich's Winter Landscape, painted in about 1811, shows a lame man out alone in a snowbound wilderness. He has put aside his crutches and sits in the snow to contemplate a mountain shrine that depicts Christ crucified. Beyond the great fir trees looms a fairytale castle or cathedral, a fantastic structure of vertigo-inducing pinnacles become a gothic silhouette in the violet light. For Friedrich captures the mystical light effects the short days of midwinter can create in a snowbound landscape: the kinds of light we have seen this week in Britain.

Friedrich's winter wonderland is a serious place. The fantastic castle may represent the kingdom of heaven. If so the pilgrim will soon get there, for having abandoned his crutches he has clearly sat down to die of cold at the feet of the Christ of the mountains. In the bleak midwinter, the eerie light that fills this painting promises redemption. Sun, snow and Friedrich are nature's boldest artists. © Guardian News & Media Limited 2010 | Use of this content is subject to our Terms & Conditions | More Feeds

December 06 2010

An ensemble approach to weather forecasting

During President Obama's recent visit with officials in India, one of the more interesting topics discussed addressed potential partnerships between the Indian Meteorological Department and U.S.-based forecasting agencies and corporations. I was asked for my opinions on this topic by a New Delhi-based correspondent. Below you'll find my abbreviated reply:

The primary beneficiary will first become apparent for end users in India, but the gains will extend beyond the borders to virtually any country/entity with a financial or social exposure to raw material prices.

Science progresses through the open exchange of information, and while on one hand the applied benefits of scientific research and the resultant applications can be commercialized and therefore protected, the open nature of collaborative agreements such as the India/U.S. item discussed can also be a great benefit to a much larger segment of society.

Many public and private weather groups have developed and refined techniques to develop monthly and seasonal weather forecasts. Forecasting, and in particular long-range forecasting, often relies on a blend of common physical/fluid dynamic principles coupled with a variety of closely guarded mathematical approaches. As such, while the basic scientific principles underlying the development of a forecast may be generally the same regardless of the source of the forecast, each public or private forecasting group puts their own spin on the forecast, where they try to separate themselves from their competition.

However, there is strength in numbers. So from the point of view of the end user, a forecast can gain a higher confidence if it is in more agreement with other reputable forecasts.

For many years, the Indian Met Department (IMD) was the only source for the agricultural sector when it came time to develop plans around the annual monsoon. For growers, these decisions include seed variety, plant/harvest dates, quantity of pesticides, etc., which carry a heavy financial burden, and oftentimes the decisions are made based on one (IMD) forecast. In recent years, the IMD has had a less than spectacular record in their seasonal rainfall forecast during the monsoon, so this serves as a time where other approaches can and should be considered.

The result of a potential collaboration via partnering with NOAA or other non-Indian weather groups can only enhance the IMD's own forecasting process. By exchanging some methods, the IMD can learn about where the strengths and weaknesses lie within their own methodology. In the long term this is a very successful methodology.

The result will then be a better forecast for not only the IMD's primary customers (the agribiz community), but others who also have an exposure to fluctuation in prices of important raw materials sourced from India. Further, better forecasting and monitoring techniques that are jointly developed will serve to provide more price transparency in the futures markets of related commodities, and minimizing price volatility is good for both producer and consumer.

There is no single forecast group than can develop a long range weather outlook that is correct 100 percent of the time. Taking an "ensemble approach," where results from several different forecasts are used to guide and continuously refine a seasonal outlook, is a safer way to approach the weather risk associated with an upcoming season.


November 11 2010

Growing new data streams

A pleasantly surprising revelation came to me during the Agriculture Outlook Americas conference, which was held in rainy Boston this week.

The global agricultural community is comprised of big agribiz (think Cargill, Deere & ADM), family farms in Mato Grosso, bankers/hedge funds (many of which have no clue how things grow), and everything in between.  As I do much of my work at the first link in the global supply chain, I attend numerous events like this each year.  When working with such a diverse group, it is often difficult to achieve consensus on anything, and particular irreverence is often displayed towards new technologies.  This is notable among the grower community, whose farms and farming techniques are often passed down through generations, just like an old watch or a wedding ring.  But after going into this latest conference expecting the usual pessimism about cooperation, consensus formed around the need and application of new sources of data.

The innovations in agriculture that grab most headlines are usually related to technologies such as new seed varieties, super-combines, or physical infrastructure that increases efficiencies in drip irrigation.  So, after one panel session comprised of investors looking for opportunities in both hemispheres of the Americas, I asked about the "non-tangible" innovations that often fly under the radar: those that require access to large databases, data manipulation creativity, and computational resources. The panel agreed that these are major focal points for the next generation of agricultural investments. Nearly every discussion that followed seemed to touch upon this theme.

The nice thing about quantifiable data for this community is that it can come from subjective sources as well as those repeatedly tested in a laboratory. A grower's logbook for instance -- containing such information as how a particular crop might respond to a specific weather pattern, the amount and type of pest-fighting application used in a given season, and local market offers -- can all be assembled into an index, which is another quantifiable data stream that users may have at their disposal.  And while upon first glance one might suppose that data streams are closely-guarded secrets, growers are probably among the most supportive advocates of open access and data sharing. What wiped out your neighbor's crop a decade ago may be the very thing that hits you this year.

In several offline conversations during coffee breaks, I offered some insight based on projects that Weather Trends is pursuing. We're working with clients to quantify relationships between weather patterns, crop disease, and agricultural yields. The potential for collaboration, and a new growth sector for this industry, was evident to everyone.

Looking ahead, I expect numerous high-quality and high-margin products to come to market that have their "roots" in both the acquisition of new types of agricultural data (ranging from genomic to planetary weather), as well as in repackaging existing data. As global food supplies are routinely subject to a number of shocks via weather, foreign exchange or geopolitics, this will be a very important platform for the global agricultural community in the years to come.


October 11 2010

Weather data and the supply chain

My introductory column focused on the broad theme of combining global weather data, sensor networks, and advanced forecasting techniques toward the early identification of weather and climate related hazards.  As discussed, early warning of potential weather-triggered problems will not prevent a physical hazard from occurring, but advance warning, even if only provided with a short lead time, may allow for some mitigation or avoidance measures to be employed ahead of an impact event.

Identifying extreme events is just one area where it's useful to apply long-range weather intelligence. In addition to the employment of various techniques to avoid human suffering, there are also numerous commercial interests who can apply this information as well. Just as a property reinsurer might want to know how either an acute or a seasonal weather event might affect premiums, growers may also want to assess crop potential, retailers might want to project seasonal product demand, and traders may want to employ a pricing strategy.

Food, energy, and weather

There are two things that each and every one of the roughly 7 billion people on the planet need: food and energy. Weather is central to both.

On the food side, advanced weather outlooks can help assess crop potential, both in terms of actual production/yield, and potential for disease pressure. As noted in my previous post, growers who anticipated crop losses from excessive heat or a lack of moisture (or both) may have purchased crop protection insurance, while manufacturing companies may have secured forward prices or managed their exposures through a hedge.

But even at a more basic level, there is a lot of environmental data available at the public's disposal. If structured and viewed in the right way, that data can provide insights on the supply side of many basic commodities. Assessing food production potential for basic necessities is what I like to call the first true step in understanding the global agricultural supply chain.

Now, there's no such thing as a perfect forecast. Even with forward-looking metrics that can be deemed 100-percent accurate, there are variables that go into determining, for instance, how many bushels per acre a particular region's wheat crop will yield. Forecasting is part art, part science. From the science perspective, the more clean, reliable data that we can obtain and plug into a model, the better we may get at determining production potential, or more importantly, highlighting areas that may be susceptible to a weather risk.

How La Niña affects milk production

Here's a simple example of the climate/food relationship: Casual weather observers are probably familiar with the El Niño Southern Oscillation cycle (referred to as ENSO). This particular phase of this large-scale physical weather driver often governs the global pattern. While an El Niño dictated much of the US pattern in 2009 (remember snow being trucked in for certain Olympic events in Vancouver?), the opposite La Niña has developed in 2010, which brings its' own set of variables. The current La Niña is one reason that our January-February outlook at Weather Trends is for a little cooler than last year for the western US. We also can use the ENSO index to assess yield potential for a commercially important commodity: milk.

For some time, we have known about the positive relationship between La Niña-like conditions and milk production for US dairy producing herds.  During an El Niño, conditions for much of the US dairy production regions may tend to be warmer and wetter, and as dairy cows exhibit sensitivity to heat stress and/or muddy fields, these conditions correlate with decreased milk production, particularly during the key months (March-July) in the annual cycle.  As the opposite La Niña pattern has been developing for the last several months, cooler temperatures have been present across much of California (the largest producing state), limiting heat stress and contributing to an active grazing season, both of which are good for milk yields.  This particular La Niña is actually shaping up to be a pretty strong event, and as a result, we have been expecting better US production numbers to follow.  Note that this does not take into account decreased herd size, so the emphasis is on milk yield per cow.

Picture 1.png

To test this idea, we can look closer at the relationship between the Southern Oscillation Index (SOI), which is an ENSO guide, and US milk production over the last decade.  Specifically, we can highlight periods where there has been a stronger trend toward positive-phase SOI in recent months relative to the 6 month moving average; the assumption being a stronger relative acceleration toward positive phase supports better milk production weather. 

Using a simple decision-tree scheme, the time series was split by grouping all months where the more recent period showed stronger positive SOI characteristics (as defined by a quantitative index).  Of this reduced group of months, we then looked at monthly normalized US milk year-over-year (y/y) production to see if stronger numbers may have been related to the index.  Therefore, using a sample size (n) of 60 months, y/y milk yields increased in 52 of these months (87 percent), verifying that a positive correlation exists.


Picture 2.png

This is not to suggest that the SOI, or any other weather parameter, is the primary factor in assessing potential milk production.  Remember, a forecast is a blend of art and science, so some subjectivity is involved. But this simple analysis does demonstrate that weather can be a key driver in the amount of milk that is flowing from producers to consumers, and it bears watching as a signal for forward pricing, and assessment of global stocks.

September 08 2010

Sensor networks and the future of forecasting

We can't control the climate, but are there ways to mitigate and avoid the negative effects extreme weather brings? I believe the starting point for potential solutions lies in data acquisition and environmental sensor networks (ESN).

Current technologies and sensors, ranging from cell phones to satellites, allow a "global environmental cyberinfrastructure" to be more than a topic for discussion at academic conferences. Researchers have studied connections and system interactions for some time, but now a broader segment of society is becoming aware of the precarious relationship between weather, climate and humanity. This awareness is sometimes motivated by the need to help. Other times there's a profit incentive. The reason doesn't matter if the result is a better low-cost global sensor network that can be tapped by anyone with a signal.

A systems approach to identifying natural hazards, coupled with a communications framework that can easily make data available to the public, is the crucial cornerstone of a functional environmental sensor network. The global monitoring of short- and long-range weather patterns and the linking of sensor-network data could allow forecasters to identify potential problems before they manifest.

The weather link between Russia and Pakistan

Weather has caused great disruption to many lives in both Russia and Pakistan in recent months. While these are separate circumstances, they share common physical factors. The following is a look at how events in one part of the world influence weather elsewhere.

The Russian heat wave

Global wheat prices spiked in early August. Much of that activity stemmed from potential crop losses in Russia, and it was helped by ubiquitous stories of parched fields and decimated crops. To be fair, part of the price spike came on the heels of a Russian export ban. Nonetheless, this story's origin is tied to weather.

The two maps below show the monthly year-over-year (2010 vs. 2009) changes in maximum temperature and precipitation for July. This weather was known to people in the agricultural and weather communities. The stage was set long ago for potential problems in western Russia.

(Click to enlarge)

(Click to enlarge)

A persistent high-pressure system centered over eastern Europe has combined with changes in the jet stream's pattern to keep temperatures high. When this high is as entrenched as it has been, it serves to do two things:

  1. It diverts a jet stream that would normally steer cooler air into parts of central Russia and northeastern Africa.
  2. It blocks moist air from the southeast, which exacerbates the dryness. This is a reason why parts of central Africa are seeing better moisture in recent months.

  3. There's an active west-to-east jet stream that travels above western Russia. This stream typically exhibits a seasonal shift to the east, and in the process allows moister air from the west to migrate into the region. The jet did not shift in July, and the result was a prolonged period of moisture-free air. When this combines with a strong high, the region experiences weather like it's seen over the last month. And when there is a high-pressure system in one region, there is often a corresponding low elsewhere.

    Flooding in Pakistan

    The low in this case has been over the mountainous region of northern Pakistan. This cold low has been the catalyst for a good portion of the excess rains. So, while located in distinct climate zones, the heat in Russia has a connection to the floods in Pakistan.

    But there's more to this puzzle. Every year, the annual Indian monsoon is anticipated throughout India and Pakistan, as much of the commercial activity that takes place in both countries is agrarian in nature. The monsoon was deficient in 2009, leading to short crops in many sectors. The arrival of the rainy season this year carried a heightened importance.

    The onset of the 2010 monsoon was healthy and most regions have been receiving beneficial moisture totals. But the placement of another area of high pressure over northeast India has, thus far, kept India's northern states dry. In the process, this high has been diverting even more moisture, which flows from southeast to northwest into central/western India and along into Pakistan.

    The first map below from NOAA's Earth System Research Laboratory depicts the storm tracks from the last week (as viewed via anomalies in outgoing longwave radiation), where the excess moisture is visible directly over Pakistan. The second map below, from the NOAA Climate Prediction Center, highlights wind anomalies over the last week. We see from this map that the stronger winds originating from the southeast were actively driving the moisture into areas that needed it the least.

    (Click to enlarge)

    (Click to enlarge)

    Pakistan's drainage infrastructure, which is silt-laden, has made the rainfall situation worse. When excess rains fall, it takes much longer to drain than necessary.

    Unfortunately, any excess rains in the coming weeks will likely be met with more problems for civilians. As of mid-August, estimates put flood-related deaths above 1,300, and more than 15 million people have been affected by the floods. Both of these figures are expected to rise.

    Satellites and risk management

    Could these separate-but-related crises have been foreseen? If so, what measures could have been initiated to mitigate some of the fallout?

    The map below is an indicator of vegetation health as derived via satellite for the wheat regions to the north of the Caspian and Black Seas. This particular graphic depicts the Normalized Difference Vegetation Index (NDVI) for the region, shown as an anomaly vs. the five-year average (for mid-to-late July). The index assigns a value for crop health, and based on the color scale shown in the legend, it is clear that the region has been exhibiting severe vegetation stress.

    The important thing to note here is that wheat prices started to increase in July, then exhibited a violent spike in early August. As this map is from mid-to-late July, we can see that by using tools such as satellite indices in conjunction with a long-range weather forecast, the current impact in wheat prices and ensuing financial turmoil could have been anticipated. To a certain degree, it could have been mitigated through a proactive physically-based risk management strategy. In addition, the same-satellite derived images that were capturing the movement of monsoon rains across India's agricultural regions could have been used to view the excess moisture in regions where the Pakistan floods originated.

    We can, of course, explore the questions surrounding how much lead time is necessary to avoid a crisis (remember Katrina). Nonetheless, it is clear that many were not aware of these disastrous systems until it was too late.

    (Click to enlarge)

    More coming soon

    This column is a starting point for discussions that examine climate, weather, sensors, networks, and their influence on society. The last couple of years have seen a heightened interest in this area from the research side, evidenced by new topics and sessions presented at the annual conferences of the American Geophysical Union, the American Meteorological Society, and others. I hope to bring some of these discussions to a broader audience while also helping readers understand how closely related their lives and decisions are to weather and climate.


Older posts are this way If this message doesn't go away, click anywhere on the page to continue loading posts.
Could not load more posts
Maybe Soup is currently being updated? I'll try again automatically in a few seconds...
Just a second, loading more posts...
You've reached the end.

Don't be the product, buy the product!