Newer posts are loading.
You are at the newest post.
Click here to check if anything new just came in.

July 25 2011

How data and analytics can improve education

Schools have long amassed data: tracking grades, attendance, textbook purchases, test scores, cafeteria meals, and the like. But little has actually been done with this information — whether due to privacy issues or technical capacities — to enhance students' learning.

With the adoption of technology in more schools and with a push for more open government data, there are clearly a lot of opportunities for better data gathering and analysis in education. But what will that look like? It's a politically charged question, no doubt, as some states are turning to things like standardized test score data in order to gauge teacher effectiveness and, in turn, retention and promotion.

I asked education theorist George Siemens, from the Technology Enhanced Knowledge Research Institute at Athabasca University, about the possibilities and challenges for data, teaching, and learning.

Our interview follows.

What kinds of data have schools traditionally tracked?

George Siemens: Schools and universities have long tracked a broad range of learner data — often drawn from applications (universities) or enrollment forms (schools). This data includes any combination of: location, previous learning activities, health concerns (physical and emotional/mental), attendance, grades, socio-economic data (parental income), parental status, and so on. Most universities will store and aggregate this data under the umbrella of institutional statistics.

Privacy laws differ from country to country, but generally will prohibit academics from accessing data that is not relevant to a particular class, course, or program. Unfortunately, most schools and universities do very little with this wealth of data, other than possibly producing an annual institutional profile report. Even a simple analysis of existing institutional data could raise the profile of potential at-risk students or reveal attendance or assignment submission patterns that indicate the need for additional support.

What new types of educational data can now be captured and mined?

George Siemens: In terms of learning analytics or educational data-mining, the growing externalization of learning activity (i.e. capturing how learners interact with content and the discourse they have around learning materials as well as the social networks they form in the process) is driven by the increased attention to online learning. For example, a learning management system like Moodle or Desire2Learn captures a significant amount of data, including time spent on a resource, frequency of posting, number of logins, etc. This data is fairly similar to what Google Analytics or Piwik collects regarding website traffic. A new generation of tools, such as SNAPP, uses this data to analyze social networks, degrees of connectivity, and peripheral learners. Discourse analysis tools, such as those being developed at the Knowledge Media Institute at the Open University, UK, are also effective at evaluating the qualitative attributes of discourse and discussions and rate each learner's contributions by depth and substance in relation to the topic of discussion.

An area of data gathering that universities and schools are largely overlooking relates to the distributed social interactions learners engage in on a daily basis through Facebook, blogs, Twitter, and similar tools. Of course, privacy issues are significant here. However, as we are researching at Athabasca University, social networks can provide valuable insight into how connected learners are to each other and to the university. Potential models are already being developed on the web that would translate well to school settings. For example, Klout measures influence within a network and Radian6 tracks discussions in distributed networks.


Strata Conference New York 2011, being held Sept. 22-23, covers the latest and best tools and technologies for data science -- from gathering, cleaning, analyzing, and storing data to communicating data intelligence effectively.

Save 20% on registration with the code STN11RAD

The existing data gathering in schools and universities pales in comparison to the value of data mining and learning analytics opportunities that exist in the distributed social and informational networks that we all participate in on a daily basis. It is here, I think, that most of the novel insights on learning and knowledge growth will occur. When we interact in a learning management system (LMS), we do so purposefully — to learn or to complete an assignment. Our interaction in distributed systems is more "authentic" and can yield novel insights into how we are connected, our sentiments, and our needs in relation to learning success. The challenge, of course, is how to balance concerns of the Hawthorne effect with privacy.

Discussions about data ownership and privacy lag well behind what is happening in learning analytics. Who owns learner-produced data? Who owns the analysis of that data? Who gets to see the results of analysis? How much should learners know about the data being collected and analyzed?

I believe that learners should have access to the same dashboard for analytics that educators and institutions see. Analytics can be a powerful tool in learner motivation — how do I compare to others in this class? How am I doing against the progress goals that I set? If data and analytics are going to be used for decision making in teaching and learning, then we need to have important conversations about who sees what and what are the power structures created by the rules we impose on data and analytics access.

How can analytics change education?

George Siemens: Education is, today at least, a black box. Society invests significantly in primary, secondary, and higher education. Unfortunately, we don't really know how our inputs influence or produce outputs. We don't know, precisely, which academic practices need to be curbed and which need to be encouraged. We are essentially swatting flies with a sledgehammer and doing a fair amount of peripheral damage.

Learning analytics are a foundational tool for informed change in education. Over the past decade, calls for educational reform have increased, but very little is understood about how the system of education will be impacted by the proposed reforms. I sometimes fear that the solution being proposed to what ails education will be worse than the current problem. We need a means, a foundation, on which to base reform activities. In the corporate sector, business intelligence serves this "decision foundation" role. In education, I believe learning analytics will serve this role. Once we better understand the learning process — the inputs, the outputs, the factors that contribute to learner success — then we can start to make informed decisions that are supported by evidence.

However, we have to walk a fine line in the use of learning analytics. On the one hand, analytics can provide valuable insight into the factors that influence learners' success (time on task, attendance, frequency of logins, position within a social network, frequency of contact with faculty members or teachers). Peripheral data analysis could include the use of physical services in a school or university: access to library resources and learning help services. On the other hand, analytics can't capture the softer elements of learning, such as the motivating encouragement from a teacher and the value of informal social interactions. In any assessment system, whether standardized testing or learning analytics, there is a real danger that the target becomes the object of learning, rather than the assessment of learning.

With that as a caveat, I believe learning analytics can provide dramatic, structural change in education. For example, today, our learning content is created in advance of the learners taking a course in the form of curriculum like textbooks. This process is terribly inefficient. Each learner has differing levels of knowledge when they start a course. An intelligent curriculum should adjust and adapt to the needs of each learner. We don't need one course for 30 learners; each learner should have her own course based on her life experiences, learning pace, and familiarity with the topic. The content in the courses that we take should be as adaptive, flexible, and continually updated. The black box of education needs to be opened and adapted to the requirements of each individual learner.

In terms of evaluation of learners, assessment should be in-process, not at the conclusion of a course in the form of an exam or a test. Let's say we develop semantically-defined learning materials and ways to automatically compare learner-produced artifacts (in discussions, texts, papers) to the knowledge structure of a field. Our knowledge profile could then reflect how we compare to the knowledge architecture of a domain — i.e. "you are 64% on your way to being a psychologist" or "you are 38% on your way to being a statistician." Basically, evaluation should be done based on a complete profile of an individual, not only the individual in relation to a narrowly defined subject area.

Programs of study should also include non-school-related learning (prior learning assessment). A student that volunteers with a local charity or a student that plays sports outside of school is acquiring skills and knowledge that is currently ignored by the school system. "Whole-person analytics" is required where we move beyond the micro-focus of exams. For students that return to university mid-career to gain additional qualifications, recognition for non-academic learning is particularly important.

Much of the current focus on analytics relates to reducing attrition or student dropouts. This is the low-hanging fruit of analytics. An analysis of the signals learners generate (or fail to — such as when they don't login to a course) can provide early indications of which students are at risk for dropping out. By recognizing these students and offering early interventions, schools can reduce dropouts dramatically.

All of this is to say that learning analytics serve as a foundation for informed change in education, altering how schools and universities create curriculum, deliver it, assess student learning, provide learning support, and even allocate resources.

What technologies are behind learning analytics?

George Siemens: Some of the developments in learning analytics track the development of the web as a whole — including the use of recommender systems, social network analysis, personalization, and adaptive content. We are at an exciting cross-over point between innovations in the technology space and research in university research labs. Language recognition, artificial intelligence, machine learning, neural networks, and related concepts are being combined with the growth of social network services, collaborative learning, and participatory pedagogy.

The combination of technical and social innovations in learning offers huge potential for a better, more effective learning model. Together with Stephen Downes and Dave Cormier, I've experimented with "massive open online courses" over the past four years. This experimentation has resulted in software that we've developed to encourage distributed learning, while still providing a loose level of aggregation that enables analytics. Tools like Open Study take a similar approach: decentralized learning, centralized analytics. Companies like Grockit and Knewton are creating personalized adaptive learning platforms. Not to be outdone, traditional publishers like Pearson and McGraw-Hill are investing heavily in adaptive learning content and are starting to partner with universities and schools to deliver the content and even evaluate learner performance. Learning management system providers (such as Desire2Learn and Blackboard) are actively building analytics options into their offerings.

Essentially, in order for learning analytics to have a broad impact in education, the focus needs to move well beyond basic analytics techniques such as those found in Google Analytics. An integrated learning and knowledge model is required where the learning content is adaptive, prior learning is included in assessment, and learning resources are provided in various contexts (e.g. "in class today you studied Ancient Roman laws, two blocks from where you are now, a museum is holding a special exhibit on Roman culture"). The profile of the learner, not pre-planned content, needs to drives curriculum and learning opportunities.

What are the major obstacles facing education data and analytics?

George Siemens: In spite of the enormous potential they hold to improve education, learning analytics are not without concerns. Privacy for learners and teachers is a critical issue. While I see analytics as a means to improve learner success, opportunities exist to use analytics to evaluate and critique the performance of teachers. Data access and ownership are equally important issues: who should be able to see the analysis that schools perform on learners? Other concerns relate to error-correction in analytics. If educators rely heavily on analytics, effort should be devoted to evaluating the analytics models and understanding in which contexts those analytics are not valid.

With regard to the adoption of learning analytics, now is an exceptionally practical time to explore analytics. The complex challenges that schools and universities face can, at least partially, be illuminated through analytics applications.



Related:


December 21 2010

Can standardization and innovation coexist in education?

"How could you personalize your music if your Pandora data was lost every time you logged in?"

The question came up during a great conversation with Matthew Rascoff of Wireless Generation about personalizing the educational experience. He was making the point that information about students is lost every time they change schools, making it impossible to use data for personalizing learning the way we personalize our music, video, and reading experiences using Pandora, TiVo, and Kindle.

Personalization is the holy grail of education technology, but it can't be achieved without mechanisms for rich data about each student's learning. And that data must be persistently stored and appropriately accessible. Matthew neatly turns the traditional metaphor of a "digital locker" on its head by replacing it with the "data backpack" -- a container that goes everywhere the student goes.

Matthew's insights led me to Wireless Generation's white paper on "An American Examination System." The paper outlines a platform for using technology to collect and store data on individual student performance. With this rich data following students, teachers will have the data available to know what students need to work on or learn next. The data can become the basis for "adaptive mass personalization."

What I particularly like about this vision is that it is designed with nicely partitioned layers: beginning with the technology and protocols for data collection and sharing, moving to a format for encoding a hypothesis about an optimal set of learning trajectories, moving to an automated and rich layer of high cognitive demand tests, and supported by grading mechanisms that are automated where possible, and draw on networks of graders where appropriate.

The idea is that by using technology, standardized tests can be graded by any teacher anywhere, not just a given student's classroom teacher. With this sort of expert crowd-sourcing, the community of teachers can become much more consistent in their expectations, and minimize the variation in grades that comes from the subjective nature of evaluating creative work.

Because the layers of functionality are segregated, this vision represents a meaningful platform approach to what I consider to be the absolute core issues for reinventing education: testing and assessment.

Standardization is what makes a platform powerful. Railroads are a platform for the transportation of goods and people because all tracks are the same size and all trains can ride on any track. This means there's an the opportunity for innovation regarding different kinds and models of trains, speed and efficiency improvements, business models for transporting goods, etc. Over time the train tracks and systems have come to support another layer of standardization -- containers -- that allow goods to be efficiently transported across railways, trucks, and ships. If that second layer of standardization had happened at the same time as the first, the resulting standards would have been far less efficient for the mass movement of goods than those that exist today. If business model competition didn't happen, the economic environment that made container standardization so powerful might never have developed.

When technology products seek to become platforms, they often look to lock up the value chain from top to bottom by standardizing every layer according to their own product's protocols. They "open" the top API layer to entice others to choose their platform. I've often heard this informally referred to as the "build it and they will come" strategy by those who are skeptical of it, and a "vertical integration" strategy by those who are fans.

In industry this plays out in the debate of "open" vs. "closed." Is the iPhone closed because of the many layers of the platform Apple controls, or open because of the breadth of innovators who can try their products on top of Apple's API's? Android embraces the idea that an architecture standardized at a lower layer will breed more innovation and eventually lead to global domination over the siloed Apple stack. The question of which layer to standardize in a cell phone platform is an open one and the economic implications of choosing the right layer are vast.

In the area of education technology, this question of where to standardize is even more critical. Whereas in industry, competing platforms will eventually answer the question of which approach generates the most innovation and success, in education, early decisions about platforms will become institutionalized and almost impossible to change for decades. This is a critical point to consider as the US Department of Education, governors, and chief state school officers all are investing heavily in a vision of the transformation of schooling through technology.

If standardization happens too far down the stack (say, below the level of student data flow protocols), the market will continue to be fragmented and there will be significant barriers for education innovations to reach students at scale. Innovators will essentially be selling school to school and district by district. If standardization happens too far up the stack, areas that are still immature in the field will become cast in concrete and innovation in those areas will be unlikely.

With a platform structure for education, the key question becomes what to standardize in order to enable wide adoption of innovations, and what things must be deliberately and specifically not standardized in order to allow new innovations to flourish.

Using assessment platforms as an example, the Wireless Generation White Paper highlights a specific theory for students' learning trajectories in math, yet at the same time provides a structure they refer to as a "honeycomb" for specifying that theory. If the field were mature in its understanding of learning trajectories, the suggested trajectories could be a useful place to standardize. But given that competing learning trajectories are still emerging, the honeycomb structure instead provides a well-defined place for critical innovation.

Wireless Generation honeycomb
An example of the honeycomb structure discussed in "An American Examination System" (PDF).

Platforms are valuable precisely because of what they do and do not standardize. Widespread standardization is difficult to accomplish, but critical to realizing the potential of technology in reinventing education. Wireless Generation has an intriguing framework for such a platform (which is probably why it was just acquired by News Corp. for $360 million). With the resources of News Corp. joined with the innovative platform approaches of Wireless Generation, standardization at the right layers is a real and compelling possibility. It will be fascinating to watch this marriage unfold.



Related:




Sponsored post
soup-sponsored
Reposted byLegendaryy Legendaryy

October 27 2010

Gaming education

There are at least three different classes of digital games in schools. Which you prefer speaks volumes about the role you believe schools should play.

The first group, the classic edu-tech games, have danced in and out of schools for so long that many kids take them for granted. Most of these programs are cute, but they fall short on pedagogical ambitions and graphic design. That doesn't make them worthless; it just limits their effectiveness. (One person's drill-and-kill can indeed be another's guiding light. When educator and blogger extraordinaire, Scott McLeod, asked, "Do most educational games suck?" he drew fire from just about all sides.)

By contrast, a handful of educators a few years ago sought to put game controls directly into students' hands by teaching them how to build their own games. Scratch, developed by the Lifelong Kindergarten Group at MIT's Media Lab, is the reigning champion here. (Here's more of my take on Scratch). There are a few others, too, including Microsoft's Kudo, a programming language that kids can use to build games for the Xbox game platform.

Screen from The Fly, a game built with Scratch
Screen from "The Fly," a game built with Scratch.

And now comes what I would dub a third approach, something that has picked up its very own buzzword before it has even reached most school gates: gamification. The term is as elegant as a teenager jawing a mouthful of bubble gum. But it suggests adding far more sophisticated game mechanics to applications -- no matter how stuffy or serious the application has been. Gamification probably has more momentum outside of schools than in. Case in point: Dean Takahashi of VentureBeat has written about how DevHub, a place for web developers, added gaming feedback and watched in awe as the percentage of users who finished their sites shot up from 10 percent to 80 percent.

Most games are naturally social, which means gamification depends on that other ubiquitous web trend, social networking. Sure, go ahead and play Solitaire. But most of us take a certain pleasure in besting the competition -- whether it's the Philadelphia Phillies or some ugly troll in World of Warcraft.

Academics are creating a skin of respectability for gamification. Byron Reeves of Stanford University has recently co-authored "Total Engagement" to outline his ideas about how gaming can turn the erstwhile plodding company man into an engaged and motivated worker. (Reeves is also putting his ideas to the test by co-founding a consulting firm, Seriousity, that will coach companies on how to do this.) The first gamification summit is slated to take place in January in San Francisco.

What does each of these approaches say about education?

The first type of games were willing to entertain kids to keep them engaged -- the "just-make-it-fun" school of thought. But any standup comedian will tell you how tough it is to keep people entertained for long. It's even harder with kids who outgrow the "fun" of a game faster than most games can evolve.

The Scratch camp is more about empowerment. Scratch appeals enormously to kids who want to control their environment and be in charge. Those who build Scratch games get feedback from others when they post their games. They say they love the comments and feel great when hundreds of others play their games.

Ultimately Scratch aficionados bring their ambitions to learn with them. I'd wager that if these kids were born a generation or two ago, they'd be building transistor radios. The Scratch kids have to be self-motivated: most use Scratch outside of school. No one makes them do it. All it took to get them going was for someone to introduce them to Scratch in the first place. That's a great argument for exposing more kids to the tools.

Gamification, by contrast, doesn't rely on internal motivation. Instead, it's using the oldest tricks in the book: providing instantaneous feedback, egging on the competition, and rewarding even tiny steps of progress. Gamification assumes that the player isn't especially motivated -- at least at the beginning -- and then provides barrels of incentives to ramp up that motivation.

I'm betting that gamification, in spite of its throat-clearing name, is going to be big in the commercial world -- and in schools. Gamification can help build kids' competitive spirits. As they gain confidence, they may become hungry for tools that put them in control. At the end of the day, those who know how to create the rules of the game, know how to win.



Related:




September 14 2010

iPod program helps school test scores

Last month, we had an exceptional panel talking about Mobile in Education at our largest Mobile Portland meeting ever. A report on how iPod Touches are making huge differences in third-grade test scores really stuck with me.

Joe Morelock, the director of technology and innovation for the Canby School District in Oregon, shared with us how Canby started a pilot program of iPod Touch devices in a single third-grade classroom. The pilot's success led to the district setting a goal of providing every third-grade student with access to an iPod Touch.

Morelock has documented the program in a presentation you can download from the school district's wiki.

Below, I've pulled out a few slides from Morelock's presentation that illustrate the remarkable improvements. These charts start to explain why the school district got behind the program so quickly.

The charts compare the performance of third graders throughout the Canby school district with those whose classroom used iPod Touches throughout the year. As you can see in the chart below, the number of students that meet or nearly meet the math requirements on a standardized test are much higher for the iPod Touch classroom (left circle).



Pie charts comparing math scores of students with iPod Touches with those throughout the district



The difference in performance is striking when looking at students with disabilities (below, left column):



Migrant and ELL students



The increase in test scores for students with disabilities appears to validate some of the early anecdotal reports that iPhones and iPod Touches were making a difference for children and adults with autism.

The program also had a positive affect on English language learners (below, right column):

Students w/ disabilities, minorities


And it's not just math scores. Here are reading test results from the same classroom:

Reading test scores

Reading test scores continued

Parents whose children have been exposed to iPod Touches in the classroom don't like the idea that their children may not have them when they move on to the next school year, so they're organizing fundraisers to purchase additional devices. Because iPod Touches are relatively inexpensive, five can be purchased for the same price that would have been required to purchase a single laptop.

The Canby School District is extending the iPod program by providing iPod Touches for all third graders district-wide during the 2010-2011 school year. In addition, pilot programs using iPads will run at the elementary-, middle- and high-school levels.

Perhaps most importantly, both students and teachers love using the devices:

You know that little boy who came up to us this morning? He loves the iPod Touches. They have made an incredible difference in his math work. He has Asperger’s, and before the iPods, he could never sit through a math class. The kid absolutely loves math now and gets As. He sits himself up at the front of the room -- he likes to be by himself -- tucks his foot up, leans on the desk and goes to town on math. It's simply amazing. -- Gale Hipp, sixth-grade math teacher. [Note: Link added.]

And simply:

This is the most fun I have had teaching in the last 25 years. -- Deana Calcagno, fifth-grade teacher.


The full panel discussion is available in the following video. Morelock's segment on the Canby School District and their iPod pilot program starts at 19:20.




Related:

June 08 2010

Don't get stuck in Edu 2010

Business entered the computer age in the 1980s. Every department had at least one computer, often more. Laborious tasks such as collecting, tabulating, and representing data were completed in an instant by modern applications such as Lotus 1-2-3 and dBase III. We began to infuse the workplace with more and more computers, dazzled by the productivity gains we were about to realize. But dramatic gains never came from just automating our existing work processes; they materialized when we transformed the way we worked. When real-time information allowed us to virtually eliminate inventory through just-in-time delivery. When we learned to collaborate across time zones and geographies. When we began to become productive in "snippets of time" thanks to e-mail in our pockets. When real-time access to information and communication enabled teams to self-organize and take ownership rather than wait for instructions to flow down the low-bandwidth, noisy and lossy channels of hierarchical communication.

In many ways, education technology is today where business was thirty years ago. Almost no one questions the promise of always-available computing and broadband connections yet we are puzzled when infusing the schoolhouse with more and more computers doesn't always yield dramatic gains. As with business, education will see the radical impact when we move from automating existing processes to transforming the way we teach and learn. When real-time information on student progress will allow just-in-time delivery of the right lesson. When students become productive in "snippets of time" thanks to on-line learning tools in their pockets. When real-time access to information and communication enable students to collaborate, research, peer review, and mentor each other rather than only waiting for information to flow down the low-bandwidth, noisy, and lossy channel of one-size-fits-all lectures.

The National Education Technology Plan (pdf) gets to the heart of this, calling for "revolutionary transformation rather than evolutionary tinkering." The plan outlines models and specific recommendations for learning, assessment, teaching, infrastructure, and productivity. It offers the U.S. Department of Education a vivid sketch of education powered by technology and shaped by the learning sciences. A careful read reveals a deeply informed picture of teaching and learning that is both aspirational and achievable and that is grounded in the most current capabilities that technology has to offer.

But technology can offer more.

In the last decade, the technology investments we made in computer literacy were largely variations on shared computer labs with desktop computers hard-wired to the Internet. More recently, some schools that can afford it have started providing a wireless laptop for each student (1:1 laptop programs). This learning experience is different in kind, not just degree, from limited hours spent in a computer lab. But it requires a whole different infrastructure -- more robust WiFi on campus, for example -- and re-architected systems that don't lose student work when the connection goes down. Consequently, much of the innovative work developed for the computer lab model doesn't translate to the 1:1 laptop model and needs to be either drastically modified or recreated at great expense and effort.

Today a few schools are beginning to experiment with technologies alluded to in the National Education Technology Plan: cloud-based services accessed by connected devices such as cell phones and laptops with mobile broadband modems. Once again, the learning experience changes in kind, not just degree, and once again the requirements on the infrastructure change, requiring re-architected systems that support devices that move across networks and are sensitive to bandwidth usage. In the business world we went through a two-stage process: first we reinvented our processes as we moved to a wired computer experience, then we transformed again as we went to using always-on, always-connected mobile devices. In education, we have the opportunity to leapfrog that intermediate step.

The plan envisions:

... a model of an infrastructure for learning [that] is always on, available to students, educators, and administrators regardless of their location or the time of day. It supports not just access to information, but access to people and participation in online learning communities. It offers a platform on which developers can build and tailor applications.

The plan also points out that both the learning sciences and technology will continue to evolve. With investments being made now in education that may not be repeated for decades, the challenge presented to technology is one of developing platforms that will not require massive tech do-overs and reinvestment as new technologies come on line.

What will happen if we merely implement current technology as a one-off investment? Will additional networks (such as peer-to-peer, personal area networks, and body area networks) soon require re-architecting the applications and services about to be developed for traditional cloud and mobile device environments? Will breakthroughs in assessment, analytics, and data visualization require re-writing content and curriculum to capture data? Will newly re-developed back-end data systems need to be thrown out and recreated from scratch to support that data capture? Will trends toward using student-owned devices in school quickly require a complete rethinking of privacy and security?

The National EdTech Plan aspires to bring together the best of what we know of teaching and learning with the very best technology has to offer in 2010, yet we can be certain that technology will offer even more in 2012, 2015, and 2020. A literal interpretation of the plan could end up doing no more than codifying the best practices and technologies of 2010. Is it possible, instead, to codify the spirit of the plan and implement the technology infrastructures that will allow education as a platform to drive innovation for decades to come?

Older posts are this way If this message doesn't go away, click anywhere on the page to continue loading posts.
Could not load more posts
Maybe Soup is currently being updated? I'll try again automatically in a few seconds...
Just a second, loading more posts...
You've reached the end.
(PRO)
No Soup for you

Don't be the product, buy the product!

close
YES, I want to SOUP ●UP for ...