Research shows .... a lotta bunk and a little truth Time for you to write a letter

The Report of the National Math Advisory Panel clearly stated that most of the Education Community does not know how to conduct scientific research. A quote from their recommendations reads,

“the Subcommittee on Standards of Evidence recommends that the rigor and amount of course work in statistics and experimental design be increased in graduate training in education. Such knowledge is essential to produce and to evaluate scientific research in crucial areas of national need, including mathematics education.“

http://www.ed.gov/about/bdscomm/list/mathpanel/report/standards-of-evidence.pdf

The NMAP panel made this recommendation because they found [my emphasis]: “The Panel’s systematic reviews have yielded hundreds of studies on important topics, but only a small proportion of those studies have met methodological standards. Most studies have failed to meet standards of quality because they do not permit strong inferences about causation or causal mechanisms (Mosteller & Boruch, 2002; Platt, 1964). Many studies rely on self-report, introspection about what has been learned or about learning processes, and open-ended interviewing techniques, despite well-known limitations of such methods (e.g., Brainerd, 1973; Nisbett & Ross, 1980; Woodworth, 1948).” Note that some of the Panel’s reference sources date back to 1948 and 1973.

This means that the billions of dollars spend on Education Research over the last 20 years (and probably before) has largely been wasted.

A paper by Robert Rosenthal, “Science and Ethics in Conducting, Analyzing, and Reporting Psychological Research”, American Psychological Society, May 1994,

http://www.psych.utoronto.ca/users/psy3001/files/Rosenthal%20'94.pdf

clearly states why poor quality of research is unethical. He states (in this paragraph he is discussing an example from education research),

“How does the poor quality of the design raise ethical objections to the proposed research? Because students', teachers', and administrators' time will be taken from potentially more beneficial educational experiences. Because the poor quality of the design is likely to lead to unwarranted and inaccurate conclusions that may be damaging to the society that directly or indirectly pays for the research. In addition, allocating time and money to this poor-quality science will serve to keep those finite resources of time and money from better quality science in a world that is undeniably zero-sum.”

The NSF Education Directorate has been funding this poor quality research for 20 years. This ethical failure meant, as the NMAP committee found, that when policy advisors went seeking results to guide important policy decisions they found the Education Research cupboard was largely bare. 20 years wasted while huge numbers of students failed to learn math. This ethical failure has two parties, the NSF grant officers and the researchers who requested and spent the money for their research proposals. Properly trained researchers in other science disciplines would not have submitted such poor proposals and if they had, their granting agency would have been unlikely to have approve them. The NSF Education Directorate must be revamped.

In contrast, the Economics Research Community has been having terrific success analyzing the river of data produced as a result of NCLB. For example, see the papers published by the Education Program of the National Bureau of Economic Research (NBER) (a non-profit research org) at

http://papers.nber.org/papersbyprog/ED.html

They can do this work because Economics Researchers receive training in statistical analysis and research design. For a nice overview of the work published by the NBER Education Program see,

http://www.nber.org/programs/ed/ed.html

I would like you to send a letter to your WA DC legislator urging a major change to the leadership of the NSF Education Directorate in order to put this important agency on the path of conducting scientifically based research.

I also think that all WA state colleges and universities should conduct an ethics review of their education research projects using the points raised by the Rosenthal paper as a starting point. The WA state legislature needs to drive this by appointing an Independent Research and Ethics Review Board.

Please feel free to copy any of the above in your letters.

Thanks,
-- David Orbits

---------------------------------
Here is another article on NSF misdirection and waste:
http://pwceducationreform.wordpress.com/2009/05/20/nsf-sends-more-money-down-the-drain/