Study reveals that teenagers like money (for NAEP testing)

That’s right:  If you want to get valid scores from a testing program that examines teenagers, you should make sure that the teenagers have a reason for wanting to do well.

Teenagers (at least some) want to get a good SAT or ACT score so that they can go to a more prestigious college or university.  So SAT and ACT scores have some validity.

Ditto for AP tests.  (Note:  Parents want good AP scores to save money on tuition.)

The 11^th grade Science FCAT?  Not so much.  (RIP)

Now, researchers from Boston College and the Educational Testing Service have discovered that 12^th graders perform significantly better on the National Assessment of Educational Progress (NAEP) if they have a financial incentive to do so.

The authors question the validity of the conclusions that have been drawn from 12^th grade NAEP testing in the past.  But what will happen when the 8^th graders hear about this?

Here’s the summary on the Education Week blog Inside School Research.

FCAT controversy – What it isn’t, what it is, and what the FDOE leadership is doing

Just to be clear on what the FCAT controversy is about (and not about):

Everybody seems to agree that the scores on both the 2010 and 2009 FCAT tests are correct.  I have not seen a single documented claim that the scores on each exam are not correct.

The questions that have been raised are about the statistics on learning gains.  To oversimplify, the learning gain for a particular student is the difference between that student’s 2010 score and 2009 score in a particular subject.  The idea is to see how much the student learned in the 2009-2010 school year (at least through the date of the FCAT exam).  The learning gain statistics play a large role in determining the school and district grades that are awarded by the FDOE.

Some school districts are saying the learning gain statistics at certain grade levels and for certain subjects show dramatic changes from the year before that do not make sense.  They are asking for an independent audit of the results by an organization not connected to the primary contractor for the FCAT, Pearson.

Some school districts say they do not seem to have any problems and everything makes sense.

The FDOE leadership, which seemed anxious to deal with the controversy and close it out quickly,  is now taking steps to be flexible and appear completely above board.  There is a new FDOE web page which is being used to release all information on the subject which reaches FDOE.  They have also brought on a new organization for the audit requested by some school districts to address concerns that the original organization designated was not independent enough.  Finally, the tight timeline on which the FDOE was intending to issue school grades has been suspended.  There is now no firm date for release of school grades.

Update (11:00 am, Thursday): Sherman Dorn was kind enough to guide me through the learning gain jungle.  The school grading procedure asks how many students in the school have achieved a year’s worth of learning.  For each student, there is a “yes” or “no” answer, and that answer is arrived at using the procedure described on pages 11-12 of the “Guide to Calculating School Grades” (the 2009 version is here).  The school grade is based in part on what fraction of the student body made these learning gains.

Update (5:30 pm, Thursday): The Orlando Sentinel is reporting that 46 school districts have declared their concerns about the FCAT results.

Think this year’s testing problems are bad? Orlando Sentinel says wait till next year, when large-scale computerized testing is implemented

Leslie Postal at the Sentinel has some ominous news on the Florida testing front:  Only two school districts say they are ready for the implementation of computerized standardized testing on a large scale next year.  A large capital investment is still required to get ready (with Florida staring down the barrel of a $6 billion shortfall in the next state budget year).  And even if we had the money, the Sentinel article raises the question of whether the technology is ready for prime time.

Perhaps it’s time to invoke the First Rule of Holes:  The first step in getting out of a hole is to stop digging.  Give Pearson (or whoever) another year to get its act together.

Bringing it in for a landing? Chancellor requires district challenges by 4 pm today

School Zone is reporting that Florida K-12 Chancellor Frances Haithcock issued orders to the state’s school districts yesterday to submit challenges to FCAT results by 4 pm today.  She specifically mentioned challenges to “any questions or specific data related to learning gains in your lowest performing 25% of students.”

Commissioner Smith issued a statement last night saying (from Gradebook) “I have the utmost confidence in the accuracy and reliability of the 2010 FCAT results,” citing multiple reviews of the results.  Smith also seemed to assert that the complaining school districts were trying to cover for disappointing results:  “it is my belief that the data these districts are highlighting is accurate.  Our districts have done a tremendous job over the years in increasing the overall level of achievement in the lowest performing quartile of students and as expected, as that achievement rises, it becomes more difficult to reproduce that success year after year.”

I will throw out three speculative comments here:

First, let’s say the Commissioner is right and the results are correct.  Isn’t a decline in the progress of our lowest quartile students exactly what you’d expect when you terminate many of the state’s reading and math coaches, as we’ve done this year?  Are this year’s FCAT results the best case ever made for restoring educational personnel – in this case reading and math coaches?

Second, SB 6 drove a wedge between the Education Commissioner and the school districts.  This argument over FCAT results may be one of the first of many chapters in the story of the decline of the relationship between the state and local agencies responsible for public education in Florida.

Third, I just finished rereading Too Close to Call, Jeffrey Toobin’s account of the Florida recount in the 2000 presidential election.  Toobin argues that the main Republican strategy was to stop recounts (“bring the election in for a landing”) by any means necessary.  The Chancellor’s memo from yesterday insisting that districts submit challenges in writing by 4 pm today is eerily reminiscent of the Republican strategy in the 2000 recount.  While the strategy ultimately worked for the Republicans in 2000, it is unlikely that parents will stand for the implementation of such a strategy here.  The Commissioner and Chancellor must be extremely careful that they can’t be seen as trifling with the futures of Florida’s children.

Update (Tuesday, 3:30 pm): School Zone is reporting that Commissioner Haithcock is giving districts one more day.

FCAT meltdown underway?

The Orlando Sentinel and St. Pete Times are reporting that superintendents of five major school districts in Florida are questioning the accuracy of the results of this year’s FCAT.  The five school districts represent 40% of Florida’s school children.

This is potentially a huge story.  I recommend you follow it along with the experts at the Sentinel and Times.

Is “college readiness” enough to ensure an adequate supply of scientists, engineers and health professionals?

If you’re an aficionado of education policy, you’re familiar with the lament that many or most of the high school graduates who arrive on college campuses are, in the words of a new report released by the National Center for Public Policy and Higher Education and the Southern Regional Education Board, “not academically ready for postsecondary studies.”  The first paragraph of the report serves (as intended) as a nice introduction to the problem:

Every year in the United States, nearly 60 percent of first-year college students discover that, despite being fully eligible to attend college, they are not academically ready for postsecondary studies. After enrolling, these students learn that they must take remedial courses in English or mathematics, which do not earn college credits. This gap between college eligibility and college readiness has attracted much attention in the last decade, yet it persists unabated.  While access to college remains a major challenge, states have been much more successful in getting students into college than in providing them with the knowledge and skills needed to complete certificates or degrees. Increasingly, it appears that states or postsecondary institutions may be enrolling students under false pretenses. Even those students who have done everything they were told to do to prepare for college find, often after they arrive, that their new institution has deemed them unprepared. Their high school diploma, college preparatory curriculum, and high school exit examination scores did not ensure college readiness.

The visibility achieved by the college readiness issue is perhaps second only to the visibility of the high school graduation rate issue.  No one would argue that these two issues are not important. 

But when it comes to the economic future of Florida and the nation, the issue of educating an adequate supply of strong professionals in the engineering, health and science professions is equally important.  (For the remainder of this post, I will reluctantly adopt the acronym “STEM” – for “science, technology, engineering and mathematics” – as an inclusive term for these professions.)  To take one very narrow example:  A group of political and business leaders gathered last week in Orlando to push for state support for the development of the clean energy industry in Florida.  The critical piece of the clean energy industry puzzle that was left entirely out of the discussion was the importance of educating the scientists and engineers that the industry will need to thrive in Florida over the long term.  There is going to be a special session of the Florida Legislature held in Tallahassee later this month.  Governor Crist wants to put a constitutional ban on offshore oil drilling on this fall’s ballot.  Some legislators are talking about pushing a program of state support for the clean energy industry during the session.  One thing is for sure:  the education of scientists and engineers will not come up.

At the moment, it is highly unlikely that it will come up during the 2011 regular legislative session, either.  Florida’s new high school graduation law, approved during the 2010 legislative session and signed into law by Governor Crist, will do little or nothing to encourage talented students to pursue careers in the STEM fields.  It will also do little or nothing to prepare these students for the rigors of undergraduate programs in these fields – making them “STEM-ready”.  The report released before this spring’s legislative session by the Florida Council of 100 and the Florida Chamber of Commerce, Closing the Talent Gap, emphasizes the importance of educating more STEM professionals but then drops the ball in making the connection between what is done in our high schools and the numbers of these professionals that graduate from the state’s universities.  As far as I can tell, education leaders in the executive and legislative branches of Florida’s government believe that they are now done with math and science in high school – “We did that last year!”  Here’s one example of this:  Last month, a consortium of twelve states that includes some of the nation’s leaders in K-12 STEM education applied for funding from the US Department of Education to begin the transformation of their high schools’ STEM programs.  The funding of this $30 million program is virtually guaranteed, since the department has allocated $30 million for high school assessment improvement and no one else has applied.  Mississippi joined the consortium, which also includes Massachusetts, New York and Connecticut.  Florida did not.

There is one glimmer of hope.  Florida’s new high school graduation law also requires the legislature’s research office – OPPAGA – to issue a report on differentiated high school diploma programs in the US before the 2011 legislative session.  If Florida implements an “Advanced Studies Diploma” for university-bound high school graduates that includes the requirement of biology, chemistry and physics at the honors level, it will at least ensure that students entering our universities are STEM-ready.  Florida’s K-12 Chancellor Frances Haithcock has already made it clear (during the January 20 meeting of the House PreK-12 Policy Committee) that she opposes differentiated diploma programs like the one recently adopted in Virginia or the one that New York State has had forever.  But perhaps – just perhaps – some community of STEM professionals in Florida will see the importance of an Advanced Studies Diploma and take the message to Florida’s policy-makers.  I’m not holding my breath, but stranger things have happened.  After all, this is Florida. 

 

New York State Regents Physics vs. Florida Honors Physics 1 – Which does this physics professor prefer?

The New York State Regents Physics course and its accompanying exam have been the gold standard in accountable honors-level high school physics courses for years.  I’ve had a few Regents Physics alumni come through my general physics courses recently.  While they were bright enough, their outstanding characteristic has been that they were as well prepared for the experience of my general physics course as the students who had taken (and even passed) the algebra-based version of Advanced Placement Physics.

A look at the benchmarks covered in the Regents Physics course tells you why.  Take a look at the Regents Physics curriculum document here, and go straight to page 26.  If the student can do the stuff listed in that table – and there is nothing in the table that requires more than an above average mathematical ability – then the student will not have trouble succeeding in the standard calculus-based introductory general physics curriculum at a university.

Now look at Florida’s one-year Honors Physics 1 course, which is presumably the equivalent of Regents Physics (There is an Honors Physics 2 course, but absolutely nobody takes it.  A second year physics student takes Advanced Placement Physics.)  The specific physics content benchmarks begin on page 3 of that document, under item 2.  They continue through item 5.  The first thing the reader might notice is that it is difficult to figure out what Florida Honors Physics 1 students are supposed to be able to do.  As a college physics professor, I can look at the New York Regents course and know exactly what those students should be able to do.  The Florida Honors Physics document leaves me puzzled.

But it’s clear that there are some things that successful Regents Physics students can do that Florida Honors Physics students cannot:  use graphs to understand motion, understand how springs and pendulums cycle energy from potential to kinetic and back again, use conservation of momentum, understand that power is the rate at which work is done, analyze simple electrical circuits and calculate how they use energy, understand how wavelengths and frequencies determine what waves do, and interpret the spectrum of a quantum system like an atom.  And they can prove it on a statewide end-of-course examination.

Florida’s Honors Physics course is adrift.  Maybe that’s why not a single publisher offered a text for Florida’s Honors Physics course during the textbook selection meeting a few weeks ago. The publishers can’t figure out what to do with Florida’s Honors Physics 1 course description.

The Common Core science standards, which are presumably in the process of being written, can’t arrive soon enough.  In the mean time, it would make sense for Florida simply to adopt (copy) the New York State Regents Physics course, as well as the exam.  My guess is that the Regents exam results will arrive long before the results of Pearson’s new biology end-of-course exam.  Given the importance of the Honors Physics course to the education of scientists, engineers and health professionals in Florida, something really has to be done.

Brave New Testing World seems like touch-screen voting machine redux

Florida is scheduled to enter the Brave New World of school testing next year with the introduction of the large-scale use of computers for high school end-of-course tests.  And leading us into that Brave New World is (wait for it…) Pearson NCS, the same company that has brought Florida’s K-12 system to its knees with delays in the delivery of FCAT scores.

Pearson led Wyoming into that Brave New World this year.  The result (as Leslie Postal at School Zone reports) was a disaster.  This is from a story published last Thursday in the Casper Star-Tribune on PAWS, the “Proficiency Assessments for Wyoming Students”:  “The testing software developed by NCS Pearson was unreliable, inaccurate and unusable, [state superintendent of schools] McBride wrote in a letter to the company’s general manager. The Wyoming Department of Education requested $9.5 million in damages plus overtime pay for employees from Pearson, citing a ‘complete default of the contract.’”

This sounds so much like the rush to the now-discredited touch screen voting systems that I’m going to link to the definitive history of the now-discredited voting systems published by the New York Times.

Florida walks away from opportunity to use RTTT dollars to develop science EOC’s

It’s official:  Florida has walked away from an opportunity to use federal Race to the Top Assessment Program dollars to develop end-of-course exams in the physical and Earth/space sciences.  This decision provides a clear signal that the state has no intention of implementing end-of-course exams in chemistry, Earth/space science and physics.

Yesterday was the deadline for proposals to the Race to the Top Assessment Program (state coverage from School Zone and national coverage from Curriculum Matters).  Florida is leading one of two large consortiums applying for $320 million allocated for assessment reform in language arts and math.  However, the state declined to join a 12-state consortium applying for a $30 million pot designated for high school end-of-course examinations.  The 12-state consortium, called the “State Consortium on Board Examination Systems,” (SCOBES) is the only applicant for the EOC money, so funding for this consortium is assured.  Several states are participating in more than one consortium, so Florida had the option of continuing to lead its language arts/math consortium and participating in the EOC consortium as well.

The SCOBES proposal seems to be exactly what the doctor ordered for Florida’s STEM woes. (“last in the nation for science,” and all that)  This is what the proposal says:

Our core vision is simple: Rather than create new high school course assessments from scratch, we will take the world’s best Board Examination Systems and adapt them for use in a high school design intended to radically improve the academic performance of American high school students. Each of these instructional systems comes with very high quality examinations. We will align them with the Common Core State Standards and adapt them for use in American schools. We will use the same Board Examination Systems, including their examinations, to construct world-class STEM programs.

Later, it says:

The STEM program will offer courses and examinations in all the STEM subjects as well as related interdisciplinary courses.

The refusal of the Florida Department of Education to participate in this consortium (with Arizona, Connecticut, Kentucky, Maine, New Hampshire, New Mexico, New York, Pennsylvania, Rhode Island, Vermont, Massachusetts, and Mississippi) is the clearest indication yet that the state has no intention whatsoever to institute end-of-course examinations in chemistry, Earth/space science and physics.  The insertion of these exams in the “when budget conditions allow” section of SB 4 in the spring was simply a feint to get the science community off the backs of legislators.

RTTT Assessment Program deadline next Wednesday

The proposal deadline for the RTTT Assessment Program, a companion to the “big” RTTT program that has entered its second round, is next Wednesday, June 23.  The assessment program includes two components.  The bulk of the money will be awarded for comprehensive assessment reform.  Florida is one of the leaders in a consortium of states applying for the comprehensive piece.  But about $30 million will be reserved for a project involving end-of-course tests at the high school level. 

Even if Florida is a member of a winning consortium of states, it is unlikely that any of the money will end up funding end-of-course exams in high school chemistry, Earth/space science or physics.