"My company gives me an annual review that affects my pay--why shouldn't teachers have the same?"

[jpgray]

It's a totally fair and reasonable question, and it seems prima facie like the situations should be comparable. In my opinion, they are not.

With most jobs, we're talking local, self-contained, self-imposed standards for a single company, for a position with identifiable and specifically-targeted goals, with the end motivation being relatively straightforward and accountable pursuit of profit. The hard numbers in many jobs don't lie, and unless your boss is a banking CEO, there's a strong disincentive for her to accept fudged numbers. A national merit pay program for teaching could hardly resemble this. Let's count the ways it doesn't:

1. The goal of public education is to provide the opportunity of learning to all, whereas the goal of business is maximizing profit. The conflict there with merit testing becomes obvious when you realize a lot of disadvantaged kids need heroic efforts, but even the best effort brings disappointingly small returns. A business that has the choice of focusing on high-return investments rather than low-return investments will drop the latter like a hot potato, and that's exactly what teachers under imposed standards do with disadvantaged students. Why encourage a bad student that's dragging down your rating to stay in school? Boot his ass out, make him hate it--you can then see a bigger payoff for your efforts.

2. Expectations of student success are not easily generalized to form a national standard, are highly situational and in all senses not readily quantifiable. How much worse is a teacher allowed to perform if her class has twice as many disadvantaged kids as another? 50% more? Just one more?

3. If not generalized, then authority must be given to administrators at individual schools to determine merit, and trusting principals and teachers to apply reward-based standards to themselves fairly on a local basis hasn't been the best of ideas. They simply do the scholastic equivalent of cooking the books by encouraging disadvantaged students to drop out, easing the standards of the curriculum, etc. There are no "hard numbers" here as there are in business--there is no numerical measure of success (grades/scores) that can't be massaged to yield an undeserved reward (unlike business, sort of like banking :P).

4. Moreover the personal risk for an administrator or teacher dishonestly inflating his/her numbers is virtually nonexistent. A principal won't lose his job just because his school's performance is falsely inflated, as there is no significant risk to keeping a happy face on passing students who don't really know the material (or on shoving out the undesirables). If we want to double-check their local results to ensure honesty, we're back to standardized tests again, and again we create an incentive for ambitious schools/teachers to push out or pawn off under-performing students at every opportunity. Which is the opposite of the ostensible goal.

[sandnsea]

1. Nobody is talking about a test-based model

http://www.democraticunderground.com/discuss/duboard.php?az=show_mesg&forum=132&topic_id=8256531&mesg_id=8256531

[jpgray]

7. "STUDENT GROWTH OBJECTIVES"

How is this to be measured? If one is rewarded for meeting these, why not cook the books? How do you stop a school from cooking the books, unless by testing?

[sandnsea]

9. Read It

Click the link, click some more links, LEARN, and don't come back with any more questions until you do.

[jpgray]

15. You don't have an answer? I've read the site.

[sandnsea]

18. The teacher writes the measurements

themselves. The program belongs to the teacher. They're the ones who pre-determine the measurement. The principal or administrator simply evaluates whether the teacher has met their own criteria.

You didn't read it at all.

[jpgray]

20. So what stops the teacher and principal from just throwing out "mets" like wildfire?

What stops them from cooking the books? I'm still not getting an answer from you.

[sandnsea]

22. The district, department, and state

If one school or department doesn't seem to really be measuring up, then the district steps in with its own set of objectives, passes that on down and the school and teachers have to come up with strategies to meet those standards.

[jpgray]

23. You're missing the point entirely. How does the district/state know the school isn't measuring up?

It's meeting all its student growth objectives! It's one of our top performing schools!

:eyes:

[sandnsea]

25. Oh, the kids move to a different school

and don't know anything.

One of them posts under the name jpgray on a message board and is proven to be incapable of logical deduction.

[ThomCat]

38. incapable of deductive logic for not agreeing with you?

What is your problem today? :shrug:

[ThomCat]

34. "Don't come back here until you do"

:wtf:

Anyone who disagrees with you is banned until they decide to agree with you? :shrug:

[Forkboy]

58. That post sums you up perfectly.

Congrats!

[roguevalley]

32. in the real world if there is a fuck up on either side of someone that probably doesn'taffects

their personal performance report. Teachers are being judged on the ability of their kids to learn and they have no control over the zillion factors that affect that. Most people aren't judged on the basis of their co-workers efforts. Nice post, OP.

[Double T]

2. Teachers should have an annual review and tenure should be eliminated.

[ret5hd]

4. meaning "fired at managements will"?

[Double T]

10. Yep. Just like the rest of US school tax payers.

Teachers need to climb down off their pedestals or get off their high horse; the world has changed and so have the benefits.

[ThomCat]

35. So you're anti union?

That's not really a democratic position.

What about if some bigots in the community don't want gays or lesbians teaching? Should they all be fired at the whim of the management in order to keep the vocal part of the community happy?

What about athiests? No atheist could get a teaching job in many parts of the county if that was the case.

If teachers can be fired at the will of management, then anytime the community has a dispute with the teacher (real or imagined) it's the teacher who is going to lose. How can any teacher work under those conditions?

Tenure developed for a reason. There were, and still are, legitimate reasons for protecting teachers from arbitrary judgments.

[Double T]

49. The original intent of a union was fine. What the union members.......

and the union executive hierarchy have made it isn't fine. The rest of US can be fired by the will of management. Come out into the corporate world and see what working for management psychopaths is really all about.

[WinkyDink]

63. Dilbert, is that you? Because your pointy-haired boss just called a meeting and you're late.

[ThomCat]

67. I was a Corporate Management Consultant for many years

in Manhattan, for global corporations. I know the corporate world very, very well. Thank you. I don't need lessons.

I managed union personnel. They had some faults, but they were always the best trained and most reliable staff.

It sounds like you just have grudges.

[Marrah_G]

70. So rather then fix the working conditions for all your response is to drag others down?

That is the Republican point of view. Not the Democratic point of view.

You should be wanting things to get better for those workers outside of the unions.

[olegramps]

75. I love these people who say the "original intent of unions was fine; but!

Tell me what is different from the original intent of unions to protect workers from arbritrary firing from today's situtation.

I will venture an answer. What was won by those who sacrificed to get workers benefits is now claimed by those who are unwilling to support unions because they think they can get it without their support. i.e., Union Dues. Simple as that. Greed.

[Bill McBlueState]

46. that's bs

You think everyone in the country except for teachers is subject to at-will employment? As that other guy says, "Teh stupid, it burns!"

[Better Believe It]

48. So you have no employee rights where you work and employment contract?

That's a shame.

All CEO's a big firms have labor contracts. If they do poorly they make more money, if they get fired they receive huge bonuses.

A regular employee gets the shaft even if they do an outstanding job unless they have a just and reasonable labor contract.

I'm sorry you have no employee rights where you work. Can you be fired at will and without cause as most employees can who lack a labor contract?

Well, maybe your some kind of boss or supervisor so it's OK to crap on those you supervise.

[Double T]

50. CEOs and Union members both abuse labor and employment contracts.

It is an interesting commonality.

[Better Believe It]

55. How do union members "abuse" their labor contracts?

Do they get multi-million dollar golden parachuts?

Do they force a manager or CEO to pay them twice as much as provided for in their labor contract?

Do union members pick on CEO's until they commit suicide?

Do union workers make the employers pay for 5 star hotels and restaurants they used when on their 2-month long European vacations.

Oh please tell us a tale of woe .... of how employers are exploited, used and abused by those greedy union workers.

I'm sure you'll spin a very sad tale.

Let me shed a tear for Wall Street and corporate tycoons. Poor, poor babies.

BOOOOOOOOOOOOOO HOOOOOOOOOOOOOOOO

[Double T]

57. The 'abuse' is on two totally different scales; it still remains abuse.

As a consequence of the 'abuse' union workers have lost a significant number of jobs while CEOs just keep screwing the hell out of the remaining workers, customers and threaten the viability of their own corporation.

[Better Believe It]

60. And be specific .... what kind of abuse are you writing about?

No glaring generalizations now. I'd like specifics on how union workers, as opposed to non-union workers have abused employers.

Now if you claim that some union workers are a-holes and are lazy that's true .... and it's even more true in my personal experience with non-union workers!

perAnd the lazy non-union workers keep working because of personal friendships with or doing special favors for people in management
positions. You won't find that much politics and favoritism at well organized union workplaces.

[Double T]

69. Empower an A-hole and lazy union worker and see what happens.

The arrogance is outrageous and there is little that can be done about it. An A-hole or lazy non-union worker can be fired on the spot. Huge difference.

[olegramps]

76. Your argument is analogous to Reagan's lies about "welfare queens."

Just plain unsupported bull shit about some phantom worker that doesn't do his part and the union protects them from being discharged. Pure Republican propaganda just like Reagan's "welfare queen" who drove a Cadillac and collected 30 welfare checks.

[WinkyDink]

64. You mean like taking sick-days that were NEGOTIATED? Or not working for FREE beyond contract hours?

Or getting the job descriptions IN WRITING?

Except: Teachers are hassled if they take "too many" sick days (i.e., the allotment); teachers work plenty of unpaid "over-time"; and the job descriptions are usually just vague enough to include assigned hall, cafeteria, and extra-curricular "duty".

For every proverbial union gold-bricker are three Peter Principle managers.

[WinkyDink]

62. I see you're of the "Misery Loves Company" School of Equity.

Well, guess what? YOU CHOSE the private sector.

[Medusa]

87. That's how it works in the real world outside of academia

so why are teachers exempt from performance reviews like the rest of us? There are some fantastic, amazing, wonderful teachers out there. And then there are those who need to have their butts kicked to the curbs for poor performance, theirs, not their students. This plan is not based on student scores and anyone who thinks it is isn't reading the details of this proposal carefully.

[knitter4democracy]

56. I taught in that kind of school, the Catholic schools.

It was so much fun having my principal stand outside my classroom door and eavesdrop and then write it up as an official observation. The contract didn't specify that she had to be in the room to see everything, just that she observe. It was so much fun having Sister scream at me when I allowed a student to write an editorial on the school lunch food not being to everyone's taste and threaten to fire me right there. It was so much fun to get a weekly reminder that I was an at-will employee during my weekly getting-screamed-at session that started to put me into early labor.

I'm just sayin'--it's not all fun and games, and it doesn't mean you get better teaching at all.

[valerief]

3. Can teachers be off-shored like the rest of us? I wonder if Corporate America

will have them replaced with kiosks and off-shore support lines.

Don't laugh. It could in the works now.

[sandnsea]

5. I've heard similar suggestions

Computer monitors and teachers in centralized locations.

[Cant trust em]

13. Most US colleges have some kind of online component now

I think it's an important step, but obviously should not be the only way we get educated.

[sandnsea]

16. It would be disaster for kids

They need the encouragement and inspiration that only a teacher can provide. We'd probably end up turning out a bunch of sociopaths.

[Cant trust em]

26. I'd imagine it being very difficult to keep kids from eating the paste if it were online.

Of course kids can't be using online classes, but I do think it's a great way to get even more people some education.

[valerief]

31. Oh, I'm sure they'll keep paste off the educational menu. They'll strap

kids to seats in front of computers and give them two pee breaks a day. Just like the corporations treat their parents.

[Cant trust em]

33. Visions of "A Clockwork Orange" are running through my head. nt

[valerief]

30. Definitely, but Corporate America doesn't care. nt

[GiveMeFreedom]

29. Yep

H1-b visas should do the trick. Import workers with different lifestyles and thick accents. Curry for lunch? :sarcasm:

[BadgerKid]

40. Yup. Search for "global campus" n/t.

[Double T]

68. The H1B visas for teachers are in the works; count on it.

[BlooInBloo]

6. Nobody should ever EVER judge teachers by the work they produce. Durr.

[ulysses]

8. nobody should ask you to read anything longer than a soundbite.

:D

[sandnsea]

12. Please Evaluate ProComp

list pros and cons so that any parent can participate intelligently in their own district. Whether one approves or disapproves, one has got to have facts when this issue comes up and it is going to come up. Thanks.

http://denverprocomp.dpsk12.org/support/pdu

[ThomCat]

36. Do you get a commision for selling it?

:shrug:

[jpgray]

11. This isn't about determining locally, with personal experience, whether a teacher is good or bad

It's determining such with national standards that are tied to money. Two completely different things. A bad teacher in a school can be identified and fired by the administrators. With national merit standards, there's little to stop a bad teacher from pushing out under-performers in her class or easing the curriculum to achieve her "student growth objectives" and receive a nice raise.

[sandnsea]

14. No it is not

You still have not read.

[jpgray]

17. I have. The teacher and principal decide whether the standard is met, no?

# When SGO results are complied, the employee and their supervisor must review the results. Final SGO results must be approved by employee's principal or supervisor and the final decision must be entered by the principal or supervisor to the online system to be eligible for the SGO incentive payout.
# The final decision is one of the following:

* Met
* Not Met
* Pending

# If the principal or supervisor reviews the SGO and determines that the teacher has completed the SGO to satisfaction, the principal or supervisor will mark the SGO as "Met" in the system.

How does one check whether it has, actually, been met?

[sandnsea]

19. By monitoring the program

throughout the year - along side the teacher and your kid.

[jpgray]

21. And if your parent is out of it, tough shit for you, kid

I believe this is the same shit disadvantaged kids have gotten from national "standards" programs since time out of mind.

[sandnsea]

24. Those schools are already shit

So those kids have got nothing to lose anyway.

And then see above.

[jpgray]

27. So in other words, scholastic opportunity will -still- be determined by the color of one's skin

And the income of one's parents.

[sandnsea]

28. That's exactly what I said

:eyes:

and that attitude is exactly why parents get sick of teachers and support anything, anything, to make them accountable to something besides their damned egos.

[ThomCat]

37. So if they have nothing to lose,

might as well spend a whole lot of money on some new program that claims to judge teachers, and might be just another boondoggle forking money into certain people's pockets.

Are you getting a commission? You push this better than some sales people.

Before any program is pushed onto our teachers as Mandatory, it better be backed up with the results it claims it will produce. Not "it's better than nothing."

Has it been tested anywhere and produced results? Has it been written up in any educational journals? Have any studies of it's effectiveness been done? While you're busy pushing this program, have you done anything more than push marketing material at people?

[olegramps]

77. I judge the failures by the failure of their parents; not the teachers.

[Hello_Kitty]

39. Why don't PARENTS get an annual review?

Why don't we hold their tax deductions and credits in abeyance pending scrutiny of whether or not they did their due diligence in ensuring that their offspring are properly fed, disciplined, and prepared for school? Really, why should we throw tax dollars at incompetent parents? :shrug:

[Lilith Velkor]

53. This should be an OP

I double dog dare ya. :popcorn:

[CTyankee]

78. Could it be that some of these incompetent parents themselves had incompetent parents?

Some of these parents simply don't know how to parent successfully because their own parenting was a disaster. We try to help these parents by stressing the importance of reading to your child, but if the parent can't him or herself read, isn't that just irrelevant? Add to that the chaos, violence and dysfunction in so many poor neighborhoods and you would end up having to indict whole communities and then society as a whole for letting this happen.

[olkaz]

41. That's a helluva dance you're doing.

/sigh

[lunatica]

42. Teachers get evaluations

I know they do in Child Development Centers.

[Zodiak]

43. I find it interesting that this debate has raged on in so many threads

And not one person has suggested coming up with an effective metric for gauging a school's/teacher's performance BEFORE imposing consequences/rewards for the results of said measure.

To me, everyone is putting the cart before the horse.

First, come up with an effective measure that all can agree on....and that is best handled by testing different methods over a period of time.

THEN you put consequences to exceeding or under-performing relative to that measure.


I want to also say (because I rarely post, so I'll getting all of my opinion in here) that too many people in America have been effectively brainwashed over the last 30 years to look to the teachers first for blame. Every time education comes up as an issue, the focus is always on the teachers....gotta get rid of the "bad apples". This is not and has never been the crux of the problem. Focusing on the teachers is a RW frame that has gone on too long.

Fund the schools, give parents more time at home, quit voting non-educators into school boards (Church pastor and Chamber of Commerce bigwigs, for example), and give up the anti-intellectualism. Fix any of these problems that have never been addressed and watch the school districts thrive. Keep focusing on punishing teachers instead of supporting them and watch the system continue to deteriorate.

By the way, I was a massive problem child in school....hated half of my teachers, and they deserved it. The ones I hated were the ones that the system coddled for political reasons (married to local bigwigs, went to the right church, etc.). Absolute control freaks who played favorites with the kids. A merit pay system based on local administrator control would only result in more of the type of teacher I hated. No, I am not for this at all, BUT we do need a good metric for measuring the performance of schools so we can put resources where they need to be.

[nichomachus]

44. Because your bosses aren't politicians who are stealing all the money for themselves

And that's the bottom line.

If teachers are subject to this charade, their pay will be determined by the most crass politics you've ever seen.

They need protection from an arbitrary, capricious, and corrupt system.

When I was a teacher in a state college, we worked for three years without a contract -- or a raise of any kind. Finally, the faculty threatened that they wouldn't attend commencement if we didn't have a contract -- which would be fatally embarrassing for the school -- so the state relented and we got a new contract. Three weeks after commencement, the governor reneged on the new contract.

[ieoeja]

45. Annual reviews are stupid for ALL jobs.

The American businessman who built the Japanese business model, I forget his name, said the biggest mistake made by US businesses is the annual review. It excuses management ignoring their principal job of managing employees for 11 months out of the year. It creates undue pressure that one month. And metrics selected at the beginning of the year may be completely meaningless by year's end.

I got nailed big time by that last. I was the only person in my company's history to fail a review.

Think about that. In sixty years I am the only person to fail a review. What more proof do you need that the entire process is silly. People should be failing these things all the time, but nobody does.

Every single project I was assigned at the beginning of the year got cancelled for reasons having nothing to do with me. That year, like most at that time in my life, my boss estimated I was once again responsible for about 90% of the entire IT department's production. But I got the perfect negative score by failing to achieve a single goal on my annual review.

It was kind of funny. Kind of sucked being the only person not getting a raise. But they did get rid of that particular method for judging performance as a result. So the review process was judged the true failure, not me.

[Thothmes]

54. Think that was a gent named Deming

[hugo_from_TN]

47. I'm an engineer and every review I've gotten has been subjective.

My manager judges my work subjectively in relation to my peers. He also consults with my peers, folks working under me, and other individuals in outside organizations I work with. Is it perfect? No, but it does a pretty decent job. Managers are responsible for reviewing the people under them and they in turn get judged on how well they do. They are responsible for identifying and rewarding good workers, and also helping substandard workers perform better or move out if necessary.

It seems teachers could be judged with a combination of review by peers and admins. They could sit in on classes periodically and review teaching skills, how teachers handle problem students and disruptions, and how they facilitate a good learning environment. None of that requires standardized testing.

To reduce the impact of interpersonal problems, some reviewers could come from outside that school. Thus a senior teacher or admin at one school would spend a few days reviewing teachers in their school and also ones in other schools where they don't work. Any obvious grudge by a principal or fellow teacher would be mitigated by the outside reviews.

[DemocracyInaction]

51. They DO have an annual review

get your info straight ... and it's been going on since Shep was a pup. It's done by the Principal.

[HiFructosePronSyrup]

52. Teachers have always had evaluations.

Just not on political quick fix whim of some thoughtless politician to appeal to the anti-labor forces and semi-literate public.

[Dinger]

59. How Many In This Thread Are Public School Teachers?

I am.

[WinkyDink]

65. 1971---2002.

[ulysses]

82. hi.

[Rosa Luxemburg]

61. they are appraised already. Is this a war on teachers?

[anneboleyn]

66. University professors DO get reviewed. At our university there is a mini review every year and a

full review the third year and then for tenure and further promotions. These reviews are extremely stringent.

[ShortnFiery]

71. Yes but if those tasked to do the evaluations don't do their homework, high ratings will default ...

to their budies/cronies.

[joeglow3]

72. MAJOR drawback to not using merit based pay

Straight out of college, my wife took a nursing job at a hospital. She loved her work (even though she worked 12 hour night shifts). If she had a light patient load, she would spend her down time helping out other nurses who had more difficult patients. If they needed help at shift change, she would stick around an extra hour or 2. She would volunteer to work holidays so those with kids could spend the day with them. In comparison, almost all of her co-workers would just sit in the nursing area if they had a light patient load. NONE would volunteer to stick around after their shift was over or to work on holidays. She even found another nurse sleeping in a patient’s room (she was caught a couple more times, but was never fired until she was caught stealing drugs almost 2 years later). Well, when her year was up and she got her raise, she quickly realized all these other people got the exact same raise she did.
Needless to say, she lost much of her motivation and eventually left for a private practice, where she has been for almost 8 years now. The hospital did not want to deal with potential for not giving some nurses a raise, so instead of losing them, they continually lose all of their best nurses.
While I agree that it would be more difficult to measure teachers, I don’t think it would be impossible. I talk with parents and administrators at my son’s private school and everyone knows who the better teachers are. There is no abstract testing that is needed. You can look at the kid’s and see the motivation.
On a different note, I have a friend who was a teacher and went back to college to get an accounting degree because of exactly this.
Finally, I agree that those teaching in a lower income area need to be paid a premium (10-20%) to entice more successful teachers.

[Nicholas D Wolfwood]

73. That's why you use individualized growth models.

If your kids perform 5% better with you (or something to that effect), you deserve a reward. That's not the strawman you use, citing heroic efforts to get poorly performing students up to an arbitrary figure, nor is it rewarding teachers of students that were good in the first place. You reward proportional improvement, which is nothing but fair.

[olegramps]

74. No it isn't a reasonable and fair question.

It is predicated on the assumption that all teachers are educating a group of students that are representative of a homogeneous cross section. Since you OP starts out with a fallacy your argument is worthless.

[TimesSquareCowboy]

79. I think there's a big straw man holding up your argument.

No one's suggesting it be a for-profit model. And, as a lawyer, I have to say that am and many other professionals and non-professionals are evaluated based on other than hard metrics like how many widgets were produced. It ain't rocket science, and is even taught in schools.

[Commie Pinko Dirtbag]

80. I advocate standardized tests for the TEACHERS.

Weed out math teachers who don't know what a complex number is, or history teachers who can't tell the two Elizabeths of England apart. Quiz them on the generals of the teaching profession too.

That'd help, I think.

[olegramps]

84. A more effect test would be one for parents.

[crimsonblue]

81. because... because...

teachers perform like the most valuable job ever and should never be questioned because it is the stupid lazy horrible childrens fault they aren't suceeding. :puke:

[caseymoz]

83. Instead of salary for merit, why not a system of merit based bonuses?

Awarded for just that year? Let's get some positive reinforcement instead of threatening teachers.

Moreover, I thought of a way to administer tests that, for the most part, gives fair numbers and which makes it practically impossible to teach to the test. See:

http://journals.democraticunderground.com/caseymoz/17

Unfortunately, I think the system I propose would be fairly expensive, but it's definitely doable, and we're at the point where we must fix education at all cost, completely and immediately. It cannot wait.

[Arkana]

85. It's harder.

Teachers are not as in-control of their jobs as in the corporate world.

[Mass]

86. The issue is not whether you should evaluate the teachers, but HOW?

A few examples that makes it difficult (not impossible, just difficult)

- My SD in a very affluent suburb of Boston has 90 % of its kids go to a 4 year college, 95 % passes the MCAS (MA standardized test), and it was this year among the 100 best high school as defined by one of the newspapers who do these rankings. Now, you could think this is an outstanding HS, but really, if you think about it, what is the role of the teachers and the role of the economical situation of the families (80 % of parents have a college degree. Many earn high 5 figures to 6 figures salaries. Many are from academia world (close to Harvard and MIT), or work in the high tech companies around Boston). How do we measure the impact of the teachers in this school?

- At the other end of the spectrum, the third city in the state, where drug and violence is rampant (thanks to Guliani pushing the drug trafficking out of New York and upnorth), many kids do not eat properly, many families do not have a high school diploma, and the city is bankrupt, not able to pay teachers a decent salary (in fact $7,000 less than in the next town across the Connecticut River, as of last year). Of course, the MCAS test scores are commensurate with the situation, and a lot fewer kids go to 4 year college, not even community college. This does not mean that the teachers who still work there do not work their ass off. But everything is against them, from the family situation of many of these students to the means that the city can and will allow to the School District and how it will be distributed among the schools.

In both of these schools, there are good and bad teachers. How do we measure that?

Another example: School District not wanting to help teachers, (or imposing pacing or methods that are inefficient for some kids or in some school situations). How do you evaluate teachers who either get good results against district policies, or stick to the policy against their better judgment (if they are not tenured, for example), and do not get good results.

These are only a few examples that make the system difficult. Yes, we all want to close the gap, but closing it means understanding the multiple causes that create success. I am not sure we are there yet. If we did, we would not have standardized testing deciding kids graduations. Some kids will not graduate because they do poorly on multiple choices exam, when another format would allow them to show their knowledge.