authenticity in language testing

Authenticityinlanguagetesting:iczEnglishCentre,ingtheintroduction,SectionIItakesalookattheconceptofauthenticityandthewaythisnotesthat,althoughourunderstandingofthenotionofauthenticityhasdevelopedconsiderablysinceitwasfirstintroducedintothelanguagetestingliteratureinthe1970s,tempttoaddressoneoftheoutstandingissues,SectionIIIsthattesttakersarewillingandar,theseattributesdonotnecessarilyincludeauthenticitywhichhashithertobeenconsidericleconcludesthatmuchmoreresearchisneededifthenaductionAnyattemptatthecharacterizationofauthenticityinrelationtoassessmenttheoryorpracticeneedstoacknowledgethatthenotionofauthenticityhasbeenmuchdebatedbothwithinthefiiedlinguisticsthenotionemergedinthelate1970satthetimewhencommunicativemethodologywasgainingmomentumandtherewasagrowinginter-estinteachingandtesting‘real-life’raleducation,ontheotherhand,hentherehasbeenmuchoverlapinthewaythetermhasbeenperceivedinbothfields,yetthe‘debates’haveremainedlargelyindependentofeachothereventotheextentthat,inarecentarticle,CummingandMaxwell(1999:178)attribute‘[t]hefirstformaluseoftheterm‘authentic’ArchbaldandNewmann(1988)’(myemphasis).Althoughmanydifferentinterpretationsofauthenticityandauth-enticassessmenthaveemerged,onefeatureofauthenticityuponAddressforcorrespondence:JoLewkowicz,AssociateProfessor,EnglishCentre,TheUniver-sityofHongKong,PokfulamRoad,HongKong;e-mail:jolewkowȰuageTesting200017(1)43–640265-5322(00)LT171OA©2000Arnold

44Authenticityinlanguagetestingwhichtherehasbeengeneralagreementovertimeisthatitisanimportantqualityfortestdevelopmentwhich‘carriesapositivechar-ge’(Lynch,1982:11).Morrow(1991:112),inhisdiscussionsofcommunicativelanguagetesting,pointedto‘theoverridingimpor-tanceofauthenticity’,whileforWood(1993)(1993:233)hasproposedthattherearetwomajorissues–thoseofvalidityandreliability–andthatthey‘coalesceintooneevengreaterissue:enticity’.BachmanandPalmer(1996),too,guethatitis‘acriticalqualityoflanguagetests’(p.23),onethat‘mostlanguagetestdevelopersimplicitlyconsiderindesigninglanguagetests’(p.24).AuthenticityisalsopivotaltoDou-glas’(1997)considerationofspecificpurposetestsinthatitisoneoftwofeatureswhichdistinguishessuchtestsfrommoregeneralpur-posetestsoflanguage(theotherfeaturebeingtheinteractionbetweenlanguageknowledgeandspecificpurposecontentknowledge).Thesamepositivesentimentisechoedinthefieldofgeneraleducationwhereauthenticassessmenthasbeen‘embracedenthusiasticallybypolicy-makers,curriculumdevelopersandpractitionersalike’,beingseenas‘adesirablecharacteristicofeducation’(CummingandMaxwell,1999:178).Despitetheimportanceaccordedtoauthenticity,therearthatauthenticityisimportantforassessmenttheorists,tknown,forexample,ethatauthenticityisvariablydefisounclearwhetherthepresenceorabsenceofauthenticitywillaffecttesttakers’nandPalmer(1996:24)sug-gestthatauthenticityhasapotentialeffectontesttakers’r,thiseffectisamongthosefeaturesofauthenticitywhichhavetobedemonstratedifwearetomovefromspeculationaboutthethiscanbeachieved,aresearchagendainformedbyourcurrentunderstandingofauthenticityandanidentifiend,thisarticlefirstreviewsthecurrentauthenticitydebatewithinthefit-ifiesarangeofquestionswhichneediclethengoesontooutlineinsomedetailastudywhichsetsouttoaddressoneofthequestionsidentified,andtosuggestthatthereisnotonlyaneedfor,butalsovaluein,asystematicinvestigationofauthenticity.

iczIITheauthenticitydebate1Theearlydebate45Inappliedlinguisticstheterm‘authenticity’originatedinthemid1960swithaconcernamongmaterialswriterssuchasClose(1965)andBroughton(1965)thatlanguagelearnerswerebeingexposedtotex(1965),forexample,stressedtheauthenticityofhismaterialsinthetitleofhisbookTheEnglishweuseforscience,whichutilizedaselectionofpublticityatthetimewasseenasasimplenotiondistinguishingtextsextractedfrom‘real-life’otuntiltheoducedthedistinctionbetween‘genuineness’and‘authenticity’oflanguagearguingthat:Genuiticityisacharacteristicoftherelationshipbetweenthepassageandthereaderandhastodowithappropriateresponse.(Widdowson,1978:80)Widdowson(1979:165)sawgenuinenessasaqualityofalltexts,whileauthenticityasanattribute‘bestowed’iew,authenticitywasaqualityoftheoutcomepresentiftheaudiencecouldrealizetheauthor’sintentionswhichwouldonlybepossiblewheretheaudiencewasawareoftheconventionsemployedbythewriterorspeaker(Widdowson,1990).Hearguedthatgenuinetextswouldonlybeconsideredauthenticafterundergo-ingaprocessofauthentication,apredtoaccountforthewaylanguagelearnerscouldprogresstowardsbeingabletoauthenticatetexts,r,indis-tinguishingbetweengenuinenessandauthenticity,Widdowsondrewattentiontotheimportanceoftheinteractionbetweentheaudienceatinctionbetweengenuineandauthenticlanguagewasnotreadilyaccepted(apointrecentlylamentedbyWiddowsonhimself;Widdowson,1998),andthediscussionofausequallytrueinlanguageteachingasitwasinthefidvocatingchangetopre-communicativetestingpractices(suchasRea,1978;Morrow1978;1979;1983;1991;Carroll,1980)equatedauthenticitywithwhatWiddowsonidentifiedasgenuineinputandfocusedontheneedtousetextsthathadnotbeensimplifiedand

46Authenticityinlanguagetestingtasksthatsimulatedthosethattesttakerswouldbeexpectedtoper-formin‘therealworld’derstandingofauthenticity,detailedinMorrow’sground-breakingreportof1978,graduallybegantofi1981,forexample,inresponsetoMorrow’s(1978)report,theRoyalSocietyofArtsintro-sthefirstlarge-scaletesttofocusonstudents’abilitytouselanguageincontext(languageuse)ratherthantheirknowledgeaboutalanguage(languageuse)(Hargreaves,1987);itwasalsotheprecursortotheCertificatesinCommunicativeSkillsinEnglishintroducedinthe1990sbytheUniversityofCambridgeLocalExaminationSyndicate(UCLES).Boththesetestswerepremisedonthebeliefthatauthenticstimulusmateriepremiseinformedthedevelopmentofmanyothertests,particularlyinsitu-ationswhereorallanguagewasbeingassessedandsimulationsofreal-lifetasksbecameapartofdirecttestsofspokenability(oficiencyInterviews)andwherelanguageforspecificpurposeswasbeingassessed,suchasintheBritishCouncil/UCLESEnglishLangu-ageTestingService(ELTS)testbattery(formoredetail,seeAldersonetal.,1987).Thisconceptualizationofauthenticitygaverise,however,,byequatingauth-enticitywithtextsthathadnotbeenalteredorsimplifiedinanyway,adichotomywascreatedbetween‘authentictexts’thatwereseenasintrinsically‘good’and‘inauthentictexts’producedforpedagogicpurposeswhichwereseenas‘inferior’.Thisdichotomyprovedunhelpfulsiroducedintherealworlddiffer(interalia)incomplexitydependingontheirintendedaudienceandtheanativespeakersnecessarilyunderstandalltexts(Seliger,1985).Learningtodealwithsimpletextsmay,therefore,beanaturalstageinthelearningprocessandonethatstudentsneedtogothrough(Widdowson,1979;Davies,1984).Usingsuchtextsinatestsituationmaysimilarlybeconsideredthemostappropriateforthelanguagelevelofthetesttakersand,hence,maybetotallyjustifition,everytextisproducedinaspecificcontextandtheveryactofextractingatextfromitsoriginalsource,evenifitisleftinitsentirety,couldbesaidto‘disauthenticate’itsinceauthenticity,accordingtoWiddowson(1994:386),is‘non-transferable’.Inatestsituationwhere,asDavies(1988)pointsout,itmaynotbepossibleorevenpracticaltouseunadaptedtexts,anobviousdilemmaarises.

icz47Howshouldsuchatextberegarded:authenticbecauseithasbeentakenfromtherealworld,orinauthenticasithasbeenextractedfromitsoriginalcontextfortestuse?Anotherareaofconcernrelatedtothsksare,bytheirverynature,n,atbest,bemadetolooklikereal-lifetasks(Spolsky,1985).Testtakersneedtocooperateandbewillingtoabidebythe‘rulesofthegame’ifsimulationsaretobesuccessfulintestingsituations,otherwisethevalidityandfairnessoftheassessmentproceduresremainsuspect(Spolsky,1985).Inaddition,real-imitednumberofsuchperformance-typetaskscanbeselectedforanygiventest;additionally,thequestionoftaskselectionforgeneralizationstobemadefrom(1979)suggestedcharacterizingeachcommunicativetaskbytheenablingskillsneededtocompleteitandthendproach,asAldersonnoted(1981),assumedthatenablingskillscanbeidentifiencouragedthebreakingdownofholistictasksintomorediscreteskills,whichMorrow(1979)himselfrecognizedasproblematicsincea‘candidatemayprovequitecapableofhandlingindividualenablingskills,andyetprovequiteincapableofmobilizingtheminausesituation’(p.153).Throughoutthe1980stheauthenticitydebateremainedfirmlyfocusedonthenatureoftestinputwithsatecentredonthedesiredqualitiesofthoseaspectsoflanguagetestswhichtestsetterscontrol,withadvocatesofauthenticitypromulgatingtheuseoftextsandtaskstakenfromreal-lifesituations(Morrow,1979;Car-roll,1980;Doye,1991),andthescepticsdrawingattentiontothelimitationsofusingsuchinputandtothedrawbacksassociatedwithequatingsuchinputwithreal-lifelanguageuse(Alderson,1981;Dav-ies,1984;Spolsky,1985).2AreconceptualizationofauthenticityInlanguageteachingthedebatewastakenforwardsbyBreen(1985)whosuggestedthatauthenticitymaynotbeasingleunitarynotion,butonerelatingtotexts(aswellastolearners’interpretationofthosetexts),rewattentiontothefactthattheaimoflanguagelearningistobeabletointerpretthemeaningoftexts,andthatan

48Authenticityinlanguagetestingproposedthatthenotionofauthenticitywasafairlycomplexoneandthatitwasoversimplistictodichotomizeauthenticandinauthenticmaterials,particularlysinceauthenticitywas,inhisopinion,n,intheearly1990s,estedthattherewasaneedtodistinguishbetweentwotypesofauthenticity:situationalauthenticity–thatis,theperceivedmatchbetweenthecharacteristicsoftesttaskstotargetlanguageuse(TLU)tasks–andinteractionalauthenticity–thatis,theinteractionbetweenthetesttakerandthetesttask(Bachman,1991).Insodoing,heacknowledgedthatauthenticityinvolvedmorethanmatchingtesttaskstoTLUtasks:hesawauthen-ticityalsoasaqualityarisingfromthetesttakers’n(1991)appeared,atleastinpart,tobereaffirmingWiddowson’snotionofauthenticityasaqualityofoutcomearisingfromtheprocessingofinput,butatthesametimepointingtoaneedtoaccountfor‘languageuse’whichWiddowson’sunitarydefieen(1985),Bachman(1990;1991)alsorecognizedthecomplexitiesofauthenticity,arguinaskcouldbesituationallyhighlyauthentic,butinteractionallylowonauthenticity,conceptualizationofauthenticityintoacomplexnotionper-tainingtotestinputaswellasthenatureandqualityoftestoutcomewasnotdissimilartotheviewofauthenticityemerginginthefinitedStates,inparticular,thelate1980s/early1990ssawamovementawayfromstandardizedmul-tiple-choiceteststomoreperformance-basedassessmentcharac-terizedbyassessmenttaskswhichwereholistic,whichprovidedanintellectualchallenge,whichwereinterestingforthestudentsandwhichweretasksfromwhichstudentscouldlearn(Carlson,1991:6).Ofconcernwasnotonlythenatureofthetask,ghtherewasnosingleviewofwhatconstitutedauthenticassessment,thereappearstohavebeengeneralagreementthatanumberoffactorswouldcontributetotheauthenticityofanygiventask.(Foranoverviewofhowlearningtheoriesdeterminedinterpretationofauthenticassessment,seeCummingandMaxwell,1999.)Furthermore,therewasarecognition,atleastbysome(forexample,Andersonetal.(1996),citedbyCummingandMaxwell,1999),thattaskswouldnotnecessarilybeeitherauthenticorinauth-enticbutwouldlieonacontinuumwhichwouldbedeterminedbytheextenttowhichtheassessmenttasknstructionof

icz49authenticityasbeingsituatedwithinaspecificcontextcanbecom-paredtosituationalauthenticitydiscussedabove.3Astepforward?Thenextstageintheauuageeducation,BachmaninhisworkwithPalmer(1996)separatedthenotionofauthenticityfromthatofinteractiveness,definingauthenticityas‘Thedegreeofcorre-spondenceofthecharacteristicsofagivenlanguagetesttasktothefeaturesofaTLUtask’(BachmanandPalmer,1996:23).Thisdefi-nitioncorrespondstothatofsituationalauthenticity,whileinteractimisebehindthischangewasarecognitionthatallreal-lifetasksarebydefinitionsituationallyauthentic,soauthenticitycanonlybeanattributeofothertasks,thatis,ametime,notallgenuinelanguagetasksareequallyinteractive;r,authen-ticityisinpartdependentonthecorrespondencebetweentheinterac-tionarisingfromtestandTLUtasks;regardingthetwoasseparateentitiesmay,therefore,nly,Douglas(2000)con-tinuestoseethetwoasaspectsofauthenticity,arguingthatbothneedtobepresentinlanguagetestsforspecifioximatethedegreeofcorrespondencebetweentestandTLUtasks–thatis,todeterminetheauthenticityoftesttasks–Bach-manandPalmer(1996)ameworkprovidesasystematicwayofmatchingtasksintermsoftheirsetting,thetestrubrics,testinput,theoutcomethetasksareexpectedtogiveriseto,andtherelationshipbetweeninputandresponse(forthecompleteframework,seeBachmanandPalmer,1996:49–50).Theframeworkisimportantsinceitprovidesausefulchecklistoftaskcharacteristics,onewhichallowsforadegreeofagreementamongtesintoaccountboththeinputprovidedinatestaswellastheexpectedoutcomearisingfromtheinputbycharacterizingnotonlytesttasksbutalsotesttakers’interactionswiththese.4OutstandingquestionsOperationalizingtheBachmanandPalmer(1996)frameworkdoes,however,rminethedegreeofcorrespondencebetweentesttasksandTLUtasks,itisnecessaryto‘firstidentifythecriticalfeaturesthatdefinetasksintheTLUdomain’

50Authenticityinlanguagetesting(BachmanandPalmer,1996:24).fyingcriticalfeaturesofTLUtasksappearstorequirejudgementswhichmaybesimchjudgementshavebeenmade,testspecificationsneedtobeimplementedand,intheprocessofsodoing,thespecifiparti-cularlylikelytohappenduringtestmoderationwhen,asrecentresearchhasrevealed(Lewkowicz,1997),considerationsotherthanmaintainingadesirberememberedthattestdevelopmentisanevolutionaryprocessduringwhichchangesandmodifiangesmay,ultimately,evenifunintentionally,rwords,thedegreeofauthen-ticityoftheresultanttesttasksmayfailtomatchthedesiredlevelofauthenticityidentifiedatthetestspecifirinrealitysuchdifferencesinthedegreeofcorrespondencebetweenatesttaskandTLUtasksaresignifissiblethatifoneweretoconsiderallthecharacteristicsforeachtesttaskinrelationtopossibleTLUtasks(atime-consumingprocess),tskscoulddisplayaconsiderabledegreeofauth-enticityintermsofinputwhileotherscoulddisplaythesamedegreeofauthenticityonlyintermsofoutput,uldfeatureashighlyauthenticintermsofrubricsincethisislikelyto‘beacharacteristicforwhichthereisrelativelylittlecorrespondencebetweenlanguageusetasksandtesttasks’(BachmanandPalmer,1996:50).Theaboveissues,allofwhichrelatetotheproblemofidentifyingcriticaltaskcharacteristics,giverisetoanumberofunresolvedques-tions:1)Whichcharacteristicsarecriticalfordistinguishingauthenticfromnon-authentictesttasks?2)Aresomeofthesecharacteristicsmorecriticalthanothers?3)WhatdegreeofcorrespondenceisneededfortesttasksandTLUtaskstobeperceivedasauthentic?4)Howcantestdevelopersensurethatthecriticalcharacteristicsidentifiedatthetestspecificationstagearepresentintheresultanttesttasksandnot‘eroded’intheprocessoftestdevelopment?Anunderlyingassumptionwhichunderp,however,ationswherelearnershavehomogeneousneedsandwheretheyarelearningalanguagefor

icz51specificpurposes,is(2000)r,incircumstanceswherelearners’needsarediverseandtestsettershaveaverylargenumberofTLUtaskstodrawupon,suchacharacterizationwerepossible,genumberofTLUtaskscharacterizedcouldensurealevelofauthenticityformosttesttasksselected,sincethelargerthenumberofTLUtaskstochoosefrom,themorelikelyitisthatthereadstothefollowingquestions:5)Cancriticalcharacteristicsbeidentifiedforalltests,thatis,gen-eralpurposeaswellasspecificpurposelanguagetests?6)Ifso,dotheyneedtobeidentifiedforbothgeneralandspecificpurposetests?Athirdsetofquestionsrelatestotestoutcome:whethertesttasksthatcorrespondhighlytoTLUtasksintermsoftaskcharacteristicsaasbeensomeresearchinthisareatosuggestthatend-usersmayproveusefulinformantdyinvestigatingoraldiscourseproducedinresponsetopromptsgivenaspartoftheOccupationalEnglishTestforHealthProfessionals,LumleyandBrown(1998)founr,theyalsofoundthatthetasksgaverisetoanumberofproblemswhichrestrictedtheauthenticityofthelanguageproduced,thatis,undthattherolecardsgiventothetesttakersprovidedinsufficientbackgroundinformationabout‘theirpatient’.Asaresult,whendiscussingthepatient’sconditionwiththeexaminer(playingtheroleofaconcernedrelative),uldsuggestthatauthenticityismadeupofconstituentpartssuchasauthenticityofinput,purposeandoutcome,leadingtothequestions:7)Whataretheconstituentsoftestauthenticity,andareeachoftheconstituentsequallyimportant?8)Doestheinteractionarisingfromtesttasksgiverisetothatintendedbythetestdevelopers?9)Towhatextentcan/dotesttasksgiverisetoauthentic-soundingoutputwhichallowforgeneralizationstobemadeabouttesttakers’performanceintherealworld?

52AuthenticityinlanguagetestingThefinalsetofquestionstobeconsidelreadybeensug-gestedthatthesignificanceofauthenticitymay,forexample,possiblethatperceivedauthenticityplaysanimportantroleintesttakers’performance,asBachmanandPalmer(1996)r,itisconceivablethatauthenticityisimportantforsome–notall–uallypossiblethatauthenticityisnotimportantfortesttakers,butitisimportantforotherstakeholderssuchasteacherspreparingcandidatesforatest(seeSectionIII).Weneedtoaddressthefollowingquestionsifwearetoascertaintheimportanceoftestauthenticity:10)11)12)13)Howimportantisauthenticityforthevariousstakeholdersofatest?Howdoperceptionsofauthenticitydifferamongandbetweendifferentstakeholdersofatest?Doesaperceptionofauthenticityaffecttesttakers’performanceand,ifso,inwhatways?Doestheimportanceattributedtoauthenticitydependonfactorssuchastesttakers’age,languageproficiency,educationallevel,strategiccompetenceorpurposefortakingatest(whetheritisahighorlowstakestest)?Willperceivedauthenticityimpactonclassroompracticesandifso,inwhatway(s)?14)Inrelationtothefinalquestion(14),itisworthnotingthemarkedabsenceofauthenticityindiscussionsofwashback(theimpactoftestsonteaching).TheclosetiedrawnbetweenauthenticachievementandautgandMaxwell(1999)goasfarastosuggestthatthereisatensionbetweenfourfactors–learninggoals,learningprocesses,teachingactivitiesandassessmentprocedures–allofwhicharein‘dynamictension’and‘adjustmentofonecomponentrequiressympatheticadjustmentoftheotherthree’(p.179).Yet,literatureo(1997),forexample,inheroverviewofwashbackdorly,Aldersonetal.(1995)–inconsideringtheprincipleswhichunderlieactualtestconstructionformajorexaminationboardsinBritain–ticity,astheaboveoverviewsuggests,,therehavebeentwoparalleldebatesonauthenticitywhichhaveremainedlargelyignorantofeachother.

icz53Discussionswithinthefieldofappliedlinguisticsandgeneraledu-cation–asLewkowicz(1997)suggests–rmore,suchdiscussionsneedtobeempiricallystionsidentifiedearliock(1997:44)hasarguedwithreferencetolanguageteaching:‘researchtodateonthistopicisinadequate,and$furtherresearchisjustifiedbytheimportanceaccordedauthenticmaterialsintheliterature’.Thisneedisequallytrueforlanguagetesting,ectofauthenticitywhichhasbeensubjecttoconsiderablespeculation,butwhichhasremainedunder-researched,isrelatedtotesttakers’lowingstudywassetuptounderstandmorefullytheimportancetesttakersaccordtothistestcharacteristic,andtodeterminewhetstudy1ThesubjectsAgroupof72first-yearstudentsfromtheUniversityofHongKongwereidentifireallfirst-yearundergraduatestudenbeenlearningEnglishfor13ormoreyears.2Thetestestswereselectadministeredfirsttoalltheadeupoffoursections:sentencestructure(15items),writtenexpression(25items),vocabulary(20items)andreadingcomprehension(30items).Thestudentswerefamiliarwiththistypeoftestastheyhadtakensimilarmultiple-choiceteststhroughouttheirschoolcareer;additionally,partoftheUseofEnglishexamination,whichisrequiredforuniversityentrance,ondtestadministeredwasanEAP(Englishforacademicpurposes)testwhich,intermsofBachmanandPalmer’s(1996)testtaskcharacteristics,-dentstookthetestattheendoftheirEnglishenhancementcourse,

54Authenticityinlanguagetestingthecoursebeingdesignedtohelpthemcopewiththeiracademicstudies(EnglishbeingthemediumofinstructionattheUniversity).Itwasanintegrated,softesttype,studentsweresomewhatfamiliarwithintegratedtestssinceoneofthepapersoftheUseofEnglishexamination(theWorkandStudySkillsPaper)Ptestwasalsomadeupoffoursections:1)2)3)4)listeningandnote-taking;listeningandwriting;readingandwriting;andsynthesizing,selectingandorganizinginfirstpartofthetest,poseofthistaskwastoprovidethestudentswithinformation,anditwas,therefore,econdpart,studentsufidentswererequiredtosummarizethemainpointusingacademictoneandthencommentontheextract’ondextractwaslonger(approximately450words),tswererequiredtocompletesomenotesonaspecififinalpartofthetestrequiredstudentstowriteanacademicessaofthisfinalpartwastohavestudentsintegrateinformationfromavarietyofsourcesinacoherentmannerandthencommentonthesubjectinquestion.3TheprocedureImmediatelyaftereachofthetests,whilethestudentsstillhadthetest-takingexperienceinmind,estionnairewasdesignedtoelicitthefollowinginformation:1)students’perceptionsofwhateachsectionofthetestwasassessing;2)students’perceptionsofhowtheyhadperformedonthetest;3)students’opinionsofhowwelltheirperformancereflectedtheirabilitytouseEnglishinanacademiccontext;4)students’perceptionsoftheadvantagesanddisadvantagesofthetesttype.

icz55Inaddition,afterthesecondtest,studentswereaskedtocomparethetwotestsintermsof:1)whichintheirviewwasabetterindicatoroftheirabilitytouseEnglishinanacademiccontext;2)whichtheyconsideredmoreaondquestionnairewhichincludedpoints(5)and(6)above,heprimarypurposesofthisquestionnairewastoelicitr,direct,structuredquestionyalsohavhereforedecidedtouseanopen-ended,unstructuredques-tionwhichavoidedalluseofjargonorcomplexterminology(question4).Theotherquestionsofinteresthere,forwhichresultsarereportedinthisarticle,arethosewhichaskedstudentstocomparethetwotesttypesandtheirperformanceonthese(questions5and6).ponsestoallthreeofthesequestionswerecollatedandthefre-quencyofresponses,whererelevant,rmore,theresponsesforstudentsscoringinthetopthirdforeachtestwerecomparedtothosescoringinthebottomthirdtoascertainwhetherperformanceonthetestaffectedstudents’responses.4ResultsofthestudyaIdentificationoftestattributes:Todeterminewhichtestfeatureswereimportantforstudents,question4ofthequestionnaireaskedrespondentstoenumefirstpointtonoteisthatthemajorityoftherespondents(approximately60%)appearedwillingfindingisconfirmedbyastatisticalanalysisofthefrequencywithwhichrespondentsidentifiedadvantages/disadvantagesforeachtesttype;thisshowsthatsignificantlymorerespondentsidentifiedthanfailedtoidentifytestattributes(Table1).Achi-squaretestofthedifferencesinthefrequencieswithwhichstudentsfailedtoidentifyadvantagesanddisadvantagesacrossthetwotesttypesshowsnosignificantdifference(Table2).Thisfindingfurtherconfirmsthatstudentswereequallyabletocommentonthepositiveandnegativeattributesofbothtests.

56AuthenticityinlanguagetestingTable1Frequencywithwhichrespondentsidentifiedadvantagesanddisadvantagesofthetwotesttypes(n=72)TesttypeAttributeNumberwhoidentified65465850Chi-squaredfpMCMCEAPEAPPositiveNegativePositiveNegative46.7225.55626.88910.8891111.000.018.000.001Notes:MC=Multiple-choicetest;EAP=Englishforacademicpurposestest(anintegratedperformancetest)Table2ComparisonofthefrequencywithwhichstudentsfailedtoidentifytheadvantagesanddisadvantagesofthetwotesttypesAttributeNumberwhofailedNumberwhofailedChi-squaretoidentify(MC)toidentify(EAP)72614222.732.500dfpAdvantageDisadvantage11.098(n.s.).480(n.s.)ReshepositiveattributesidentifiedwerethecomprehensivenatureoftheEAPtest(notedby17%ofrespondents)andtheapparentusefulnessofthemultiple-choicetest(notedby36%ofrespondents).AmongthenegativeattributesweretheamountofwritinginvolvedintheEAPtest(notedby25%ofrespondents)andthefactthatthemultiple-choicetestwasnotinteresting(notedby8%ofrespondents).However,fewrespondentsnotedtestfeatureswhichcouldbeidentifi72respondents,only9(12.5%)notedexplicitlythatanadvantageoftheEAPtestwasthaer10respondentssawthewritingcomponentasanadvantage,aturehas,however,,thechi-squaretest–comparingthefrequencywithwhichthefeaturewasnotedwiththefrequencywithwhichitwasnotnoted–revealsthatthisdifferencewasstatisti-callysignificantatp=.000(Table3),quencywithwhich‘lackofauthenticity’onthemultiple-choicetestwasnotedasadisad-vantagewashigher,r,whencomparedwiththefrequencywithwhichthisfeatureofthetestfailedtobenoted,thedifference

iczTable3Frequencywithwhichauthenticity/lackofauthenticitywasnotedTesttypeNumberwhoidentified1926Chi-square57dfpEAPMC5.55616.05611.000.018Note:seeTable1forabbreviationswasagainstatisticallysignificant,atalevelofp=.018,suggestingthatlackofauthenticitywasalsoperceivedasunimportantforthemajorityofrespondents(Table3).Attributeswhichrespondentsidentifiedaslikelytoaffecttheirper-formanceincludedtheirfamiliaritywiththetasktype,theirmotiv-ationtodowell(withanumberofstudentsbeingmoremotivatedduringthemultiple-choicetestwhichdidnotcounttowardstheirfinalgrade,butwhichtheysawasgoodrevisionpractice)rseemeddiscouragedbytheamountofwritingrequiredintheintegratedEAPtest,mpareswith19respon-dentswhonoionshipbetweenacademicperformanceandtesttype:Ques-tion5setouttodeterminewhichofthetwoteststherespondentsconsideredtobeabetterindicatoroftheirabilitytouseEnglishinanacademiccontext,thatis,e4shows,thepercentageofrespondentswhoidentifiedthemultiple-choicetestwasverysimilartothepercentagewhoidentifi,14%4Students’perceptionofthetesttypewhichbetterassessedtheirabilitytouseEnglishinanacademicsettingMCEAPBothNoresponseIrrelevantresponse7Percentageoftotal(n=72)EAP:percentageoftopthirdEAP:percentageofbottomthirdMC:percentageoftopthirdMC:percentageofbottomthird36264838383852225Note:seeTable1forabbreviations

58AuthenticityinlanguagetestingStudents’hoscoredinthetop-thirdforeithertestweremorelikelytoselecttheintegratedtestasbetterreflectingtheirabilitytoperforminEnglishinanacademicsettingthanthosewhoperformedinthebottomthird(seeTable4).Amongthereasonsprovidedforselectingthemultiple-choicetestwerethatitassessedawiderrangeofskillsandthatthetestwastakenunderlesspressure,withsomerespondrast,thosewhoselectedtheintegratedEAPtestnotedthattheskillsbeingassesionshipbetweenteachingandtesting:Responsestoquestion6indicatethataconsiderablenumberofstudentsfailedtoperceiveaconnectionbetweenwe5shows,while38%ofthestudentsindicatedthatthemultiple-choicetestbetterassessedtheskillstaught,42%sonsforselectingoneorothertestinthisinstancewereverysimilartothoseforselectingwhichtestbetterassessedstudents’nfavourofthemultiple-choicetestidentifieditscomprehen-siveness,itsobjectivityintermsofmarking,andthefactthatitdidnotincludealisteningcomponfavouroftheintegratedtest,alsonotedit5Students’perceptionofthetesttypewhichbetterassessedwhattheyhadbeentaughtintheirEnglishclassesMCEAPBothNoresponseIrrelevantresponse841308601388Percentageoftotal(n=72)EAP:percentageoftopthirdEAP:percentageofbottomthirdMC:percentageoftopthirdMC:percentageofbottomthird38433938354243354246790134Note:seeTable1forabbreviations

icz595DiscussionMostoftherespondentifiedawiderangeofattributes,butnonewasidentifirmore,ssiblethatstudentsdidnotthinkofnotingaspectsoftestrelevancewhencompletingtheopen-endedquestionand,ifprompted,,studentswereabletoidentifyotherattributes,tudentswerespecificallyaskedtofocusonthoseaspectsoftheteststheyconsideredimportantandlikelytoaffecttheirperformance(somethingtheyareunlikelytodoovertlyundermosttestconditions),itwouldseemplausibletocon-clunlyseenasimportantbysomeoftherespon-dents,andauthenticitywasonlyoneofanumberofattributesthatmayhavebeenidentifissiblethatthisattributeistakenforgrantedbymanytesttakersandnotedonlywhen,contrarytotheirexpectations,rmore,authenticpearstobetrueforthegroupofrespondeultsfurthersuggestthatstudents’perceptionsofwhatatestistestingandhowthatrelerformingwellseemedbet-terabletorecognizetheconnectionbetweentrealsobetterabletoseetherelationshipbetweenwhatwasterresultissdnotnecessarilyhavethesameteacher,butthevariabilityofresponseikelyexpla-nationisthatsomeoftherespondentsassociatelanguageassessmentwithproficiency-typetests,yviewmultiple-choicetestslusionThisstudyislimitedinthatitreliedonstudents’self-perceptionsoftestqualities,andtheresearch,however,showthatvalu-ableinsightscanbeobtainedaboutauthenticityfromtest-taking

60Authenticityinlanguagetestinginformantsandthattheirresponsetotesttasksultsofthestudyalsoraisowthattesttakers’eitisanimportanttestattributelikelytocontributetotheirperformance;forothers,andtheseresultsareinlinewithBachmanandPalmer’s(1996)notionthatstakeholders’perceptionsoftestauthenticitydiffernotonlyacrossbutalsobetweengroupsofstakeholders;atthesametimetheysuggestthattheremaybeamismatchbetweentheimportanceaccordedtoauthenticityticitymaybeoftheoreticalimportanceforlanguagetestersneedingtoensurethattheycangeneralizefromtesttonon-testsitu-ations,henatureoftheinputintermsofthetestsandtheteachingleadinguptothetestwerethesame,factorsotherthanthecorrespon-dencebetweentestandTLUtasksmusthaveaffectedstudents’amewayasvanLier(1996)arguesthatauthenticityofmaterialsusedinteachingcontextsconstituteonlyonesetofconditionsforauthenticitytobediscernibleinthelanguageclassroom,soitwouldappearthattestinputconstitutesonlyonditionssuggestedbythedatarelatetotesttakers’tswhoscoredinthetopthirdoneithertestweremorelikelythanthosewhoscoredwithinthebottomthirdtoidentifytheinte-grareprobablyfurtherconditionswhichaffecttesttakers’perceandimportanceofad,however,appearthatnosingleconditiofconditionsinteractingwiththetestinputwillaffecttesttakers’perceptegreeofcorrespondencebetweentestandtarget-languageusetasksmaybeanecessarybutinsuffitoinvestigatetheneedtoextendtheresearchagendatootheraspectsofauthenticitysothatinthelong

icz61runthequestionspositedinthefiwaythedebateonauthenticitywillbemovedforwardfromonethathasbeenlargelytheoreticaltoonethatisbasedonresearchfiencesAlderson,J.C.1981:rson,hes,A.,editors,Issuesinlanguagetesting::on,J.C.,Clapham,l,D.1995:dge:on,J.C.,Krahnke,nfield,C.1987:ReviewsofEnglishlanguageprofigton,DC:n,L.1990::n,L.1991:Whatdoeslanguagetestinghavetooffer?TESOLQuar-terly25,671–n,LandPalmer,A.1996::,M.1985:dLinguistics6,60–ton,G.1965::n,D.1991:rniaCurriculumNewsReport,16/l,B.J.1980::,R.A.1965::g,well,G.1999:mentinEducation6,177–,A.1984:ValidatingthreetestsofEnglishlanguageprofigeTesting1,50–,A.1988:es,A.,editor,TestingEnglishforuniversitystudy::s,D.1997:Languageforspecifiham,son,D.,editors,7:cht:KluwerAcademic,111–s,D.2000:Assessinglanguageforspecifidge:,P.1991:an,S.,edi-tor,ore:SEAMEORegionalLanguageCentre,103–aves,P.1987:RoyalSocietyofArts:rson,J.C.,

62AuthenticityinlanguagetestingKrahnke,nfield,sofEnglishlanguageprofigton,DC:icz,J.1997:-lishedPhDdissertation,,wn,A.1998:Authenticityofdiscourseinaspecifi,EandJames,G.,editors,ng:TheLanguageCentre,TheUniversityofScienceandTechnology,22–,A.J.1982:‘Authenticity’inlanguageteaching:hJournalofLanguageTeaching20:9–,K.1978:-don:,K.1979:Communicativelanguagetesting:revolutionorevol-ution?InBrumfit,nson,K.,editors,:OxfordUniversityPress,143–,K.1983:TheRoyalSocietyofAran,R.,edi-tor,:CollinsELT,102–,K.1991:an,S.,editor,ore:SEAMEORegionalLanguageCentre,111–k,M.1997:hLanguageTeachingJournal51,144–,P.1978:r,H.W.1985:Testingauthenticlanguage:geTesting2,1–y,B.1985:geTesting2,31–r,L.1996:Interactioninthelanguagecurriculum:awareness,:,D.1997:ham,son,D.,editors,-ume7:cht:KluwerAca-demic,291–son,H.1978::son,H.1979::son,H.1990::son,H.1994:uarterly28,377–son,H.1998:Context,resentedatTESOLAnnualConvention,Seattle,17–21March1998.

icz63Wood,R.1993:dge:ix1QuestionnaireEnglishCentreTheUniversityofHongKongInstructionsons1–ons5and6refertobotyouthinkeachofthefollowingsectionsweretesting?SectionB:..............................................................................................................................................................................................................SectionC:..............................................................................................................................................................................................................SectionD:.............................................................................................................................................................................................................ldoyouthinkyouhaveperformedoneachofthefollow-ingsectionsofthistest?nBabove80%71–80%61–70%51–60%41–50%below40%tebelowhowwellyouthinkyouroverallscoreonthistestwillreflectyourabilitytouseEnglishinyourstudies?Verywellx------x------x------x------x------x------xNotatallSectionCSectionD

completedthistest,whatdoyouthinkarethegoodandbadpointsofthistypeoftest?est,thePracticeTestorthisEnd-of-CourseTest,doyouthinkisabetterindicatorofyourabilitytouseEnglishinyourstudies?Why?est,thePracticeTestorthisEnd-of-CourseTest,doyouthinkbetterassesseswhatyouhavelearnedduringyourEASclasses?ouforyourco-operation.