Joint Bilateral Upsampling

JicrosoftResearchDaniLischinskiTheHebrewUniversityMattUyttendaeleMicrosoftResearchAbstractImageanalysisandenhancementtaskssuchastonemapping,col-orization,stereodepth,andphotomontage,,forexposure,chromaticity,disparity,labels)ationalandmemoryghgeneralpurposeupsamplingmethodscanbeusedtointerpolatethelowresolutionsolutiontothefullresolution,nstratethatincases,suchasthoseabove,theavailablehighresolutioninputimagemaybeleveragedasapriorinthecon-textofajointbilresultsforeachofthgories:I.3.7[ComputerGraphics]:Three-DimensionalGraphicsandRealism—Color,shading,shadowing,andtextureKeywords:bilateralfilter,lingisafundamen-talimageprocessingoperation,typicallyachievedbyconvolvingthelow-resolutionimagewithaninterpolationkernel,andresam-plingtheresultonanew(high-resolution)g[1990]upsampledinthismannertypicallysufferfromblurringofsharpedges,becauseofthesmoothnesspriorinherentinthelinearinter-polationfir,fortheapplicationscitedabove,additionalinformngthisinformatioosetoleveragethefactthatwicular,wedemonstratethatajointbilateralupsam-pling(JBU)operationcanproduceverygooresultsforstereodepth,imagecolorization,adaptivetonemapping,andgraph-cutbasedimagecomposition.2BilateralFilters1IntroductionAvarietyofnewimageanalysisandimageprocessingmethods,bothautomaticanduserguided,haverecentlybn-cludestereodepthcomputations[ScharsteinandSzeliski2002],imagecolorization[Levinetal.2004;YatzivandSapiro2006],tonemappingofhighdynamicrange(HDR)images[Reinhardetal.2005],andapplicationsofminimalgraphcutstoimagecomposi-tion[Agarwalaetal.2004].Allofthesemethodologiesshareacommonproblemoffindingaglobalsolution:apiecewisesmoothfunctiondescribingsomevalueofinterest(depth,chromaticity,ex-posure,label,etc.)limagescontinuetogrowinsizefromonequartermillionpixelvideoframestomulti-Megapixeldigitalphotos,torecentGi-gapixelimagesarisingfromspecializedcameras[Flint2007]andfromstitchingmultipleimagesintoapanorama[Kopfetal.2007].Suchhighresolutionsposeadifficultchallengeforthemethodscitedabove,whichtypicallyrequireatleastlineartimeand,moreimportantly,,inordertooperateonsuchhighresolutionimages,theymustfiolutionisavailableforthesmallerdownsampledimage,thequestionthenbecomeshowtoupsamplethesolutiontothefullThebilateralfilterisanedge-preservingfilter,originallyintroducedbyTomasiandManduchi[1998].Itisrelatedtobroaderclassofnon-linearfilterssuchasanisotropicdiffusionandrobustestimation[Barash2002;DurandandDorsey2002;Elad2002].Thebilateralfilterusesbothaspatial(ordomain)filterkernelandarangefirmally,forsomepositionp,thefilteredresultis:Jp=1kpq∈Ω∑Iqf(||p−q||)g(||Ip−Iq||),(1)wherefisthespatialfilterkernel,suchasaGaussiancenteredoverp,andgistherangefilterkernel,centeredattheimagevalueatp.Ωisthespatialsupportofthekernelf,andkpisanormalizingfactor,thesumofthef·gfirepreservedsincethebilateralfilterf·gtakesonsmallervaluesastherangedistanceand/lywehaveseentheintroductionofjoint(orcross)bilateralfiltersinwhichtherangefilterisappliedtoasecondguidanceim-˜,forexample,whentryingtocombinethehighfrequenciesage,Ifromoneimageandthelowfrequenciesfromanother[Petschniggetal.2004;EisemannandDurand2004].Thus,Jp=1kpq∈Ω∑˜p−I˜q||).Iqf(||p−q||)g(||I(2)˜ydifferenceto(1)isthattherangefilterusesI2.1ACM Reference FormatKopf, J., Cohen, M., Lischinski, D., Uyttendaele, M. 2007. Joint Bilateral Upsampling. ACM Trans. Graph. 26,

3, Article 96 (July 2007), 5 pages. DOI = 10.1145/1239451.1239547 /10.1145/ght NoticePermission to make digital or hard copies of part or all of this work for personal or classroom use is granted

without fee provided that copies are not made or distributed for profi t or direct commercial advantage

and that copies show this notice on the fi rst page or initial screen of a display along with the full citation.

Copyrights for components of this work owned by others than ACM must be honored. Abstracting with

credit is permitted. To copy otherwise, to republish, to post on servers, to redistribute to lists, or to use any

component of this work in other works requires prior specifi c permission and/or a fee. Permissions may be

requested from Publications Dept., ACM, Inc., 2 Penn Plaza, Suite 701, New York, NY 10121-0701, fax +1

(212) 869-0481, or permissions@.© 2007 ACM 0730-0301/2007/03-ART96 $5.00 DOI 10.1145/1239451.1239547

/10.1145/1239451.1239547PreviousWorkThebilateralfiandDorsey[2002]appliedthebilateralfil-tertoHDRtonemappingandalsodescribedafastapproximation,whichwasrecentlyimprovedupon[ParisandDurand2006;Weiss2006].RamanathandSnyder[2003]usedthebilateralfiethodisre-strictedtoBayerpatternswithafixedsmallupsamplingfactor, Transactions on Graphics, Vol. 26, No. 3, Article 96, Publication date: July 2007.

96-2 • Kopf et etal.[2005]mentionusingabilateralfir,yetal.[2001]upsampethodestimatesanalignmentmapping,andthenuseswarpingandfintbilateralupsamplingyzation:Asimilarlinearsystemtothoseinthetonemap-pingmethodscitedabovearisesinthecolorizationandrecoloringmethodofLevinetal.[2004].Thus,again,alsotrueforthemorerecentcolorizationmethodofYatzivandSapiro[2006],whichdoesnotsolvealinearsystem,mplealow-resolutioncolorizationresult,wefirstconvertitintotheYIQcolorspace(ortoanyothercolorspaceseparatinglu-minancefromchrominance),andthetonemappingexample,onecansDepth:Stereomatchingisafundamentaltaskinimageanal-ysis,whosegoalistodeterminetfferentap-proachestostereomatchinghavebeenexploredovertheyears(foracomprehensiveoverviewsee[ScharsteinandSzeliski2002]).Inmanyofthesemethodsanoptimizationproblemofsomesortissolved,yieldingapiecewisecontinuousdisparityfihniquecanbeusedtoupsamperathersmooth,-cutbasedimageoperations:Severalrecentinteractiveim-ageeditingtechniquesinvolvefimple,theinteractivedigitalphotomontage[Agarwalaetal.2004]systemusesgraph-cutoptimization[Boykovetal.2001]tocomputetheleastobultoftheoptimizationisalabelmap,indiedourj,tchingalgorithmthencomputesalabelmap,whichassignsalabeltoeachoftheremainingunconstrainedpixels,plicationdiffersfundamentallyfromthepreviousones,be-causeherewehaveaquantizedsolution(adiscretenumberofla-bels),rmore,yourtechniqueinthefollowingway:w-resolutionsolutionpixelwithanon-zerobilateralweightvotesforit’5demonstratesourtechniqueforthisapplication.3JointBilateralUpsamplingIncontrasttogeneralpurposeimageupsampling,intheproblemsthatweareinterestedin,additionalinformatiohigh˜,andalowresolutionsolution,S,computedforresolutionimage,Iadownsampledversionoftheimage,weproposeasimplemethodthatappliesajointbilateralfiaistoapplyaspatialfilter(typicallyatruncatedGaussian)tothelowresolutionsolutionS,whileasimilarrangefilterisjointly˜.Letpandqdenote(integer)appliedonthefullresolutionimageI˜coordinatesofpixelsinI,andp↓andq↓denotethecorresponding(possiblyfractional)coordinatesinthelowresolutionsolutionS.˜isthenobtainedas:TheupsampledsolutionS˜p=1Skpq↓∈Ω∑˜p−I˜q||)Sq↓f(||p↓−q↓||)g(||I(3)Thisisalmostidenticaltoeq.(2)withtheexceptionsthatweareconstructingahighresolutionsolutionasopposedtoanimage,,thatq↓oretheguidanceimageisonlysparselysampled,andtheperformancedoesnotdependontheupsamplingfactor(seeSection5).4ApplicationsInthissectionwedemonstratetheusefulnesspping:WiththeincreasingpopularityandutilityofHighDynamicRange(HDR)imaging[Reinhardetal.2005],theretyofsuchmethodshavebeenproposedovertheyears(see[Reinhardetal.2005]foranextensivesurvey).Someofthesemethodsproducehigh-qualityresults,butrequiresolvingaverylargesystemoflinearequations[Fattaletal.2002;Lischinskietal.2006].Althoughthesesystemsaresparseandmaybesolvedefficientlyusingmulti-resolutionsolvers[Szeliski2006],handlingtoday’smulti-megapixelimagesremainsachallenge:oncethedataexceedstheavailablephysicalmemory,ythejointbilateralupsamplingfithelow-resolutionHDRimage,andT(I)respond-inglow-resolutionsolutionisthendefinedasthepixelwisequotientS=T(I)/rwords,thesolutionisanexposuremap,posuremapsaregenerallysmoothbutmayhavedisconti-nuitiesalongsignificantimageedges[Lischinskietal.2006].Thus,attheexposuremapmayhaveasinglechannel(ifonlytheluminancehasbeenadjusted),ormultiplechannels(tosupportarbitrarytonalmanipulations).Figure2showshowapplyinganexposuremapup-sampledusingourtechniquecompareswithanumberofstandard5PerformanceandAccuracyThecomplexityofthejointbilateralupsamplingoperationisO(Nr2)whereNistheoutputimagesizeandristhedomainfiformanceisproportionaltotheoutputsizeandnottotheupsamplingfactor,becausethedomainfiresultswehaveuseda5×5Gaussian,whichisveryfastbuts Transactions on Graphics, Vol. 26, No. 3, Article 96, Publication date: July 2007.

Joint Bilateral Upsampling • 96-33025200,0120,010,0080,00160,00140,00120,x24x48x816x1632x320,0060,0040,00202x24x48x816x1632x320,00080,00060,00040,000202x24x48x816x1632x32JBUNearestBicubicGaussMSEMSETone MappingColorizationMSEDepth from StereoFigure1:MSEerrorprofisignificantlymple,thetonemappertook80secondsfora3.1megapixelimage,whileourupsamplingtookoorizationsolverofLevinetal.[2004]wasevenslower,dabove,duetothememoryissuewecannotrunaveryhighresolutionsolutionsoupsampmagescanbecomputedinasinglesweep,successfullyappliedourmethodtoupsampletonemappingsolutionsformulti-gigapixelimages[Kopfetal.2007].Inourexperiments,wegenerallysetthedomainfilter’sGaussianσdto0.5with5×gefilterGaussianσlowingdefaultvaluesworkedwellfortheimageswetried:colorization,stereodepth,andgraph-cutlabelingsusedimageswithcolorvaluesnormalizedtothe[0,1]in-terval.σr=dthatsettingσ1showsMSEerrorprofiutetheerrors,wecomputedafullresolutionsolution(orsimplyusedthefullresolutioncolorimageordepthmapforcolorizationandstereodepth)downsampledbyfactorsof2,4,8,16,,weperformedupsamplfilterperformedwellatalldownsamplinglevels,and,asexpected,thereprisingly,racticeitoftentuomeoftheapplicationsrequiresomeUI,youneedenoughimageleftto,forexample,uredingeodesicdistance,whiletherangekernelwouldbeovertheGaussianSphere(differencesinnormal).WelookforwardtotryingthejointbilatencesAGARWALA,A.,DONTCHEVA,M.,AGRAWALA,M.,DRUCKER,S.,COLBURN,A.,CURLESS,B.,SALESIN,D.,ANDCOHEN,nsactionsonGraphics23,3(PH2004),294–,mentalrelationshipbetweenbilateralfiltering,adaptivesmoothing,ansactionsonPatternAnalysisandMachineIntel-ligence24,6,844–,Y.,VEKSLER,O.,ANDZABIH,ansactionsonPatternAnalysisandMachineIntelligence23,11,1222–,F.,ANDDORSEY,lateralfinsactionsonGraphics21,3(PH2002),257–,F.,HOLZSCHUCH,N.,SOLER,C.,CHAN,E.,ANDSILLION,nsactionsonGraphics24,3(PH2005),1115–NN,E.,ANDDURAND,nsactionsonGraph-ics23,3(PH2004),673–,ilateralfiansactionsonImageProcessing11,10,1141–,R.,LISCHINSKI,D.,ANDWERMAN,nsactionsonGraphics21,3(PH2002),249–,,J.,UYTTENDAELE,M.,DEUSSEN,O.,ANDCOHEN,nsactionsonGraphics26,3(PH2007).LEVIN,A.,LISCHINSKI,D.,ANDWEISS,nsactionsonGraphics23,3(PH2004),689–NSKI,D.,FARBMAN,Z.,UYTTENDAELE,M.,ANDSZELISKI,nsactionsonGraphics25,3(PH2006),646–653.6ConclusionWehavedemonstratedthebenefitsofajointbilateralupsamplingstrategywhenahighresolutionrapplicationswehaveshownallimproverelativetoprevious“blind”evethisstrategyisample,aglobalilluminationsolutioncomputedoveracoarsesimplifiedmeshcanbeupsampledtoafiainfilter’skernelmightACM Transactions on Graphics, Vol. 26, No. 3, Article 96, Publication date: July 2007.

96-4 • Kopf et ledResultNearestNeighborBicubicGaussianJointBilateralGroundTruthFigure2:ToneMapping:ThelowresolutionexposuremapsolutionatleftisatscalerelativetothefiinsetstotherightshowresultsfromNearestNeighbor,GaussianInterpolation,BicubicInterpolation,JointBilateralUpsampling,atJBUdoesn3:Colorization:insetscorrespondingtofourdifferentupsamplingmethodsareshownontheright(withincreasedsaturation).,S.,ANDDURAND,pproximationofthebilateralfidingsofECCV2006).PETSCHNIGG,G.,SZELISKI,R.,AGRAWALA,M.,COHEN,M.,HOPPE,H.,ANDTOYAMA,lphotographywithflashandno-flnsactionsonGraphics23,3(PH2004),664–TH,R.,ANDSNYDER,lofElectronicImaging12,4,633–RD,E.,WARD,G.,PATTANAIK,S.,ANDDEBEVEC,Y,H.S.,GUO,Y.,HANNA,K.,KUMAR,R.,ADKINS,S.,ANDZHOU,stereocamera:anibrap-pdingsofSIGGRAPH2001,451–TEIN,D.,ANDSZELISKI,-ternationalJournalofComputerVision47,1–3,7–KI,nsactionsonGraphics25,3(-GRAPH2006),1135–,C.,ANDMANDUCHI,ralfidingsofICCV1998,839–,dianandbilateralfins-actionsonGraphics25,3(PH2006),519–G,mputerSocietyPress,LosAlamitos,,L.,ANDSAPIRO,ansactionsonImageProcessing15,5,1120– Transactions on Graphics, Vol. 26, No. 3, Article 96, Publication date: July 2007.

Joint Bilateral Upsampling • 96-5Figure4:StereoDepth:achdetailimageistNeighborBicubicGaussianJointBilateralFigure5:eft:enter:ight:tomrowshowsadetailtha Transactions on Graphics, Vol. 26, No. 3, Article 96, Publication date: July 2007.