Computing Optical Flow with Physical Models of Brightness Variation Λ - PDF

IEEE Conferene on Computer Vision an an Pattern Reognition, Hilton Hea, vol. II, pp , 2000 Computing Optial Flow with Physial Moels of Brightness Variation Λ Horst W. Hausseker Davi J. Fleet Xerox

Please download to get full document.

View again

of 8
All materials on our website are shared by users. If you have any questions about copyright issues, please report us to resolve them. We are always happy to assist you.

Small Business & Entrepreneurship

Publish on:

Views: 18 | Pages: 8

Extension: PDF | Download: 0

IEEE Conferene on Computer Vision an an Pattern Reognition, Hilton Hea, vol. II, pp , 2000 Computing Optial Flow with Physial Moels of Brightness Variation Λ Horst W. Hausseker Davi J. Fleet Xerox Palo Alto Researh Center, 3333 Coyote Hill Roa, Palo Alto, CA Astrat This paper exploits physial moels of time-varying rightness in image sequenes to estimate optial flow an physial parameters of the sene. Previous approahes hanle violations of rightness onstany with the use of roust statistis or with generalize rightness onstany onstraints that allow generi types of ontrast an illumination hanges. Here, we onsier moels of rightness variation that have time-epenent physial auses, namely, hanging surfae orientation with respet to a iretional illuminant, motion of the illuminant, an physial moels of heat transport in infrare images. We simultaneously estimate the optial flow an the relevant physial parameters. The estimation prolem is formulate using total least squares (TLS), with onfiene ouns on the parameters. 1. Introution This paper uses physial moels of time-epenent rightness variation in image sequenes to estimate optial flow an physial parameters of the sene. Physial auses of rightness variation inlue hanging surfae orientation with respet to a iretional illuminant, motion of the illuminant, an physial moels of heat transport in infrare images suh as iffusion an eay. We wish to estimate optial flow an the relevant physial parameters. Many omputer vision appliations require aurate estimates of the optial flow fiel. Although stuie extensively [1, 11], reliale optial flow omputation still remains iffiult in many ases. Prolems arise from the omplex physial proesses involve in sene illumination, surfae refletion, an the transmission of raiation through surfaes an the atmosphere [12, 24, 19]. Without a moel of image formation it is not possile to unamiguously relate spatiotemporal rightness to motion. Λ Part of this work was performe while HWH was with the Interisiplinary Center for Sientifi Computing, Heielerg University, Germany. Portions of this work were supporte y the Deutshe Forshungsgemeinshaft (Image Sequene Analysis to Investigate Dynami Proesses). The authors thank M. Blak, B. Jähne, A. Jepson, an O. Nestares for helpful isussions, an C. Gare for proviing the rotating sphere sequene. a Figure 1. Illustration of errors in the optial flow estimation ue to rightness hanges. (a) onstant rightness (orret flow fiel), () exponential eay, () iffusion. Common optial flow tehniques assume rightness onstany. For graylevel images g(x; t), where x =(x; y), the traking of points of onstant rightness amounts to fining a path x(t) along whih image rightness is onstant, i.e., g(x(t); t) = ; (1) for some onstant. Taking the total temporal erivative of oth sies of (1) yiels the well-known rightness hange onstraint equation (BCCE) [12]: g t = g xv 1 + g y v 2 + g t =0; (2) where v =[v 1 ;v 2 ] T =[x=t; y=t] T is the optial flow that we wish to estimate, an g i enotes the partial erivative of g with respet to the oorinate i 2 fx; y; tg. Beause (2) provies one onstraint in two unknowns it is ommon to omine onstraints at neighoring pixels, assuming that the optial flow fiel is loally onstant or affine [1, 2]. This proues a system of linear equations that an e solve using stanar (weighte) least squares [14], or total least squares (TLS) [25, 26]. To further onstrain the estimates, the regions an e extene into time if v is smooth within loal temporal winows [7, 27]. For the enhane rightness hange moels (Setion 3) we onsier here, it is very important that the neighorhoos are extene to more than two frames. With only two frames, one an only moel rightness hanges that are linear in time. If rightness is not onserve, then the optial flow fiel estimate from (2) an e severely iase [4, 17, 18, 24, 3, 20, 9, 19]. Causes of rightness variation inlue moving illumination envelopes, hanging orientation of surfaes uner iretional illumination, an atmospheri influenes in outoor appliations. Other instanes our in sientifi appliations that quantitatively investigate ynami proesses [13]. Figure 1 illustrates the influene of rightness hanges on optial flow estimation with two examples of physial transport proesses in infrare images, namely, exponential eay an iffusion; although the surfae translates in eah ase, the flow fiel that onserves rightness may onverge or iverge. This paper esries a generalize framework for inorporating rightness hanges into motion analysis using physial moels. Brightness hanges are either parameterize as time-varying analytial funtions or y the ifferential equations that moel the unerlying physial proesses. We only require that the rightness variation e linear in the moel parameters, not in the image rightness or in the spatiotemporal oorinates. With this, we otain a linear system of equations that onstitutes a straightforwar generalization of the rightness onstany assumption. Estimates of the parameters an then e otaine using TLS. We show that this proues improve optial flow estimates, an it allows us to estimate aitional information that haraterizes the physial proesses. TLS error ovariane matries [21] are use to quantify the auray of the optial flow an the rightness hange parameters. 2. Previous Work Brightness variations have een moele y [4, 17, 18, 24, 3, 20, 9]. A general framework is propose in [20] where the rightness hange etween two frames onsists of a multiplier an an offset fiel: g(r + ffir) g(r) = m(r)g(r) +(r); (3) where, for notational onveniene, r = x T ;t ΛT enotes a spae-time 3D vetor. It is ertainly true that all hanges etween two images an e moele aoring to (3). However, this approximation only yiels the instantaneous rightness hange, whih oes not allow us to isriminate ifferent physial auses of rightness hanges, or to onstrain the estimation to satisfy partiular physial moels. In relate work on target traking, Hager an Belhumeur [9] omine illumination hanges an pose-epenent geometri image istortions into a parameterize moel. They use roust area-ase regression to fit the image to a linear omination of asis templates (eigenmoels). One isavantage of the approah is that the asis set must e ompute from the target, uner varying illumination, prior to the traking. Also, the resulting parameters speify a loation in the eigenspae of training images, rather than a t (x -1,t -1 ) g( x,)- t g(, t ) x x 0 0 (x 0,t 0 ) nonlinear onstant (x 1,t 1 ) linear Figure 2. Illustration of the generalize moel that allows the ojet rightness to hange within a few images. Solutions of the rightness onstany assumption are onfine to the gray plane (g(x; t) g(x 0;t0) =0). physial moel of the rightness variation. Blak et al. [3] express the hange etween two frames of an image sequene as a mixture of auses, inluing oth motion an illumination effets. But they have not onsiere realisti, time-varying, physial moels. Moreover, their use of mixture moels, roust statistis, an the EM algorithm are omputationally expensive ompare to the linear solution evelope here. The tehniques mentione aove are onfine to rightness hanges etween two images; they o not exploit the physial nature of rightness variation over more than two frames. Our approah generalizes the temporal rightness hanges in ways governe y the unerlying physial proess. By onfining the lasses of permitte solutions to those of physial relevane we onstrain the solutions an simultaneously estimate the parameters of interest. 3. Physis-Base Brightness Variation Given a physial moel, onstraints on rightness variations an e speifie y a funtional relation or, more iretly, y ifferential equations that relate temporal rightness hanges to the spatial image struture. As a generalization of rightness onservation (1), shown in Fig. 2, we efine a temporal trajetory x(t) along whih rightness an hange aoring to a parameterize funtion, h : g(x(t);t)=h(g 0 ;t;a) ; (4) where g 0 = g(x(t 0 );t 0 ) enotes the image at time 0, an a =[a 1 ;:::;a Q ] T enotes a Q-imensional parameter vetor for the rightness hange moel. Without loss of generality, we hoose a parameterization suh that a =0 proues the ientity transformation, h(g 0 ;t;a = 0) =g 0. While a is assume to e onstant within small temporal winows, h is expresse as a funtion of time to e ale to apture nonlinear temporal rightness hanges. Taking the total erivative of oth sies of (4) yiels a generalize rightness hange onstraint equation: where f is efine as: g x v 1 + g y v 2 + g t = f (g 0 ;t;a); (5) f (g 0 ;t;a) = t [h(g 0;t;a)] : (6) With the onstant rightness moel (1), f =0, an (5) reues to (2). Given onstraints like that in (5), our goal is to estimate the parameters of the optial flow fiel v, an the parameters a of the physial moel f. In (5) we use a onstant flow moel for whih the flow parameters are v 1 an v 2. But it is straightforwar to use other linear parameterize moels suh as affine motion [6]. Finally, there are two ifferent ways to speify the form of f. One an ompute f y (6) using a known analytial form of h, or one an hoose f aoring to the ifferential equations of the unerlying physial proesses. We will illustrate this elow in Setions , efore returning to the general formulation in Setion Exponential Deay In ertain instanes, rightness in infrare images is well moele using exponential eay. This ours in appliations where heat is remove from the thin layer on an ojet surfae that infrare ameras are imaging. In many suh ases the rightness funtion h in (4) has the analytial form h(g 0 ;t;») = g 0 exp (»t) : (7) In this ase, the parameter vetor a reues to a salar eay onstant, a =». Therefore, from (6) it follows that f (g 0 ;t;») =»g 0 exp (»t) =»g(x(t);t) : (8) This is the well known ifferential equation of exponential eay. It states that the rate of hange at any time is proportional to the urrent value. Beause f is linear in», wean estimate v an» using linear methos (see Setion 4) Diffusion Another elementary physial transport proess, use to moel rightness hanges in infrare image sequenes is iffusion. If the rightness hange an e moele as isotropi iffusion in the image oorinates, then it epens on the spatial image struture aoring to the well known iffusion equation: g t = D 2 2 = f (9) where D is the salar iffusion onstant. Beause f is linear D, we an use linear methos to estimate v an D (Se. 4) Moving Illumination Envelope Brightness hanges ause y moving, non-uniform illumination envelopes have een onsiere in the 2-frame ase [20]. Here we fous on illuminants with a relatively narrow envelope, suh as flashlights or spotlights. Diffuse shaows provie another ase of interest when they are ast on a surfae the motion of whih we wish to estimate. We moel the image as the prout of an unerlying surfae aleo funtion g that translates with image veloity v, an an illumination envelope E (surfae irraiane) that translates with veloity u: g(x;t) = g (x t v) E(x t u) : (10) The rightness variation of g is ause y the relative motion etween the envelope an the surfae refletane. To haraterize this rightness transformation it is onvenient to use a oorinate frame of referene that is fixe on the unerlying surfae refletane, i.e., x r = x t v. The motion of the envelope relative to this referene frame is given y u r = u v an the image rightness eomes g w (x r ;t) g(x r + t v;t) = g (x r )E(x r t u r ); (11) where g w (x r ;t) is a warpe version of g(x;t) for whih the motion of the surfae refletane is stailize. To parameterize the rightness variation through time, we approximate E(x r t u r ) y a Taylor series, up to seon orer with respet to time, aout the point (x r ; 0): E(x r t u r ) ß E(x r ) t re T u r t2 u T r H u r; (12) where re an H are the graient an Hessian of E at (x r ; 0). Sustituting (12) into (11) yiels a rightness funtion: h = g»e t re T u r t2 u T r H u r Then, from (6), the rightness hange, f, is given y : (13) f = a 1 + a 2 t; (14) where a 1 = g re T u r an a 2 = g u T r H u r. The rightness hange f is linear in the parameters a =[a 1 ;a 2 ] T. When the moving envelope an e approximate y (12) the quarati time-varying moel in (13) reues the ias in optial flow estimates. By omparison, if the envelope were nearly linear, a first-orer temporal moel for h, an hene a onstant moel for f, woul suffie. In either ase, solving for the polynomial oeffiients of the moel in a oes not allow us to separate the exat shape of E from its motion u r. However, it oes provie information aout the omine impat of oth. 3.4. Changing Surfae Orientation The last ase we aress here onerns rightness variations ause y surfae rotation uner iretional illumination. As is well known, even Lamertian surfaes exhiit rightness hanges if the angle etween the surfae normal n an the iretion of inient illumination l hanges with time. Although one might attempt to evenly illuminate a sene to avoi these effets, iretional illumination annot e avoie in most ases. Examples inlue outoor senes in iret sunlight, inoor illumination through a single winow, an exploration of ark senes using a ollimate light soure. In some appliations one might intentionally use a iretional soure to enhane eges while simultaneously traking surfae properties. Given a omination of amient illumination an a fixe, istant, point light soure from iretion l (where jjljj =1), the surfae raiane from a Lamertian surfae with unit normal n an e expresse as n T l, where 0 is the amient omponent an 1 is proportional to surfae aleo. If we assume a rotating oy, then we an write the surfae normal at time t as n t = R t n 0, where R t is a 3D rotation matrix an n 0 is the normal at time 0. Then, the timevarying raiane eomes n T 0 R T t l. The extent to whih the raiane hanges with time epens on the angle etween the light soure iretion l an the axis of rotation l 0. To see this, let l = ff l 0 + fi l 1, where jjl 0 jj = jjl 1 jj =1, ff = l T l 0, an ff 2 + fi 2 =1. With this, one an show that the time varying raiane eomes L(t) = ff n T 0 l 0 + fi n T 0 R T t l 1 : (15) Finally, with some manipulation of the seon term, one an show that this reues to the general form of L(t) =» 1 +» 2 os (!t + ffi) ; (16) where» 1 = ff n 0T l 0,» 2 = 1 fi, an! enotes the frequeny of the temporal moulation whih epens iretly on the rate of ojet rotation. It is ovious from (16) that raiane, an hene image rightness, is not linear in parameters of interest [!; ffi]. However, all possile angles etween visile (opaque) surfaes an the illumination iretion are onfine to the interval [ ß=2;ß=2]. Within this interval the osine an e approximate y a seon-orer polynomial, whih provies our rightness funtion that is linear in its parameters: h(g 0 ;t;a 1 ;a 2 ) ß g 0 1+a 2 1 t + a 2 t ; (17) where the parameters a 1 an a 2 are funtions of! an ffi. Using (17), f an e approximate y f = g 0 a 1 +2g 0 a 2 t; (18) whih is linear in the parameters. One an approximate parameter set, a 1 an a 2, is estimate using (18) in (5), the quarati approximation of h an e fitte to (16) to estimate the parameters! an ffi Generalize Formulation In Setions the parametri rightness hange moels were linear in the parameters a. For the osinusoial rightness hange in Setion 3.4 we approximate the rightness funtion, h, with a seon-orer polynomial in t. In general, all smoothly varying funtions an e loally expane y a Taylor series an approximate y a polynomial of orer Q, an therefore we an assume that h is analyti in a set of parameters a = [a 1 ;:::;a Q ] T without loss of generality. Aoringly, rememering that h(g 0 ;t;a = 0) = g 0, we an expan h as a Taylor series aout a =0: h(g 0 ;t;a) = g 0 + QX k=1 k : (19) Using (19) we an express f, the temporal rightness variation efine in (6), as f (g 0 ;t;a) = h(g 0;t;a) t = QX a k ; (20) k where a k is assume to e onstant through time within loal winows of temporal support. As h is analyti in a we an exhange the orer of ifferentiation to otain the general form of our onstraints: f (g 0 ;t;a) = QX k=1 k =(raf ) T a : (21) That is, f an e written as salar prout of the parameter vetor a, an a vetor ontaining the partial erivatives of f with respet to the parameters a k. 4. Computational Framework In eah of the aove formulations we otain linear onstraints that relate the variales of interest an noisy measurements. The general form of the onstraints, assuming a onstant optial flow moel, an e expresse as Λ T p h = 0; with = (raf ) T ; rg T T ;g t (22) Λ an p h = p T T ; 1 ; p = a T ; v TΛ T ; where p ontains the parameters of interest (the flow fiel parameters an the rightness parameters of h), an p h enotes the homogeneous ounterpart of p. The (Q +3)- imensional vetor omines the image erivative measurements an the graient of f with respet to the parameters a. This form of onstraint is easily generalize from a onstant moel to higher-orer parameterize motion moels [2, 6], suh as an affine moel. Equation (22) is just one onstraint in several unknowns. To further onstrain the parameters, p, we assume that p is onstant within a loal spae-time region. We then use a olletion of N suh onstraints at neighoring pixels in the region to otain a linear system: G p h =0; with G =[ 1 ;:::; N ] T : (23) Assuming IID Gaussian noise in the measurement matrix G, a maximum likelihoo estimate of p is given y the TLS estimator [21, 23], often formulate as the minimum of a p T h GT Gp h p T p : (24) h h In pratie, errors in measuring temporal image erivatives are often larger than errors in measuring the spatial erivatives. Also, erivative measurements at ajaent pixels are often orrelate. Thus, one must renormalize the onstraints efore using TLS to avoi ias in the estimates [15, 16]. Although not always referre to as suh, many reent approahes to optial flow omputation (e.g., [5, 8, 22, 18, 26]) are ase on TLS or relate tehniques. To quantify the measurement error in terms of an error ovariane matrix, we use the Hessian of the negative loglikelihoo evaluate at the TLS estimate, ^p h =[^p T ; 1] T.In [21], this is shown to e H = fl ff 2 n jj ^p M 1 h jj 2 jj ^p h jj 2 (^pt h D^p h ) I Q jj ^p h jj 4 h(^p T h D^p h )^p jj^p h jj2 (M^p + A T ) i ^p T ; (25) where D G T G is a (Q+3) (Q+3)matrix, A ontains the first (Q +2)olumns of G, is the last olumn of G, M A T A, an I Q+2 enotes a (Q +2) (Q +2)ientity matrix. The fator fl is efine as fl = ff 2 = ffn 2 + ff2, where ff 2 enotes the variane of the expete istriution of graients in G, an ffni 2 Q+3 is the ovariane of the IID Gaussian noise. If the signal to noise ratio (SNR) is high (ff 2 fl ffn 2 ), then fl ß 1. The error ovariane matrix suggeste y [21], is then given y ± = H Experimental Results We have applie the tehnique to oth syntheti an natural images sequenes. These inlue sientifi appliations with infrare image sequenes, as well as more onventional omputer vision appliations. In eah ase we measure the optial flow an the rightness hange parameters. We also ompute error ovariane matries, given aove in Se. 4, that serve as onfiene ouns on the estimates [21] Changing Surfae Orientation Figures 3(a,) show two frames from a omputer generate image sequene of a ranomly texture 3D sphere uner iretional illumination. The sphere was renere to e illuminate uner an angle of 45 ffi with respet to viewing iretion, an it was rotating aout a vertial axis through Figure 3. Rotating sphere uner iretional illumination. (a, ) Frames 1 an 5. (, ) Optial flow estimates an unertainty ellipses for the onstant an quarati temporal rightness moels. its enter. The angular veloity of the sphere was varie in several experiments, staying within the spatiotemporal sampling limits impose y the sale of the spatial texture (this allowe us to avoi the nee for a oarse-to-fine estimation strategy in the urrent experiments). The temporal rightness funtion was moele with the quarati approximation to the osinusoial relationship in (17), as esrie in Setion 3.4. For omparison, we also otaine estimates using a loal linear approximation (3) an the rightness
Related Search
Similar documents
View more...
We Need Your Support
Thank you for visiting our website and your interest in our free products and services. We are nonprofit website to share and download documents. To the running of this website, we need your help to support us.

Thanks to everyone for your continued support.

No, Thanks