Speech Transcript - JOINT DISPARITY AND OPTICAL FLOW BY CORRESPONDENCE GROWING

0:00:14	and you
0:00:15	hello
0:00:16	and
0:00:17	going to talk about the signal then an estimation of stereo party
0:00:21	and optical flow
0:00:24	we have a
0:00:25	mean a curve video set up it means we have to calibrate it and synchronise came
0:00:30	observing a scene and producing a stream of
0:00:33	is is is there are images
0:00:35	and the task is to compute
0:00:37	uh
0:00:39	this this party map
0:00:42	can
0:00:47	is there are sir
0:00:48	the middle
0:00:50	i i can show
0:00:51	just
0:00:53	uh the computed disparity map between
0:00:56	uh between they're a pair
0:00:58	and
0:01:02	thank you and the optical flow maps
0:01:05	between a
0:01:06	a consecutive images
0:01:08	and
0:01:09	together with the calibration this kicks that's that's we D scene so
0:01:13	so you you flow it's emotion you of it means that for
0:01:15	each reconstruct point we have that to it that it's but C
0:01:21	so how does it work with a
0:01:23	a simple geometrical
0:01:25	uh
0:01:25	we have a you can cameras
0:01:27	and to a point in three D space
0:01:29	this point project to the images
0:01:32	and
0:01:33	having a the corresponding
0:01:34	all these two images
0:01:36	we can you can reconstruct
0:01:38	C point white a relation
0:01:41	that and
0:01:41	this this point moves
0:01:44	in time T plus one
0:01:45	when not or of creation
0:01:46	to gain project you me
0:01:49	and the from the corresponding from these part we can compute we can reconstruct it
0:01:53	one in three
0:01:54	but we need to know that
0:01:56	points belong want to get or
0:01:57	that
0:01:58	uh
0:01:59	these course
0:02:00	and
0:02:01	therefore we need to compute also the correspondence
0:02:04	i in D
0:02:05	in no uh it of the images
0:02:07	which is
0:02:08	in fact you can flow
0:02:12	uh
0:02:13	so that
0:02:14	that the the task is the plot we we
0:02:17	we are given
0:02:18	we are and points
0:02:20	yeah the the point
0:02:22	X time that's time one
0:02:25	for
0:02:26	and
0:02:27	we should compute
0:02:28	the
0:02:29	one at time T possible one in the next frame
0:02:32	where is really point
0:02:34	means we we have to compute
0:02:37	uh
0:02:39	this optical flow
0:02:40	and
0:02:42	uh this topic of flow in right image in class
0:02:44	or or we can compute
0:02:46	yeah to of no into a in that image and the just in and in the in the second frame
0:02:51	or or we can compute as is per uh optical of in the right frame
0:02:54	and the disparity
0:02:56	i
0:02:57	in the in the second frame
0:02:58	which means that these problems are couple
0:03:01	that we can
0:03:02	a that they usually to each other
0:03:05	a you because we have more constraint
0:03:07	that the un use task because he's user
0:03:11	uh before or
0:03:12	well explained and the output hmmm
0:03:14	uh show you the results that that you can have tuition
0:03:18	well well
0:03:19	but you can expect
0:03:21	uh so
0:03:22	uh this is an input image let image each or is also right image but i don't show it
0:03:27	it looks it looks similar this is the disparity map
0:03:30	or or to it
0:03:31	where my course are close to the camera
0:03:33	and the black is
0:03:35	and that are on mixed pixel
0:03:38	and
0:03:39	uh
0:03:40	this this is this is that
0:03:42	and this is the motion map this is a horizontal component of the optical flow and
0:03:46	vertical component of of the call for what gain or coding
0:03:50	uh new means that
0:03:51	right and down motion
0:03:53	yellow or right means
0:03:55	left and a motion
0:03:57	i and like the video
0:04:02	uh you can see D uh how to this party and motion yeah walls
0:04:06	well
0:04:08	so this basis
0:04:09	the in fact this
0:04:10	this team of
0:04:11	to presentation of the see the
0:04:14	so everything single i see now
0:04:16	or one on a correspondence problem the correspondence problem is one of the fundamental problems in computer vision and approach
0:04:22	leads
0:04:23	stop button
0:04:24	"'cause" we have a lot of ambiguity but
0:04:26	what what else
0:04:28	uh
0:04:29	i for constraint of course
0:04:31	and the
0:04:32	new you we use a constraint that each there and optical flows
0:04:35	some
0:04:36	spatial smoothness
0:04:37	because
0:04:38	never break is to have similar parties
0:04:41	it's two
0:04:41	but not
0:04:42	at occlusions or object and function boundaries and then process
0:04:45	because
0:04:46	and of the disparity and optical flow changes abruptly
0:04:49	except for or or your quick and about to pound
0:04:53	there some solution in the age sorry
0:04:55	is to use an explicit regularization
0:04:57	uh
0:04:58	this
0:05:00	this
0:05:01	so it's typically it to M R F and
0:05:04	partition addition relation
0:05:06	uh which is
0:05:07	and to because it very computationally in in dense
0:05:11	and
0:05:11	moreover it produces
0:05:13	it's very
0:05:14	uh a it it might
0:05:16	produce
0:05:17	the artifacts
0:05:18	which were cost them that were prior model so over the we data
0:05:23	and
0:05:23	this is
0:05:24	not suitable for some applications
0:05:27	i was to be we can
0:05:29	there are other approaches which are discriminative but it's
0:05:32	which just keeps us
0:05:34	gives up but the
0:05:36	i make use part of the solution and finds only the on a bit used to use far
0:05:40	and
0:05:40	we we we for to do to in this way
0:05:43	lee back
0:05:45	or for this purpose we you know that it test seed growing technique
0:05:49	uh the basic idea of this going to technique is that we have a set
0:05:53	initial correspondences
0:05:55	so called
0:05:56	and
0:05:58	uh uh the these
0:05:59	the other correspondences are are found
0:06:02	in a small neighborhood around
0:06:04	these C
0:06:06	and then these
0:06:07	you correspondences are you seats end
0:06:10	the that this way the
0:06:11	the
0:06:12	the growing process continue
0:06:15	and
0:06:16	we have a recently developed a you growing stereo or not not a not but only mister
0:06:22	do you
0:06:23	some of these that is was published in but mean
0:06:25	and a we view be scrolling not work my such way of between
0:06:29	uh a of energy minimization that centre
0:06:32	complete complete D local metal
0:06:34	because the
0:06:35	neighbouring structure of the solution is not leak more completely
0:06:39	it's uh
0:06:40	the the growing process
0:06:41	rick is is the solution implicitly
0:06:44	a the other uh_huh dissolves
0:06:46	a robust against the initial seeds
0:06:49	and it's very fast
0:06:50	do to to search space reduction action it's
0:06:52	and power three
0:06:54	goes to and about to where
0:06:56	and power to is the
0:06:57	the
0:06:58	and and and it's quite is that is the size of the images
0:07:01	so that that the
0:07:02	exhaustive this part space
0:07:05	is of this i
0:07:07	and the the the work of the uh and that it doesn't produce
0:07:11	fully
0:07:11	then so results
0:07:13	we don't match all pixels in the scene but only
0:07:16	set uh
0:07:17	some subset
0:07:19	which is
0:07:20	for for for many applications that is factor
0:07:23	so before i'll explain the seen how how we grow to see the why we
0:07:28	you if how the
0:07:30	uh stammer or row wink our work
0:07:34	uh so is there and we have only two images
0:07:37	left and right and
0:07:38	but say this is the correspondence seat which are
0:07:41	the the the C can be obtained by matching a distinct
0:07:44	uh have for distinctive if you image features
0:07:48	and
0:07:49	uh the growing process
0:07:50	finds the correspondences
0:07:52	in the in the neighbourhood of the C so would it's a four to the right
0:07:56	uh it
0:07:57	performs the local optimization of the image correlation
0:08:02	i still
0:08:03	because
0:08:03	the pixel which matches the best
0:08:05	and if the the correlation is about a trace row then this is accepted
0:08:09	you match
0:08:11	and uh and uh uh where same
0:08:13	and
0:08:14	okay the same
0:08:16	and these are you match found these matches because seat ten
0:08:19	okay
0:08:20	a process with pete
0:08:22	uh in
0:08:23	you can see it for
0:08:24	this is as there are there are with the seat
0:08:27	and the disparity map
0:08:28	scroll
0:08:29	so from a single seat you can grow to be
0:08:32	a large so
0:08:35	uh to up to
0:08:38	uh
0:08:39	with with this with the
0:08:40	scene flow it's
0:08:42	it's
0:08:43	pretty much yeah
0:08:44	similar
0:08:45	we have to grow simultaneously disparity map and optical flow
0:08:50	so we have
0:08:51	well us there oh
0:08:53	to there there is that
0:08:54	time one and uh
0:08:55	time T plus one
0:08:57	and
0:08:58	the seat
0:08:59	it's not
0:09:00	a pair of images at at a pair
0:09:02	points
0:09:03	a course for it the
0:09:04	see
0:09:05	is a correspondence of for or
0:09:09	so it
0:09:10	fully determine
0:09:11	the
0:09:12	that the
0:09:12	the local scene flow
0:09:15	and
0:09:16	uh
0:09:16	we have given it is point to map at time T one from
0:09:20	from stereo matching i ching for from a previous frame
0:09:23	and then the seats
0:09:25	the seats are they and here we used in our implementation by matching harris
0:09:29	or
0:09:30	and
0:09:31	low look look as gonna tracker
0:09:33	of harris point
0:09:35	to obtain
0:09:36	a local of the "'cause"
0:09:39	and then we we the growing process is the same so it
0:09:43	a a it looks in the neighbourhood of the of the
0:09:46	initial C
0:09:47	the it is like a local bic you can five speed in the paper
0:09:50	but the
0:09:51	it locally fines
0:09:52	locally maximise the correlation the correlation measures
0:09:56	uh
0:09:57	the similar to
0:09:58	all all
0:09:59	all
0:10:00	all three
0:10:02	correspondences
0:10:03	and if the correlation is about a racial
0:10:07	exceeds X threshold and
0:10:09	and you match just found
0:10:11	before
0:10:12	a a a a a a it covers this what
0:10:14	and that's it it's
0:10:15	this is that
0:10:16	pretty straightforward extension all of these
0:10:19	we
0:10:20	a of this uh
0:10:22	there are going to work but
0:10:24	it it it works quite nice
0:10:27	so well uh
0:10:29	for for the results we perform a think that the ground two experiments want to have a way to
0:10:34	uh the performance of the outboard in
0:10:36	so we seem to ties the
0:10:39	a or playing with
0:10:40	the tech
0:10:40	texture play
0:10:42	which works moving and we
0:10:44	we can at the
0:10:46	a what with with
0:10:47	with noise
0:10:48	and we also
0:10:50	the D
0:10:51	what
0:10:51	the seem like we texture
0:10:53	and that
0:10:55	the the the conclusion of this expert experiment is that
0:10:58	the
0:10:58	similar in is
0:10:59	estimation of joint formulation of optical flow and disparity
0:11:03	uh a house a lot and how
0:11:05	and uh uh the other work is better than
0:11:08	independence dependence on top to cool well
0:11:11	you can find it doesn't the paper is i'm not cool
0:11:13	in
0:11:15	uh
0:11:16	some
0:11:17	some under real results
0:11:20	uh i again the same left image disparity map
0:11:23	and to a motion the horizontal and vertical components of the of
0:11:27	though
0:11:28	you can
0:11:29	no the is that the was between objects are what
0:11:33	and and they are no
0:11:34	that no smoothing got up there are no
0:11:37	uh
0:11:38	even for a for for you objects which are
0:11:41	close
0:11:42	to each other different that the different motion that the uh the are is are not confusing
0:11:47	there are some and that we that the solution is not fully that that
0:11:51	but the
0:11:52	i
0:11:53	i believe that for many applications is
0:11:55	now
0:11:56	another example
0:11:58	uh the
0:11:59	i was clapping can
0:12:01	and C D
0:12:02	uh
0:12:04	colours so
0:12:06	and that are
0:12:08	and under example is uh
0:12:11	moving camera that was predisposed was
0:12:12	is that the camera
0:12:14	and that the rooms
0:12:15	moving we was mounted on a and that may we can arrange and strolled a tab
0:12:20	street
0:12:21	so we can
0:12:22	a
0:12:24	the result of the T O what you can find here
0:12:29	so the the cameras with be quite right
0:12:31	and that that the C
0:12:32	C is complex they are many
0:12:34	uh a menu objects in three D the various motions
0:12:38	but has to hence the
0:12:39	cars and
0:12:41	the
0:12:43	and in the
0:12:44	they are shot boundaries
0:12:47	a is just a a you objects are
0:12:49	and
0:12:50	nicely lead
0:12:51	these thing which from from the background
0:12:54	you with
0:12:55	in in that and motion
0:12:59	so
0:13:00	uh the conclude a uh i would summarise the problem so the proposed that work
0:13:05	it
0:13:06	a
0:13:06	a large displacement between frames
0:13:08	more we have more than
0:13:09	for a certain pixel in the last
0:13:12	a stick for forty
0:13:13	which is
0:13:14	which is a disaster because it four
0:13:15	uh a or was which you for of the variational
0:13:19	optical flow and of
0:13:20	because the S you in and is not motion
0:13:23	uh that or
0:13:26	bandages is that
0:13:27	i the the which in boundaries are what reserve
0:13:30	uh
0:13:31	as a problem for
0:13:33	hmmm i wouldn't with strong relation they tend to smooth out
0:13:36	clues about every
0:13:38	is but to maps are
0:13:40	for late like what we are the the
0:13:43	but are we have a better result than three by friends there one
0:13:46	for based thing optical flow
0:13:47	yeah you could you could see some flickering is true but the it's a we much better than computing frame
0:13:52	by frame
0:13:54	and in that the other i think is that
0:13:56	uh the other
0:13:57	lost
0:13:57	it to our implementation is
0:14:00	how to sell to team
0:14:01	still are like matlab
0:14:03	it's france
0:14:04	in this or a solution about five or what i have a five seconds it up with that normal
0:14:09	what to P C
0:14:10	and
0:14:11	it means that there was not a significant extra post with respect to a a single step um
0:14:17	uh
0:14:18	uh uh the reason is that a low low covered make a as a low of with a complexity
0:14:24	the the search space
0:14:25	which is to be but the correspondences are sort
0:14:29	is
0:14:30	and how to five for each pixel you have to find that this partly or something vertical to control
0:14:35	three uses two
0:14:36	and squared
0:14:37	the quite the size of the image
0:14:40	and
0:14:40	that well because that's results are not lead that spot seven
0:14:44	we had a nest and version of this song one disciple
0:14:46	a point in that
0:14:48	a a in the by
0:14:50	i wine
0:14:51	uh
0:14:52	for for for processing the store
0:14:53	with with also motion prediction and we have a
0:14:56	where in this if you got a paper we have uh
0:14:59	experiments a person with other state-of-the-art
0:15:01	my thoughts in a in a scene flow special the for the topic of
0:15:07	okay thank you
0:15:13	can have some questions
0:15:21	yes
0:15:32	uh i just one thing did you have bet against uh we did but it it has it's and are
0:15:36	if you so that you would i come undone god
0:15:40	now in this case we didn't comparing in a better T as it because uh a the our our or
0:15:45	move does not provide fully dense results and
0:15:48	it's not comparable a few
0:15:51	it you compare only some part of the solution and
0:15:54	to the this party
0:15:56	everywhere
0:15:56	so it's
0:15:57	it wouldn't be fair compare
0:16:04	and other question
0:16:10	the man
0:16:10	a question
0:16:11	you so
0:16:12	for for is uh
0:16:15	the the results are not then so but the if somebody needs to be to have some and we don't
0:16:21	to have a any idea of to make it a
0:16:23	a more dense
0:16:25	okay
0:16:26	the
0:16:27	it that the uh miss some sub or it's how to control the trade-off between density and or right
0:16:32	uh we break her in in for our applications that the
0:16:36	that there are
0:16:38	less errors and
0:16:40	oh it's also has density
0:16:41	which is natural
0:16:43	i have but you can you can slide control
0:16:45	if you relax the artwork and to match then
0:16:48	uh the
0:16:49	uh
0:16:50	then you are getting more more and more illusions send
0:16:54	then in a
0:16:55	do you want to knit some
0:16:57	okay
0:16:57	course you can do some processing
0:16:59	a some
0:17:00	some are are a lot of use
0:17:03	you if you know what you are looking at you can interpolate place some some it you can learn some
0:17:08	so
0:17:09	two for one
0:17:10	uh
0:17:12	this is another way and T
0:17:14	another another option is of course two
0:17:17	to
0:17:17	incorporate the primary goal you mice
0:17:20	similar or to the the
0:17:22	you to my station that
0:17:28	okay there is no question ms
0:17:30	a course again
0:17:34	i

JOINT DISPARITY AND OPTICAL FLOW BY CORRESPONDENCE GROWING

Stereo and 3-D Processing

Presented by: Jan Čech, Author(s): Jan Cech, Radu Horaud, INRIA, France