Speech Transcript - TEMPORAL RECURRENCE HASHING ALGORITHM FOR MINING COMMERCIALS FROM MULTIMEDIA STREAMS

0:00:18	okay are are about the non here one my name's is on you and uh
0:00:23	i'm from national institute of informatics
0:00:25	uh in japan
0:00:26	so today a uh this talk is on our recent work
0:00:30	but i know use a temporal recurrence hashing our reason
0:00:33	uh for mining commercials
0:00:35	from a to a string
0:00:38	so come from mining is an important uh preprocessing task you know you though
0:00:42	uh a cost "'em" at i an can and uh uh market research
0:00:46	it i'm i'm at detecting and localising a
0:00:49	duplicate commercial sequences from a large scale
0:00:52	uh a video archive
0:00:54	or to scheme and a
0:00:55	one month
0:00:56	us archive there could be
0:00:58	chains
0:00:59	oh of uh thousands of uh a duplicate
0:01:02	uh commercial sequences
0:01:04	so many
0:01:05	uh detecting
0:01:06	and the localising these so many uh commercial sequences is too time consuming and uh
0:01:11	uh labour intensive
0:01:13	so an automatic
0:01:14	a commercial mining technique is needed
0:01:18	so one direction of commercial mining
0:01:20	is known in is not each based uh commercial mining it use it the
0:01:25	uh the intrinsic uh
0:01:26	characteristics of commercials
0:01:28	forty as a up to an eight for detecting on
0:01:31	for instance
0:01:33	a a use some can trees
0:01:34	the T V stations may uh at some
0:01:37	one a monochrome
0:01:38	or silence frames uh into that under speech into neighbouring commercial segments
0:01:44	so if we can detect the position of these frames then we can use it to get the
0:01:49	uh duration uh i mean the location of the commercial sequence
0:01:54	so most
0:01:55	and no each based
0:01:56	a techniques are uh efficient
0:01:58	but it's not generate enough because of that data dependent
0:02:02	uh a knowledge they use
0:02:04	uh because these up real knowledge
0:02:06	uh maybe be barry this can trees and time
0:02:11	i where there is one up to an order that can be used for detecting commercial but to view
0:02:16	uh not to be scanned all times
0:02:19	that is
0:02:20	commercials are reputations
0:02:22	so this kept to just take you never change
0:02:24	and uh inspired the
0:02:26	another uh direction known as reputation based commercial mining
0:02:31	so most uh uh reputation based techniques are
0:02:35	and super wide
0:02:36	generate more generate
0:02:38	but uh a lot uh can board
0:02:41	so
0:02:42	in this study we proposed a simple but very effective
0:02:46	a call reason
0:02:47	uh for for the and supervised
0:02:50	uh generate each and out high speed commercial mining
0:02:53	so in this study there is no
0:02:55	uh training that he's not harry provide a before and what do we have a is only a very wrong
0:03:01	uh to tree
0:03:02	and uh the a priest does not depend on any prior knowledge that my
0:03:07	they have can't result time
0:03:10	and uh also the are rings is
0:03:12	very fast
0:03:13	uh for ten hour stream
0:03:15	uh the
0:03:16	person time was only a four seconds
0:03:19	and for one man
0:03:20	a
0:03:21	the time was this simple forty two minutes
0:03:23	and for for five here we do stream the processing time was this some twenty one hours
0:03:28	it's very far
0:03:30	so the proposed
0:03:31	uh uh we're is is
0:03:33	a a two-stage hashing out
0:03:35	and uh i i really should i we explain the a two state one by one
0:03:40	and before explaining the first stage by like to discuss that
0:03:43	difference between commercials
0:03:45	and and new duplicate video
0:03:47	and
0:03:48	a it's you mean know a by definition your to decades
0:03:51	a carriers that's all approach
0:03:54	to to D uh identical videos
0:03:57	it's videos are normally are derived from an original
0:04:00	deal
0:04:01	by means of various is transformation
0:04:03	such as uh and coding your train picture
0:04:06	and something that
0:04:08	and on the other hand
0:04:09	commercials
0:04:10	ah
0:04:11	exact duplicates derived from the original video deal on any transformation
0:04:16	so commercials can be considered as a special case of near to the kate we'd
0:04:21	so this is the main difference between commercials and near to K
0:04:24	and in the case of commercials
0:04:26	uh the fragments
0:04:28	for example the frames all the shops sub shots
0:04:30	oh the but videos can be translated into
0:04:33	uh a a be every compact
0:04:35	uh in
0:04:36	so that identical fragments across that duplicates can be mapped to exactly the same thing to for
0:04:42	so based on this assumption if we insert
0:04:45	the fragments
0:04:46	i into a hash table by regarding the fingerprint as a you but in in X
0:04:50	uh a a hash collision
0:04:51	we occur in the corresponding has pocket
0:04:54	so
0:04:55	uh
0:04:55	the duplicate
0:04:57	uh fragments can be easily detect it
0:04:59	by based on a
0:05:01	uh could each a tech process
0:05:03	so these assumptions
0:05:04	uh
0:05:05	uh not to reasonable for in the case of near to
0:05:08	as but only reasonable for exact duplicates like commercial
0:05:13	so in this study we propose
0:05:15	applying a luminance based fingerprint stage she to the be do stream and also a light
0:05:20	uh
0:05:21	and use it all your hashing
0:05:23	technique to the audio stream
0:05:25	and we didn't
0:05:26	test all
0:05:27	the the you existing techniques but to be believe that the
0:05:31	the proposed our reason is performance
0:05:33	uh you by rent to with a
0:05:35	fingerprint sticky so any existing one can you
0:05:39	so we apply these two fingerprinting straight is two
0:05:43	all frames of the but
0:05:45	and uh
0:05:46	so here the same color indicates
0:05:49	uh frames
0:05:49	uh with the same fingerprint
0:05:52	and uh the canteen hours frames
0:05:54	with the same finger
0:05:56	uh a a and boat into a fraction
0:06:00	so
0:06:00	is a huge hole based on the
0:06:03	the condition shouldn't the hash collision uh a tech that's we can use the to detect duplicate fragments
0:06:08	but please note that
0:06:09	the goal of commercial mining is not to detect
0:06:12	these do you keep experiments but to detect
0:06:14	do P eight
0:06:15	see
0:06:17	a uh the commercial sequence is normally composed of
0:06:20	uh site chains all few hundred all
0:06:23	a fragment
0:06:24	so here we read got
0:06:26	uh that duplicate fragment parents that the basic unit
0:06:29	and a project them to of the time X
0:06:32	uh from this figure we can also so strong temporal consistency
0:06:36	uh among these pairs
0:06:37	for instance the
0:06:39	uh positions of the fragments of consecutive
0:06:41	and of the temporal interval
0:06:43	each in each to fragment
0:06:45	almost the same
0:06:46	so this kind of temporal consistency is very useful for distinguishing
0:06:50	duplicate sequences from non duplicate one
0:06:54	and the the time more time
0:06:56	the commercial mining a task can be formulated into a
0:06:59	searching for
0:07:00	duplicate fragment carrots with high temporal consistent
0:07:04	so one sort of and to this is to a i
0:07:07	a a pairwise matching based on temporal information to all
0:07:11	pairs scope duplicate right
0:07:13	so people and P do you know the number of the
0:07:16	the confusion cost
0:07:17	is in your to the scale and P
0:07:20	and we can see and P stomp be a very very large number
0:07:23	and uh a little or
0:07:25	uh can can in cost can be obtained by of lighting pair
0:07:28	was making based on temporal information
0:07:31	to all
0:07:32	uh all sets
0:07:33	of duplicate fragments
0:07:35	uh so give a and all you noting the that's
0:07:38	oh
0:07:39	this this actually
0:07:40	and all use than then all the
0:07:43	uh a the are of beans in the hash table
0:07:45	and the sake even
0:07:47	and no you know from this so the condition cost
0:07:50	a unit know to the sky or and all it's
0:07:52	so you
0:07:53	uh not
0:07:54	you you shouldn't enough
0:07:55	and uh
0:07:56	the besides size is two solutions there is another interesting
0:08:00	uh study which of flights of fragment only
0:08:04	to with that you paid for and pair
0:08:05	and the
0:08:07	but
0:08:08	in your two and P
0:08:09	uh
0:08:10	which is very efficient
0:08:11	but the because the single operation cost
0:08:15	oh with the
0:08:16	fragment growing straight is
0:08:18	five so the overall process time in this case
0:08:21	almost
0:08:22	a that in the previous of
0:08:25	so
0:08:26	in this study you propose applying a second stage had she to that duplicate that paris
0:08:32	so that's the computer cost
0:08:34	can be in your two
0:08:35	and P
0:08:36	we uh a lower single operation
0:08:41	a you're of B we got each duplicate we can then to pair at the basic unit
0:08:46	and uh we propose
0:08:48	two hash
0:08:48	functions
0:08:49	uh to translate the temporal information into
0:08:52	in prince
0:08:54	so the first fingerprints is the temporal position
0:08:57	a more right
0:08:58	and in this case the just it is that to many it's so that uh uh uh as get of
0:09:03	a fragments
0:09:04	can be a a mapped to the same it is the neighbouring ring finger
0:09:08	and uh the second you print is the temporal interval
0:09:12	between the two of fragment
0:09:14	and uh the is that second
0:09:17	and based on these
0:09:18	to different
0:09:20	all pairs of
0:09:21	uh
0:09:22	a duplicate for not pairs
0:09:24	i insert it into a two dimensional hash table
0:09:27	and the
0:09:28	uh but doing so that duplicate frame paris
0:09:31	with high temporal
0:09:32	is this since C can be ultimate sent into the same
0:09:35	B
0:09:36	so that the time-consuming cameras making can be
0:09:39	avoid
0:09:40	and the to detect a the high temporal sit
0:09:43	we
0:09:44	a a used would uh
0:09:45	recurrence hashing histogram
0:09:47	from the hash table
0:09:49	and uh because that you hate
0:09:51	a a friend of pairs with fight "'em" consistency
0:09:54	have been
0:09:55	but same boat into the same B so this in normally
0:09:58	form a local maxima
0:10:00	uh in this
0:10:01	it
0:10:02	and in this case the been embedding
0:10:04	uh indicates that temporal duration of duplicate
0:10:07	uh second
0:10:09	so you eight
0:10:10	sick is can be easily detected by
0:10:12	uh searching for local mixing
0:10:14	from
0:10:15	uh hashing kids
0:10:17	so this fall for the explanation of the proposed are reason
0:10:20	and uh it's very simple but it's
0:10:23	because we
0:10:24	didn't you ever making on this yeah
0:10:27	each you face and here
0:10:29	and is that um
0:10:30	hash table
0:10:31	so that can see in if the be note and B which is much
0:10:34	a lower than that of related stuff
0:10:38	and we
0:10:39	uh a better at the actress
0:10:41	oh
0:10:42	the proposed a reason by using a ten hour as
0:10:45	and also uh one man
0:10:47	stream and uh five years being were used for evaluating the vision
0:10:52	and uh
0:10:54	we i was
0:10:55	you in both C
0:10:57	and frame level
0:10:58	the sequence level really phase how this side
0:11:01	our
0:11:02	yeah
0:11:03	uh detect and the identify the commercial segments
0:11:06	and of the frame that
0:11:07	in right
0:11:09	uh for precise at that
0:11:10	are can local
0:11:12	a commercial sick
0:11:13	for example four
0:11:14	from each frame the sequence start
0:11:16	and the uh to be trained the sequence in
0:11:20	of the results uh some right
0:11:22	this table
0:11:23	uh we implemented to state of the art that studies for comparison
0:11:27	the first one for a
0:11:29	right
0:11:30	i five ring of light
0:11:31	uh
0:11:32	pairwise matching matching to with a duplicate kate for the pair
0:11:35	and the second one proposed by green
0:11:38	uh
0:11:39	uh of lights
0:11:40	uh pairwise matching to
0:11:42	uh
0:11:43	all sets of right
0:11:45	or we can say all
0:11:46	the nonzero the overall
0:11:47	hash
0:11:48	it's
0:11:49	uh
0:11:49	okay
0:11:51	and uh
0:11:52	able T are straight here in case our camp roll
0:11:55	recurrence hashing of
0:11:57	and a B here in the case of video to mine
0:12:01	and uh for the statistics
0:12:03	uh a P R F
0:12:05	a respectively precision recall or and F one score
0:12:08	uh the sub fix
0:12:10	S
0:12:11	and uh
0:12:13	uh the subjects as
0:12:15	means the frame the sequence table and F for the frame level
0:12:18	find the T here in place of for that simple
0:12:22	so from this table we can see that to
0:12:24	uh our or
0:12:25	uh uh of formant
0:12:26	the baseline
0:12:27	all right
0:12:29	and and especially the much sort of the time
0:12:32	uh them straight i
0:12:35	uh oh
0:12:36	the
0:12:38	besides also implemented a an existing
0:12:41	all you hashing technique for
0:12:43	for reason
0:12:44	and again in this case
0:12:45	uh the
0:12:47	propose are is out of from that the uh baseline
0:12:50	uh uh for all current you have
0:12:52	uh
0:12:53	well yeah i've got introduce that's uh this baseline of light
0:12:57	the frame
0:12:58	uh a role in speech to that you kate
0:13:00	a fragment pair
0:13:02	and uh is not that
0:13:04	uh up to this point to we
0:13:05	in D V really of like a lot to the video and audio stream
0:13:09	so one question here
0:13:11	what we have to you be in the great
0:13:13	these two streams
0:13:15	uh so
0:13:17	uh the we and audio streams can and the reading
0:13:20	uh it
0:13:21	frame level uh in the print the able or is
0:13:25	and for efficiency and the scalability
0:13:28	uh reason we propose in the region
0:13:30	uh and integration add to that is a
0:13:33	so from these table you of the of that's to the sequence level rick or
0:13:38	the sequence that we also almost
0:13:41	one hundred present in both places
0:13:43	so this is my a to you in section variation to combine the detected a commercial second
0:13:49	and that would or the
0:13:51	uh rate of false alarms
0:13:52	uh
0:13:54	results in the number all misses
0:13:57	and on that of the vision is that uh a uh in the case of frame level of innovation
0:14:01	the preceding a is your T five or then the recall in both we do and
0:14:05	all of is
0:14:06	and this of this uh i by task
0:14:09	a union
0:14:09	vision
0:14:10	to combine the detected
0:14:12	commercials
0:14:13	a frame
0:14:14	and uh
0:14:16	so
0:14:17	uh
0:14:18	the results a some fries in this table and we can see that's to
0:14:22	uh the
0:14:23	six as the devil it once for you to
0:14:25	uh ninety eight point one percent
0:14:28	and the frame level if one score in
0:14:30	uh
0:14:31	intro two
0:14:32	ninety seven one four was
0:14:33	uh so
0:14:34	the them and without yeah demonstrated to the vector
0:14:38	oh the proposed uh in separation street
0:14:41	oh whether be applied
0:14:43	uh the i reason to the one month we stream
0:14:46	and the process and find was this and fifty
0:14:49	uh i it's for we do and this then
0:14:52	uh forty to me
0:14:53	uh for audio
0:14:54	so this again them stated that the height of regions
0:14:57	oh
0:14:58	our
0:15:01	find the we applied to
0:15:02	a of you applied our our isn't to a five here year string
0:15:06	this stream was divide
0:15:08	uh in into sixty one month sub streams
0:15:11	and uh our our reason was in D V do already of to each sub string
0:15:16	and we
0:15:17	for formant uh how low computing
0:15:19	and uh
0:15:21	conducted a the recipes
0:15:22	for used fifteen months
0:15:24	for spread
0:15:25	and the the final process time was this and twenty one
0:15:29	oh
0:15:29	you be that the person this you will be don't can see
0:15:32	the commercial mining
0:15:34	commercial detection
0:15:35	we just some of the
0:15:37	to take cost
0:15:38	uh at least to five
0:15:40	five years for us to
0:15:42	so
0:15:43	what i want to say
0:15:44	our our
0:15:45	yeah
0:15:48	so we also conducted some
0:15:50	statistics
0:15:51	for each detected most
0:15:53	so
0:15:54	uh
0:15:55	we hope this that he things could be have
0:15:58	market research
0:15:59	uh and uh
0:16:01	commercial producers and uh uh complete company
0:16:04	so
0:16:05	this example is a beer promotion
0:16:07	uh for this he's around the horizontal axis indicates the final day
0:16:13	and the was co wine case the for a
0:16:15	three
0:16:16	so from this speaker we can see that a for this commercial there was
0:16:20	no broadcast from two em to yeah yeah
0:16:24	and actually this very
0:16:25	somehow somehow go to stand
0:16:27	so i of another one
0:16:29	in this case
0:16:30	the
0:16:31	horizontal axis
0:16:32	also in case that i'm of that
0:16:34	but the what cool one indicates a they a week from a send a man they
0:16:39	to set a
0:16:40	so we can have so of a real data
0:16:43	shape from
0:16:45	actually when be for long as before
0:16:47	a might a
0:16:49	but actually know
0:16:51	uh this very yeah data
0:16:53	shape was also from from all other
0:16:56	i'll call related commercials
0:16:58	and we group of this find a be found the reason
0:17:00	actually this is because in japan
0:17:02	there is of what entry restriction
0:17:05	reach probably bits
0:17:06	the pro house
0:17:07	uh of code or oracle
0:17:08	our whole commercials from
0:17:10	a five yeah
0:17:12	two
0:17:12	five P M
0:17:13	on the
0:17:14	and from five em to you have i am
0:17:17	get
0:17:18	so this is one of them
0:17:20	and a example is of she's actually
0:17:23	and uh we can see that this my thought was
0:17:26	most
0:17:26	a to be row house
0:17:28	a a wrong
0:17:29	five T M
0:17:31	actually a
0:17:32	so this i believe D V believe this is because
0:17:35	uh
0:17:37	five P M is of fine or the white
0:17:39	i i with two
0:17:40	what in or
0:17:41	and the commercial have them
0:17:43	uh no don't need to just by or Z
0:17:45	or something
0:17:47	and the the third example is a
0:17:49	how a commercial
0:17:51	and uh we can also the time zones is high
0:17:54	a broad cross frequency
0:17:56	uh for instance uh problem
0:17:59	six the M two
0:18:00	from six am to
0:18:01	eight A and than most fathers
0:18:03	having reference
0:18:05	and the
0:18:06	a
0:18:06	scalable clock we that stop
0:18:09	and uh after six P M
0:18:11	uh
0:18:12	when most fathers have finished their work and watching T V
0:18:16	um
0:18:17	so
0:18:18	uh actually these observations can also be observed in this figure
0:18:22	and the one explanation of this could be the car
0:18:26	is
0:18:26	okay it towards uh
0:18:29	uh a rather than you know five
0:18:31	and the not the explanation for be
0:18:33	i know you most uh uh a japanese and the father Y
0:18:38	i mean you in the money so the card either
0:18:40	a a pretty for most
0:18:42	uh popular T V vol
0:18:43	for
0:18:44	mail than female what
0:18:47	uh
0:18:48	so so that time limitation i'm going to us keep
0:18:51	the can i like
0:18:54	the
0:18:54	this will you for a kind of change
0:19:10	we
0:19:10	a a i have a as we can jump then use a and not advertising companies going to use your
0:19:16	technique need to to do some the marketing research
0:19:18	uh
0:19:20	currently be no yeah but to be a a considering to contact low companies
0:19:24	and C uh better the are interested in our research
0:19:28	and a normally i
0:19:29	i i i don't think from things are interested T in this but uh
0:19:33	uh actually there are more
0:19:35	some commercial producers
0:19:37	uh who
0:19:39	uh
0:19:40	that would interest in this work and the contract cost we are considering each a about the corporation of the
0:19:46	collaboration
0:19:47	uh
0:19:48	true
0:19:49	maybe we can provide some
0:19:51	uh so is
0:19:52	uh for
0:19:54	so for so and i is or something like that
0:19:56	or most a search yeah
0:19:57	thank you the set point is stuff i i'm standing are doing the commercial we you pattern and discovery right
0:20:03	so uh i think to extend you want to discover some out types also deals
0:20:09	something like a
0:20:10	um
0:20:11	suppose we you all some
0:20:13	do use we do
0:20:14	yeah i and the should be do is a very special types of would be to with a specialised up
0:20:19	at patterns
0:20:20	uh you mean
0:20:21	for detecting is
0:20:23	duplicate be deal yeah all other oh i we maybe
0:20:26	um of L out to depends on the type of that you could be was that you want to detect
0:20:32	for instance
0:20:33	uh in case of sports D also uh
0:20:36	suppose we deal in "'cause" sparse we do they are some rib or cost of the sports aims for video
0:20:41	uh of course
0:20:42	a lot of channels that means in this case the the
0:20:46	the but also almost same but in the news broadcast
0:20:49	the the the the
0:20:49	T V then use program produce a make at some
0:20:52	uh a oh out or
0:20:54	text
0:20:55	captions in to the B also that
0:20:57	the fingerprinting based
0:20:59	straight she is not suitable for in this case
0:21:01	and more robust
0:21:03	uh
0:21:04	technique
0:21:05	is needed so this why i i
0:21:08	i pointed out that that's this work is quite difficult is quite different from
0:21:12	uh do near kate
0:21:13	a detection it's a kind of exact duplicate detection
0:21:17	okay i to so much
0:21:19	thank you

TEMPORAL RECURRENCE HASHING ALGORITHM FOR MINING COMMERCIALS FROM MULTIMEDIA STREAMS

Multimedia Indexing and Retrieval

Presented by: Xiaomeng Wu, Author(s): Xiaomeng Wu, Shin'ichi Satoh, National Institute of Informatics, Japan