Speech Transcript - Characterizing the Response Space of Questions: a Corpus Study for English and Polish

0:00:18	and the last talk of the social beyond characterizing the response place of questions
0:00:23	a corpus of us english and polish
0:00:45	so
0:00:46	relevance
0:00:48	in some sense
0:00:49	all in the sense of a competition coherence
0:00:53	which is amplified here in this example one
0:00:56	where
0:00:57	the on the on the first line of answers are kind of relevant
0:01:01	is that chain you yes it's a louis fourteen replica you
0:01:06	and second line is i
0:01:08	the ones that are not appropriate
0:01:10	so this notion is the cornerstone of fears of dialog the same way that say
0:01:15	grammatic allergies to syntax
0:01:18	and you could argue that basically what the during test is about is exactly relating
0:01:23	to
0:01:25	to relevance and whether that the that that's a good test for
0:01:29	you know when we managed to simulate a human intelligence
0:01:34	so i'm gonna a restrict attention to data corpus study relevance relating to queries
0:01:41	possible responses to queries
0:01:43	and
0:01:44	a bit surprising they perhaps
0:01:47	even if you restricted in this way the been actually very few
0:01:50	comprehensive attempts to characterize it
0:01:53	the some references in a paper and also in some early work that we done
0:01:56	the that a talk about moment
0:01:59	so the some
0:02:01	early
0:02:04	discussion of this in the language acquisition a literature
0:02:07	and some discussion this in some conversational analysis
0:02:12	literature
0:02:13	which is primarily that it to show that there's a at a difference between
0:02:18	three classes on so as not know announces a non responses
0:02:22	and
0:02:24	so in this study were based i've as it looked at n different languages and
0:02:29	show incident gone distributions and
0:02:31	we also flying i'm not the paper they won't korean the child quite different distributions
0:02:36	between say you the results that style but i don't english and the results and
0:02:39	korean but mainly about this
0:02:41	basically about this these three classes on says non ounces and a
0:02:46	a non responses
0:02:49	so today i'm gonna talk about starting by taxonomy that we developed for just characterising
0:02:55	query responses appear responses to queries
0:02:59	something that's sporty characteristic to certain race
0:03:03	and
0:03:06	we then will mention of a basic hypothesis that we used to just scale up
0:03:10	to the general case
0:03:13	talk a bit about annotation scheme in the results
0:03:15	and very briefly
0:03:17	talk about how one might model relevance and what the ready complications then hence
0:03:24	so the starting point of this work in the typology that we developed in the
0:03:29	top level of work of skin myself developed in some work that was published in
0:03:35	the
0:03:36	in two thousand sixteen in the journal of language modeling
0:03:40	and this is a wide-coverage taxonomy for question-question a sequence is was tested on
0:03:46	the bnc child's
0:03:50	the be corpus mammoths corpus
0:03:52	and there was also their formal modelling of the resulting classes in the framework of
0:03:59	our costs t l
0:04:03	so
0:04:04	this study consisted of about fifteen hundred slightly less than computing how to query a
0:04:10	response to as
0:04:12	and what are merged with seven classes of questions what are called elegy classes for
0:04:16	all not all corinne sponsor but for the two people those one
0:04:21	in this study
0:04:22	so we have yet clarification requests
0:04:25	things like hamlet as a response to what time it about
0:04:29	depending questions
0:04:31	so these are things like does anybody want to one m spread a given the
0:04:35	way
0:04:35	where you can do the inference that one question depends on the other whether anybody
0:04:40	wants to ban and strive depends on whether you gonna give it away
0:04:45	a classical motive which questions about underlying motivation what's the matter why
0:04:52	for class
0:04:54	whether responses and the changing the topic
0:04:56	well as you on so always yours
0:04:59	a fixed cost questions the duck a wet whether the you're trying to understand what
0:05:05	which way you're supposed on sit you know what makes black coffee is
0:05:09	which country
0:05:13	and the final to one is questions with the presuppose on so whether
0:05:17	question-response is somehow
0:05:19	indirectly in indicating on the to the first question
0:05:22	and the seven cases where
0:05:24	the
0:05:25	response ignores initial question but still addresses the same situations of things like
0:05:31	do you go you wanna go down have a look at that now what is
0:05:33	what when there and the response is why haven't they finished yet white with of
0:05:37	a is about the workmen so it still about the same situation but it's not
0:05:40	at all responding about the to the question
0:05:44	so that was that those of the seven classes we found the i need to
0:05:49	characterize
0:05:51	cool question response to questions which is about twenty percent of all at least at
0:05:56	the time we found was about twenty percent of all
0:05:58	responses to questions
0:06:00	and a main above hypothesis for this study
0:06:03	e is
0:06:05	that responses drawn from all concerning these class of questions
0:06:10	plus direct indirect on the food
0:06:13	that's going to exhaust the response space of a query
0:06:16	okay
0:06:19	so basically
0:06:20	you get the following kind of scheme
0:06:22	so a response to question can either be a non se and here you have
0:06:28	to subclasses direct on as an indirect on says
0:06:30	and ultimately in the paper also discusses that these actually needs some extras the process
0:06:34	within them
0:06:36	and if it's not i don't on so then it can either be a question
0:06:38	response like we've
0:06:41	already discussed with these seven classes
0:06:43	or it could be a noun so it can be the kind of gone response
0:06:47	so a kind of an acknowledgement
0:06:50	two classes that i'll give an example a second this the i don't know class
0:06:55	and this is difficult to provide response glass
0:06:57	and then declarative responses the about
0:07:00	the these issues that also already all rows in the in the question kind of
0:07:04	response
0:07:05	so the i don't know is this kind of
0:07:08	very not uncommon kind of response where and equally the this is a difficult provider
0:07:15	announce the case and acknowledgement of course you all the very familiar with those guys
0:07:22	so the data for english comes from the bnc
0:07:28	the be corpus and not on the map task corpus
0:07:31	so as you
0:07:32	you probably most of you familiar with these corpora the bnc is a
0:07:37	ta p honestly conversations
0:07:39	be contain speech or dialogues from the class courses
0:07:42	and map task consists of donald the code for direction providing task
0:07:47	so we took about five hundred past and b and c two and fifty from
0:07:53	b and about
0:07:55	slightly less and five wonderful map task
0:07:57	and
0:07:59	basically
0:08:00	the way this a good was a random ten selection of turn units ending with
0:08:04	a question mark
0:08:06	where we also eliminated type questions and turns with missing text and tens of missing
0:08:11	text
0:08:14	the polish data was taken from the scruggs corpus which is basically the spoken part
0:08:19	of the polish at national corpus
0:08:22	and that consists of that corpus consists of about two and fifty thousand utterances
0:08:27	and for this we chose about two hundred yes
0:08:33	okay so the basic results
0:08:35	all that for english the
0:08:40	the other classes is it is less than three percent so we have
0:08:44	more or less close to ninety something ninety seven percent coverage with this taxonomy
0:08:51	perhaps not hugely surprising
0:08:54	the most we can cluster responses in all three corpora in english
0:08:57	and approach i direct on says
0:08:59	in the bnc the biggest next biggest classes clarification requests
0:09:03	so be the next biggest classes indirect announces the map task the second biggest is
0:09:08	you know actually
0:09:12	ignore the case where you respond with another utterance which is about the same situation
0:09:16	but it's not respond to the question
0:09:19	so you can already see that is fair amount of variability across corpora
0:09:23	for polish the two most frequent last response is a on says so direct ones
0:09:28	and indirect ones
0:09:31	and then the next to a frequent classes or the i don't know class
0:09:35	and the ignore class
0:09:39	so this is roughly the results and obviously it'll be a bit hard for you
0:09:42	this is all in the paper so you can if you in the resulting in
0:09:46	detail you can you can see it there but you can see at the top
0:09:49	you have the of course the most of the masses taken by the
0:09:54	the direct on says
0:09:57	but with
0:09:58	the task oriented of course getting much more direct then
0:10:02	something like to be and see that the and open corpora like b and c
0:10:06	and spokes
0:10:07	and there you see and then you can see
0:10:10	that's there's a fair amount of variability
0:10:14	across corpora fulfilled you different kind of classes telling you that you know you're not
0:10:18	gonna get a good on you can't there's no chance of getting good characterisation of
0:10:22	this problem just by looking one corpus
0:10:25	and as we found in the in the question study at is quite a large
0:10:30	variability across corpora in terms of these kind of distributions so the nature of the
0:10:37	corpus really
0:10:38	again that's not very surprisingly influence is very much the kind distributions you get
0:10:45	as far as real reliability goes
0:10:49	so we did a in a i just speak about the english part of reasons
0:10:54	the time but the polish is discussed in table two so we did an intent
0:10:59	eight is a study we had would had to my main annotators were also
0:11:05	paper
0:11:07	and a work try to students in object linguistics
0:11:12	l two speakers of english and then to when assemble training sessions with the me
0:11:17	and
0:11:19	both annotated around five hundred paths and from this we extracted five hundred calmly bad
0:11:26	has
0:11:27	and
0:11:28	we got a cap of for our about one sixty five a group and of
0:11:32	about one sixty six
0:11:36	there
0:11:38	ninety four cases where the annotation to the disagreements where annotations agreements a occurred
0:11:45	the main disagreements concerned direct on says this is indirect on says so weak that's
0:11:51	about a third able to disagreements
0:11:54	it could no versus
0:11:56	change the topic acknowledgement a direct depending question and a direct answer
0:12:02	and acknowledgement this is the
0:12:06	so direct indirect disagreements mostly occurred with why questions how questions and what is x
0:12:12	doing questions
0:12:14	and visa cases where on says all by a lot sentential
0:12:19	and for which has been significant can promising theoretical literature on how to characterize onset
0:12:24	so just to give a couple of examples
0:12:27	so we have here case with the why question why deep tan'll to know that
0:12:34	well as the new guy
0:12:35	so the annotators disappear i was a direct or indirect and eventually was a resolve
0:12:41	to indirect
0:12:45	and is another example a web
0:12:48	this is a four to one again to why question i thought very nice is
0:12:53	it no it isn't what is why isn't it "'cause" it isn't
0:12:56	and this with again to go clean direct on statistical model
0:13:01	and eventually resolved to an indirect on sit since it indirectly indicated is actually no
0:13:05	reason
0:13:07	okay so this is just we just to give you a sort of flavourful for
0:13:12	full
0:13:13	the nature of kind of disagreements and
0:13:14	ultimately
0:13:16	the fact that probably
0:13:19	this is a kind of task
0:13:20	where
0:13:21	a notion of annotate a more sophisticated notion annotation we wait which doesn't necessarily
0:13:26	lead to a resolution but leads to actual different kinds of judgements having to maintain
0:13:32	it is probably needed
0:13:35	okay so the final thing i'll just mention is
0:13:39	that sort of formal analysis that it
0:13:42	that is needed in order to solve this to
0:13:46	two can describe this problem formally
0:13:50	so in our original paper we provided rules within the cost is the follows them
0:13:56	that
0:13:58	characterized
0:13:59	how the coherence or of
0:14:02	these seven class of questions that can the kind of "'cause" response to questions
0:14:06	and to the extent that what we've
0:14:09	what the study is shown is that basically
0:14:13	the class all of
0:14:15	responses
0:14:15	is
0:14:17	basically on says
0:14:20	plus
0:14:21	direct indirect bounces plus
0:14:25	things that are address these basic issues
0:14:28	then we already have essentially a complete characterization of the response space
0:14:34	which i in again potentially in the in implement able form in the sense that
0:14:38	this is the cost easier formalism is i is it is a sort of information
0:14:41	state type formal is them so it's but actually giving you a
0:14:47	has potential for implementing a kind of
0:14:54	a for dialogue manager
0:14:56	so just to make a few a few comments in that respect the most basic
0:14:59	a notion of answered i you might we might say is
0:15:03	something one has been a cool simple answer would
0:15:08	so if you think of what a question is for mathematical if you essentially a
0:15:11	some kind of a allowed abstract
0:15:14	where does for broke white ball questions it's a i'm abstraction of a empty set
0:15:20	of variables and for the rich questions over a set from of one or more
0:15:25	variables
0:15:26	then
0:15:27	a simple utterances are of course for polar questions just the two polar opposites
0:15:32	and
0:15:33	for
0:15:36	for all other research questions they are on the instantiations and then negation is
0:15:42	and this is actually a system plots the hood if you're the corpus has pretty
0:15:46	good coverage as
0:15:49	we know from this is of course and a way of they're pretty a direct
0:15:55	way of talking about slot filling
0:15:58	but that the ultimate notion of on subword which a goal here about nist had
0:16:02	encode about a similar in the real lecture
0:16:06	have to be
0:16:08	actually ultimately
0:16:10	if you want really wide got a coverage have to include things the go beyond
0:16:14	simple onset would so it has to accommodate conditional
0:16:19	we demoralised and quantification on says
0:16:22	so this addresses some of these kind of questions that the silicone is been asking
0:16:26	what all these poor people who are just a filling slots
0:16:32	and so
0:16:34	that was so
0:16:36	again i'm not of the i don't have time hated to see how to say
0:16:38	how you can have formally deal these opinion the discussed in the recollection questions
0:16:46	but at the same time even though that there has been discussion of how to
0:16:49	accommodate these kind of
0:16:51	on says to
0:16:54	so that the that also direct on says
0:16:57	with still lacking a comprehensive empirically based experiment extracted tested account for
0:17:04	of right a wh words okay so the all of this the reading lectures based
0:17:08	on based on very small number of a examples just for a small number wh
0:17:13	words
0:17:16	and of course additional notion of their questions needs is some if an exhaustive knows
0:17:21	which has to prevent wrap traumatised
0:17:24	and
0:17:26	whether responses exhaustive well
0:17:29	can determine whether response of except the required for a query so this leads to
0:17:33	what we i mentioned before that we need to find a great sub division of
0:17:36	the honest categories
0:17:40	and therefore on the base of about this and some notion of a source the
0:17:44	best one can define question dependence
0:17:47	and that's that the basis for instance for kind of a rules that you can
0:17:52	give the dialogue manager like if a question some discussion respond with an utterance which
0:17:57	is a few specific another with this either provides an answer a whole a dependent
0:18:02	question-response so that an example of the kind of way of
0:18:07	characterizing the coherence of
0:18:09	various some classes of a responses
0:18:12	the fine across all mention which is the another very big class and has again
0:18:17	fit as a important
0:18:22	implications for the kind of information that you need annual
0:18:25	representations is clarification requests
0:18:30	sold in work by again there's been a quite all of the reckon work on
0:18:34	that going back to what by a matthew purver myself
0:18:38	where we showed how to account for the main class of clarification requests
0:18:42	users using rules that enable clarification questions to be relevant a given utterance
0:18:48	so the basic idea
0:18:50	we are going at any for details is that involves accommodating to context certain kinds
0:18:56	of clarification questions
0:18:58	with rules of this basic format
0:19:01	so that the input is so much as you given so much as you want
0:19:06	something state you would the constituent of this is actions on that application
0:19:11	then you can accommodate any of these kind of a this class of questions what
0:19:15	a mean by you one what would today is that you one or a kind
0:19:19	of a confirmation kind of questions
0:19:21	but knows to do this you could not do this just on the basis of
0:19:25	having
0:19:27	content based that the content of the question as input you need the whole sign
0:19:34	that's associated with an utterance
0:19:38	okay
0:19:39	so
0:19:41	conclusions so i presented here and initial study for the for
0:19:46	what we've as possible we can see the first detailed form in depend characterisation of
0:19:50	response basic queries
0:19:53	and k s
0:19:55	a lot of things that need to be done
0:19:57	so one thing is cross question type in comparison so as to set the that
0:20:02	the question-response pairs that we looked at
0:20:04	was selected randomly and obviously it's interesting to consider distribution responses relative to fix parts
0:20:09	of questions
0:20:11	so different foster wh questions polar questions and so on and again we can be
0:20:14	facial that they'll be different distributions different fit for different parts of questions
0:20:20	we need to apply machine learning to acquire the response classification scheme of course so
0:20:25	there's been some work on this severance than the men ability of nonsentential utterances
0:20:29	so that that's a
0:20:32	subclass of the kind of was a response exist
0:20:34	so that gives hope for the non the bit of some of the sound classes
0:20:38	we anticipate that it's a some of these classically pretty difficult to learn for instance
0:20:44	the ones that a
0:20:47	heavily based on inference like indirect ounces and a more will change the topic
0:20:54	obviously
0:20:56	as everybody here is interested in down to just implementation so we'd like to test
0:21:01	these in a in a dialogue system with a fairly sophisticated management the of that
0:21:05	for instance of the goat is class
0:21:09	so there's been some initial experiments and work by arrive at allen and gotten but
0:21:14	and of course another here we gave you a
0:21:18	bit of work on english and polish
0:21:20	but of course get which only show you some differences
0:21:24	so is a signal us a given challenge we think is of see how you
0:21:27	test this classification with languages that
0:21:31	lack
0:21:32	speech corpora such as
0:21:35	about ninety five percent ninety percent of aligned is on have
0:21:39	so we we're starting doing some work on this respectively we go and we and
0:21:45	just by using online games
0:21:47	online games the proposed
0:21:56	we have a few minutes of questions
0:22:03	hi first of all thank you for your talk this is really interesting
0:22:07	so the question that i had was that if you could go back to slide
0:22:10	seven please
0:22:16	so you're example here for the changing of the topic a it seems to me
0:22:21	that this is not exactly a changing of the topic because you're staying on the
0:22:25	same topic they asked you what your answer was in us to what there's at
0:22:28	the same general topic but rather more of an indirect refusal to insert
0:22:32	so i was wondering it seems like changing the topic is always an indirect refusal
0:22:37	to answer and we consider a refusal to answer as part of your ontology
0:22:43	so i mean this is a kind of indirect basically thing that you might is
0:22:47	an implicature all providing this kind of response is that you know you're trying to
0:22:53	i mean you certainly not addressing this issue right
0:22:57	so we i
0:22:58	in our original work we actually suggested that these kind of it commented that when
0:23:04	you provide is kind of response
0:23:06	then
0:23:07	maybe for posting reasons that the most common way to do this is by taking
0:23:12	have a question which is kind unify able with the with the original one with
0:23:16	more general one you know we should talk about what or whatever is well though
0:23:21	this is you know
0:23:23	so this work some for many cases but
0:23:25	i mean the more general thing is just to provide i mean you could you
0:23:28	can you know you can sort of do this kind of changes topic
0:23:32	in a way that it can be less smooth of course but you know by
0:23:35	throwing something that is quite different and these things also happen so this
0:23:39	this is this the smoothest way of doing it just from in proposing point of
0:23:43	view
0:23:43	but it's not have a well that's gonna be
0:23:46	in this way
0:23:47	so you the basic dynamics
0:23:51	for this coherence have to ultimately allow you to that
0:23:55	as a consequence potentially
0:23:57	to get one of these questions eliminated
0:24:00	so that works in the in the setup did you did you see any instances
0:24:05	of like direct refusals to answer a question in your corpora
0:24:08	where somebody asks a question somebody just as i don't i don't one answer that
0:24:11	are i refuse to install i mean they're the coming the not very calm the
0:24:15	this another common but and how would your skin like what class would you with
0:24:21	that many well that's the
0:24:27	so that's here so these of the character ones that a about the issue that
0:24:31	is down the underlying issue of changing the topic i see thank you
0:24:37	we have time for another question
0:24:45	thanks a i wanted to follow up on your future work about the relating to
0:24:50	quite a types of questions
0:24:53	and i guess you have and only about but i wonder how much of your
0:24:58	differences in the corpora might be due to different distribution of questions that are in
0:25:04	those corpora versus distribution of types of answers to those types of questions
0:25:10	then also maybe could use quickly comment on how you define question "'cause" i think
0:25:15	you so there's just a question mark them a corpus so you're
0:25:18	you can probably one reason you're not including declarative for direct a no so basically
0:25:24	we in terms of a pharmaceutical questions we just
0:25:28	doing this kind of family
0:25:32	so i mean that's kind of building on the fact that
0:25:35	you know transcription has decided this is a question
0:25:40	and so that basically means that it's typically going to be i'd average questions all-pole
0:25:44	questions
0:25:46	which could be either you know they could also be so the declarative the data
0:25:50	that have a question mark the end so but they usually have the same basic
0:25:53	function as well as a sort of draw people what
0:26:00	so you're set your i mean i guess because since we have done this we
0:26:04	don't know
0:26:06	and we i think it's all the cnn interesting question also exactly you know
0:26:12	i again i'm not aware of what the street address this all that is look
0:26:16	at you know what the difference we have actually in a as the c l
0:26:20	paper that we had a two thousand seven of oracle financial mapping and me we
0:26:25	actually did have some tables of the different distributions of different kinds of wh questions
0:26:31	both
0:26:32	clauses and lexical ones so there is some work on this actually but you know
0:26:41	that was just for one of hope that i think of some b and c
0:26:44	so it's
0:26:47	forward
0:26:50	i don't think the speaker one moment

Characterizing the Response Space of Questions: a Corpus Study for English and Polish

Oral Session 6: Evaluation and Data

Jonathan Ginzburg, Zulipiye Yusupujiang, Chuyuan Li, Kexin Ren and Paweł Łupkowski