Speech Transcript - Towards Full Text Shallow Discourse Relation Annotation: Experiments with Cross-Paragraph Implicit Relations in the PDTB

0:00:16	okay so
0:00:18	i talk is on discourse relation annotation
0:00:21	in very research for modeling discourse relations
0:00:24	i realise corpora annotated with such relations so for example we have
0:00:29	the rst dt corpus based on the rst the penn discourse treebank pdtb based on
0:00:35	at a time
0:00:36	and the and adjust corpus are based on sdrt their other corpora as well
0:00:41	so in other frameworks
0:00:42	i'm not covering all of them here
0:00:45	the penn discourse treebank which is the focus of my talk is a large-scale annotated
0:00:49	corpus
0:00:51	annotated over a one million word a wall street journal corpus
0:00:55	it's been used widely in the community for a lot of experimental work
0:00:59	as well as the framework we now apply to annotate
0:01:04	other text including of the champs and languages
0:01:08	however the current version of the corpus pdtb to does not
0:01:12	providing supposed to validation of its source text
0:01:15	there's work on going to address these gaps
0:01:18	in between speech
0:01:19	using the next version of the corpus pdtb three we do so
0:01:24	either a current work addressing the gaps
0:01:28	focuses on intra sentential relations which are relations of arguments in the same sentence
0:01:34	along with some modifications to existing annotations most of which involve modifications
0:01:40	resulting
0:01:42	in the sense hierarchy that out show later on
0:01:45	i talk focuses on the critical kind of gal in the
0:01:48	class of intra sentential relations which and relations with arguments and different sentences
0:01:53	so just a very quick overview of the annotation framework for those of you are
0:01:57	not familiar with it
0:01:59	the pdtb follows the lexically grounded but purely neutral approach to the representation of discourse
0:02:04	relations
0:02:04	which means that the annotation shallow
0:02:07	without committing to dependencies or structures beyond individuals relations
0:02:11	i discourse relations hold between two abstract object arguments that are named r one and
0:02:16	arg two using syntactic conventions
0:02:19	in the example that you see here and another example of the arg one is
0:02:22	in fact alex and r two is in bold
0:02:25	relation such a good either by explicit connectives in which case the
0:02:29	the get the relation type label of explicit
0:02:32	so in this example but is the explicit discourse connectives that relates this two sentences
0:02:37	in the relation of contrast
0:02:38	and that's because
0:02:40	these two attorneys offices men had from manhattan teachers e
0:02:44	they average of different number of criminal cases
0:02:52	and when relations are not triggered by explicit connectives and that of the by decency
0:02:56	between sentences and you're multiple things can happen first be made hidden for discourse relation
0:03:02	for which we can insert a connective
0:03:05	and such that the resulting text sounds reasonably readable and coherent
0:03:09	the relation type label for this is implicit
0:03:12	so in this example here
0:03:15	they're talking about the mac issues being hit but with investors and then the second
0:03:19	are two examples
0:03:20	talks about how this company just goes offer of the ventures was oversubscribed
0:03:25	the annotator inferred what we call the institution relation for most of the inserted the
0:03:30	connective for example it sounds
0:03:32	reasonably readable and coherent
0:03:34	in other cases we infer a discourse relation
0:03:37	but inserting a connective explicit relations used for just under c and that's because the
0:03:42	relation is been expressed in some other man and not for the connective
0:03:46	the relation type here is labeled all flex
0:03:48	so in this example below
0:03:51	and the we have a subject verb sequence that prompted expressing the relation of results
0:03:56	between the two sentences
0:03:59	basically the plant that sometimes been destroyed and people at this other company know what
0:04:03	it the same thing's gonna happen to them and then the r two sentence talks
0:04:07	about
0:04:09	what they are going to do as a result of that vary
0:04:13	to other of relation types can be got mean these that used in context and
0:04:18	jaw and nora and actually later on and when to talk about how they're done
0:04:22	some revisions to these two labels but basically and real are entity gazed relations which
0:04:27	means that
0:04:28	you cannot insurgent any explicit connective
0:04:30	and the sentences are related by virtue of some and a for reference
0:04:35	but just like to some entity across the two sentences
0:04:37	some of these relations actually do involve coherence relations
0:04:41	all the pdtb doesn't regard them as such
0:04:43	and they involve a background relation or of continuation relations so this example
0:04:48	the first one for injured is actually sort of a background where you have this
0:04:52	demonstrable these as the art for it link up for the r two sentence giving
0:04:57	some background about that are from nikos and their derivatives
0:05:01	and lastly we get an oral where not to mean intensity based relation holds
0:05:06	and this is due there are some changes is to how the sleepless finally drawn
0:05:14	with respect to arguments of a relations arguments can the annotated depends upon the type
0:05:18	of relation
0:05:19	so the r two of explicit relations is always
0:05:21	some part of the sentence or clause containing connective
0:05:24	but the art one can be anywhere in the private x
0:05:27	for all of the relation types like i said are what are not or only
0:05:30	annotated when adjacent
0:05:33	arguments can be extended to include additional clauses and sentences in all cases except nor
0:05:38	l
0:05:38	but there's a strong minimalistic constraints that wise inclusion of only the minimally necessary text
0:05:44	need to train i
0:05:48	in section finally the sense hierarchy the in the in the work that we did
0:05:53	be using the modified since hierarchy
0:05:56	in before pdtb three which was presented that law last year
0:06:00	one going to law the details you want to the more have some slides for
0:06:04	later on
0:06:05	but basically at the top level four classes from pdtb to a reading
0:06:10	temporal
0:06:12	comparison contingency an expansion of their been changes
0:06:15	at level two and level three
0:06:16	most of these changes involve a one-to-one mapping
0:06:20	from and to be to be due to its tree which we have implemented automatically
0:06:25	others are reviewed and annotated manually
0:06:28	in this what we came up to new senses that we got evidence for
0:06:32	one is have a for all four question answer pairs
0:06:35	and the other is introducing level three sentences for the asymmetric instantiations relation
0:06:42	okay so back to the focus of this talk
0:06:45	the as i said just in a critical gap in the class intersentential relation so
0:06:50	if you look at the current version of the corpus
0:06:53	you'll find that all sentences at containing an explicit connective
0:06:57	that's really that sentence to something in the prior text
0:07:01	have been annotated but almost all there are some gaps
0:07:05	and then
0:07:06	within paragraphs all the sentences without such a connective have been annotated
0:07:13	but
0:07:14	in the paragraph the first sentence of the paragraph to process the paragraph boundary
0:07:20	remain an annotated in the current corpus
0:07:22	so in this example here which shows the for six sentences of an article the
0:07:26	last one six as an explicit connective at a paragraph boundaries the mt lines indicate
0:07:31	paragraph boundaries
0:07:32	that has been annotated are one is not shown but the sense contrast indicates that
0:07:37	the annotation in the corpus
0:07:39	if you look at this third paragraph the internal implicit relations
0:07:45	between svms fourth conjunction and between that's for ns five conjunction is also been annotated
0:07:50	what's not annotated what are not annotator is to industry in and that's because they
0:07:55	are at the paragraph boundaries
0:07:57	there are more than twelve thousand such an annotated tokens in the card version of
0:08:00	the corpus their total for almost forty thousand tokens the corpus
0:08:05	these an annotated tokens constitute thirty percent of all intersentential discourse context
0:08:11	and eighty seven percent of all across paragraph intra sentential context the remaining thirty percent
0:08:15	being a transcriber explicit relations
0:08:20	so why worry about things for the forces automatic prediction there's been some work to
0:08:24	show that
0:08:25	they we can get improvements and very hard task of implicit relations stance classification
0:08:29	with the sequence model and also other work that incorporates features but no neighboring relations
0:08:35	but there's also the goal of understanding and global discourse structure so the a shallow
0:08:40	analysis the pdtb is also in service of the emergent discourse a global discourse structure
0:08:46	which you can get by combining the individual variations together but in order to do
0:08:50	that we need the complete sequence of relations over texas which is not their corpus
0:08:55	currently
0:08:57	so our goals are to identify a challenges and explore the feasibility of annotating
0:09:01	these course paragraph implicit relations on a large scale
0:09:05	and to produce a set of guidelines to annotate such relations reliably and also a
0:09:10	representative subset of pdtb text
0:09:13	annotated with complete sequences of intra sentential relations
0:09:17	and this can be done by merging the existing interest relations in the pdtb across
0:09:22	paragraph implies that are currently annotating
0:09:26	in our experiments we selected a fifty four texts from the pdtb corpus to cover
0:09:30	a range of sub genres and lines
0:09:33	they contain four hundred and forty paragraph initial sentences which we call
0:09:39	current hyper for sentence si pfs
0:09:41	and that are not already related to the prior text by an intersentential explicit connective
0:09:46	and the experiments were spread over three phases
0:09:50	that's just how things happened we didn't pan adapt
0:09:54	in phase one we study text to develop it can be understanding of the task
0:10:01	two expert annotators which is basically myself and kate forbes the second order
0:10:05	we work together to discuss annotate ten text
0:10:10	containing hundred and thirty tokens
0:10:11	a but we did not enforce the pdtb adjacency constraint for implicit because we wanted
0:10:16	to explore the full complexity of the task
0:10:19	each token was annotated for its relation type
0:10:22	sense and minimum spans
0:10:26	what refinement phase one was that fifty two percent of the paragraph initial sentences to
0:10:31	their prior are one arguments from
0:10:33	and adjacent unit involving a
0:10:35	prior paragraphs last sentence which is p l s for short
0:10:40	the remaining forty eight percent form the non-adjacent relation
0:10:44	this argument distribution is similar to that of course a graph express it's which are
0:10:49	also non-adjacent roughly half the chart
0:10:51	so whether this would be shown more generally something that we wanted to explore
0:10:57	with for their annotation in the next phase
0:10:59	we also found that working together we could isolate and agree upon b r one
0:11:04	of not only the adjacent relations but also the non-adjacent ones
0:11:07	so second hypothesis was to explore whether both adjacent and on adjacent relations could be
0:11:14	annotated reliably on a large scale
0:11:18	this led us to a big out
0:11:21	another hundred and three tokens over ten text
0:11:24	in which we did doubled like the annotation that was
0:11:29	that would give us
0:11:30	the results to
0:11:32	to understand whether
0:11:34	this would be advantageous large scale
0:11:37	and be annotated these tokens regardless of whether the arguments adjacent or non-adjacent
0:11:43	is the results from phase two
0:11:46	the first thing you know what is that the agreement on whether and it not
0:11:52	relation is adjacent or non-adjacent just that
0:11:55	binary decision
0:11:56	was reasonably high at seventy six percent
0:12:00	but when we looked into each of these groups are within the ones that on
0:12:05	which we agreed to be adjacent and the ones on which would be to be
0:12:08	non adjacent
0:12:09	and we found that generally exact match agreement in which the tokens for you need
0:12:14	for type sense an argument spans
0:12:16	with low for both
0:12:18	which shows the general difficulty of the task
0:12:20	of annotating and discourse paragraph in place it's
0:12:25	when you relax
0:12:26	at the argument matching
0:12:30	to relax them in a multi constraint so we did two kinds of relaxation on
0:12:35	the arg min max and one with sentence-level max with you disagreed at the sentence
0:12:38	level on some part of a span
0:12:41	we allowed that to quantize agreement
0:12:44	and also relaxing that even for the to allow for soup residential overlap
0:12:49	lead to further both of these like to further boost an agreement
0:12:56	but what what's interesting what's actually agreement what's much worse for
0:13:02	the non-adjacent a relation than for adjacent relations of the non-adjacent but the forty seven
0:13:07	percent and the adjacent relations where a texas sixty one percent
0:13:11	so that is to so that and also when we discuss the disagreements
0:13:17	we found that while it was almost possible to reach consensus
0:13:21	the time and effort that was required for and you to getting the non adjacent
0:13:26	relations was twice greater than for it you to get better adjacent relations
0:13:31	this led us to conclude that annotating the arg one of identifying be are gonna
0:13:35	on it uses was
0:13:37	in the with the current
0:13:39	state of the guidelines and the baby doing things is prohibitive so large scale annotations
0:13:44	therefore for now a decision was made to maintain the pdtb adjacent against change pitches
0:13:49	you know
0:13:50	you know we consistent with the existing constraints for adjacency
0:13:54	and focused on full annotation of only adjacent relations
0:13:57	but we also wanted to annotate the presence of a not reduce it implicit relation
0:14:03	which is not there right down the pdtb
0:14:05	with some kind of underspecified marking and we use the label of north somewhere else
0:14:10	for that
0:14:12	this led us to going back to what is that evaluated we consider
0:14:17	the way the labels of interest and we're are assigned in the current version of
0:14:22	the pdtb
0:14:23	so in the current assignments we get an enter a if there is an entity
0:14:26	based on here installation
0:14:28	holding between i one and r two and the discourse that expanded around some entity
0:14:32	you're not too
0:14:33	either by continuing the narrative around it
0:14:36	or supplied background about
0:14:38	but we also did not intra currently
0:14:40	is that does not hold if this was inactive well here installation of background a
0:14:45	continuation
0:14:46	but it's just some entity coreference between the two arguments
0:14:49	and this is the case even if r two forms an on and you also
0:14:54	upon the non-adjacent implicit relation
0:14:57	we didn't know we have if and rather or no discourse relation holds
0:15:01	but this is the case even if r two is also part of a non-adjacent
0:15:05	implicit relation
0:15:06	and we get an oral when the r two is not part of a discourse
0:15:10	at all
0:15:12	this happens the by lines like to do you have to alter sort of information
0:15:16	or if the start of a new article in a single was you general file
0:15:22	which can happen sometimes
0:15:27	so that our goal to encode the presence of non jews an implicit relations
0:15:31	the current assignments are problem
0:15:33	because this information is spread across vote labels so we
0:15:37	da
0:15:38	presence of an implicit non-adjacent relation
0:15:40	it's better cross enter eleanor l so we cannot tell
0:15:45	identify that
0:15:46	an ambiguous lee
0:15:48	the current assignments also confound the presence of
0:15:50	a semantic into debates coherence relation with the presence of milk reference and that's the
0:15:55	problem with in be here
0:15:57	so what we want to do is to unambiguously identified non-adjacent an implicit relations just
0:16:03	the presence of it
0:16:04	but which we use the label was them
0:16:07	this also allows us to get
0:16:09	that semantic
0:16:10	entity based coherence relations unambiguously
0:16:13	and also a unambiguously identify the two pieces of nowhere
0:16:19	and which are two is not related to anything in the project
0:16:23	and one is get this is an example of an underspecified non-adjacent implicit relation but
0:16:27	if i start to talk about it is usable the five minutes
0:16:32	so employing the decisions and enhancement made in phase two in phase three the remaining
0:16:38	two hundred and seven
0:16:40	have a cross paragraph tokens from thirty four text what double blind the annotated again
0:16:46	the be enhanced guidelines
0:16:48	and these of the results from face three and in order to do a consistency
0:16:52	comparison to see the differences of given the phase two results here as well
0:16:56	the first thing to know what is that the agreement on whether that relation was
0:17:00	reduced and are not adjacent that binary decision
0:17:02	was approximately the same which is good
0:17:06	the second thing that over the agreed tokens at the proportion of non adjacent relations
0:17:11	was also approximately the same as in phase two
0:17:14	and this supports the hypothesis about the high frequency of knowledge is an implicit and
0:17:18	therefore
0:17:19	what suggesting that the word pair annotating
0:17:22	overall agreement with the most relaxed metric what an argument spans with higher phase three
0:17:28	and sixty two percent and phase two features that forty three percent and this is
0:17:32	partly because of the back-off to underspecified annotation of non adjacent relations
0:17:36	but also
0:17:39	but also a we have a high agreement on the scent annotation of the adjacent
0:17:44	relations which is sixty nine percent and based you from sixty one percent in phase
0:17:48	two
0:17:48	other just partly due to were enhanced guidelines for annotating the same injured relations
0:17:55	the improvement on the sensors also better reflect argument agreement so there's an increase in
0:17:59	exact match to forty two percent from twenty four percent and phase two there is
0:18:03	less agreement due to the super sentential argument overlap
0:18:08	thirty percent reduction to thirteen percent and thirty percent and face to
0:18:12	there is more disagreement of the sentence level so we have fourteen percent disagreement
0:18:17	at sentence level from seven percent face to but these are not close the loop
0:18:21	they showed that people minus syntactic differences upper example one attitude included or excluded an
0:18:26	adjunct or an attribution trees with the other didn't
0:18:31	so that's not such a major semantic difference the final distributions over the all the
0:18:36	fifty four text you also but back to the phase one interface to detect and
0:18:39	reality that the enhanced guideline
0:18:43	additional in the talk table there
0:18:44	as you can see there is and the final glued data shows an equal proportion
0:18:49	of adjacent a non adjacent relations again supporting hypothesis about the distribution
0:18:54	the senses show that forty percent of the these course parameters it's have are elaboration
0:19:00	relations to start with detail
0:19:02	forty five percent on five senses with greater than five percent frequency
0:19:06	and the remaining fifteen percent sentence sentences with less and pipes and frequencies are spread
0:19:11	across nine different senses
0:19:14	in conclusion
0:19:16	adjacent implicit discourse relations across paragraphs can be annotated reliably
0:19:21	are gold standard sense distribution
0:19:23	together with the frequency of the semantic and rows suggest that was paragraph implicit relations
0:19:29	carry
0:19:30	very semantic content and
0:19:31	standard proportions
0:19:33	and are therefore what annotating
0:19:35	the current goal is to annotate approximately two hundred pdtb which is about seven hundred
0:19:40	tokens a two hundred text with these guidelines and which is estimated which we have
0:19:45	estimated required three minutes per token on average it's approximately thirty five minute thirty five
0:19:50	hours of annotation time parameter
0:19:53	the annotations will be distributed publicly by a get hard hopefully by the end of
0:19:58	this man
0:20:00	most of the text and the subset are also annotated in rst dt corpus so
0:20:04	it will allow for useful comparisons of relation structures across the two frameworks
0:20:10	a few juggles include a studying the distribution of sensors and patterns of sentences in
0:20:15	the text along the lines previous work
0:20:18	but now able for text relations sequences
0:20:22	we also want to develop guidelines of identifying
0:20:25	the arg ones of the more difficult non-adjacent implicit relations to ensure that it can
0:20:29	be done reliably and efficiently
0:20:32	and to this end we're looking at enhancements
0:20:35	to the pdtb annotation to better lower formant in visualization which is not possible currently
0:20:41	the tool
0:20:42	all these intra sentential relations and their arguments in the text
0:20:46	we also want to explore a two pass annotation methodology that would allow the more
0:20:50	difficult across paragraph
0:20:51	non adjacent relations to be annotated in the second pass
0:20:55	because the sequences of intra sentential relations from the first pass the adjacent once and
0:21:00	then trivial systematic structures to inform the second pass annotation
0:21:05	thank you
0:21:12	you very much having question
0:21:20	i'll start
0:21:21	so
0:21:24	it is not this and annotating a non-adjacent relation is a very difficult task for
0:21:33	so you see
0:21:36	i want to build a model trained on the state it takes relations with distinct
0:21:43	properties this model
0:21:46	have to
0:21:48	to be able to accurately predict
0:21:52	these non-adjacent
0:21:55	right so in these sequences models
0:21:59	kind of approaches
0:22:01	they try to do joint modeling aware there trying to predict entire sequences so the
0:22:10	the contextual information the neighboring relations are very would be a very important feature in
0:22:15	the production of these knowledge is an implicit relations
0:22:18	so although it's not the case all of the time
0:22:21	but in many of these cases
0:22:24	you get these non-adjacent relations where the intervening material is just
0:22:28	at
0:22:29	is a real operations all of what's annotated as the non-adjacent arg one
0:22:34	so if you can
0:22:36	if you can get that in the structure of the relations
0:22:40	labels correctly for this for that intervening material
0:22:44	then when you get to the next sentence that itself gives you
0:22:48	the information to sort of course that's to the next higher level
0:22:53	that's one of things
0:22:55	and then there's a very useful feature is enough around
0:22:58	so there's a lot of discourse the axis
0:23:01	that appears in these non-adjacent context
0:23:05	because when you want to refer to any binned event eventuality that's non-adjacent you end
0:23:10	up using these definite descriptions that there are data they take nature
0:23:19	thank you
0:23:27	okay let's think the speaker

Towards Full Text Shallow Discourse Relation Annotation: Experiments with Cross-Paragraph Implicit Relations in the PDTB

Oral Session 4: Context in Discourse and Dialogue

Rashmi Prasad, Katherine Forbes-Riley and Alan Lee