Speech Transcript - The Role of Conversation Context for Sarcasm Detection in Online Interactions

0:00:16	okay thanks and but i'll and the buttons and then this a joint work with
0:00:20	the next five rate and small integration
0:00:24	so
0:00:26	some present detection on viral are any detection has recently become a popular and of
0:00:31	the problem
0:00:32	and the method of the these detection research is looking into the utterances of what
0:00:37	the point and of utterance and in isolation
0:00:40	however
0:00:41	as we know that sarcasm is a complex phenomena in natural language and speaker intent
0:00:46	is often unclear a list to provide additional context so this conversation context background knowledge
0:00:53	from of doctors
0:00:54	the topic of discussion in section
0:00:56	and that's why some researchers have argued that these are the sort of additional context
0:01:02	has to be provided to understand sarcasm bitter
0:01:05	and that actually triggers to our research here we look into basically to research questions
0:01:11	that far does conversation context held in predicting some present but conference in context is
0:01:17	that one type of context
0:01:19	and can identify what part of the conversation context actually triggering a sarcastic reply
0:01:26	so before going to the modeling let's look into some of the examples that we're
0:01:31	looking you know researcher
0:01:33	so this that we'd
0:01:34	and whether a user is saying that one more reason to feel really great of
0:01:39	what's line and then to put the has text are present which would be used
0:01:42	as a label
0:01:43	now is looking to that with this is
0:01:46	a hard to understand that why this to it is a sarcastic to eat right
0:01:50	but of the same time we see that the street was a reply to one
0:01:54	other tweet with the symbolic can see that at the rate
0:01:57	and
0:01:58	that was the reply to another to it from another user
0:02:01	where it says that plane window shades or open so that people can see
0:02:05	so we can understand that the user is sarcastic about the experience of line here
0:02:10	and that's why to put i would that has text are present
0:02:13	no let's look at a another domain
0:02:16	and this
0:02:16	these the discussion thread or stun it's taken from the internet argument corpus
0:02:21	but this user again
0:02:24	replying to some other user and reading about sarcastic about their reading have it
0:02:30	and think that all your reading too much into by
0:02:33	is to look into the context forcing we see that the point x this way
0:02:36	a lot longer than we because it's taken from a discussion forum posts and i'm
0:02:41	going to the detail of the post but what's the person is saying that you
0:02:44	think that well the what is not sixty million years old it's only two thousand
0:02:48	years old and they bring
0:02:50	definitely different arguments
0:02:52	and the user to is just saying that go you're eating too much into by
0:02:55	the and that's what they're
0:02:57	sarcastic about
0:02:59	so the outline of my job here two days like with fast discuss about though
0:03:04	i don't results are some of the state-of-the-art in sarcasm detection i'll also talk about
0:03:08	but data
0:03:09	and then i'll go over the first research question that can conversation context l in
0:03:14	detection some thousand and then they're seconds question that what the context actually triggering the
0:03:19	sarcastic reply and can identify that
0:03:22	and finally alicante would and a point to some future work
0:03:28	so as explained before there's a sergeant sarcasm environment detection in the recent years
0:03:33	and that of the research looking to see this as a typical binary classification problem
0:03:38	of not be which we we're doing like for an entity tagging or
0:03:42	relation extraction that sector
0:03:44	but there are some of the directions of the research also has come up in
0:03:47	started and the second recently and out just if you talk about that are for
0:03:50	instance right of it all the look into the context in can really characteristic of
0:03:54	sarcasm a then in our previous work in yemen we looked sarcasm detection as a
0:04:00	what sense disambiguation problem
0:04:02	i miss rate all they have looked into cognitive feature set as i tracking feature
0:04:06	for sarcasm data sent to find a lot they look into timbre posts where images
0:04:11	and the text can be combined for a second predictions and yesterday we saw one
0:04:16	poster from a horribly at all that the looked into rhetorical questions where
0:04:20	a lot of sarcastic utterances are actually rhetorical questions
0:04:24	from the respected of role of context
0:04:27	that's all the users have looked into orders context so that they actually model orders
0:04:31	previous x in a previous
0:04:34	post like into it
0:04:35	to understand that if the authors of sarcastic on that
0:04:38	are there is some work in the conversation context also especially balance but they look
0:04:42	into like n-gram characteristics and their performance we don't the context was not much different
0:04:48	and finally we think that combining wall knowledge how would be like really crucial to
0:04:54	understand sarcasm but
0:04:55	and the same time we think it's
0:04:56	much harder problem so for instance if we see this to it
0:05:01	that users thing that driveway that we are having a long time so that means
0:05:04	that
0:05:05	alignment has something to do with green and this sort of all knowledge is very
0:05:09	difficult
0:05:10	to include
0:05:11	in a system
0:05:13	your research we
0:05:14	particularly look into the conversational context here
0:05:18	so we use two sets of data the four cities like discussion four
0:05:22	and that was that it partially released in the lasagna conference eigenvalue rugby at all
0:05:28	and the structure is that you have a sarcastic response and which replies to a
0:05:33	context and vocalic discussion forum post
0:05:37	some of the characteristic of the data that the data was like allocated that comment
0:05:40	level using crowdsourcing and it's a balance set of training data close to five thousand
0:05:46	a post
0:05:47	from the context and response
0:05:50	we also collected data from twitter
0:05:52	and we look specifically into sarcastic with which are actually like previous tweet and when
0:05:58	we
0:05:59	we collected the previous fight using this parameter like at user
0:06:03	and then whenever possible we just collect the full set of the dialogue so sometimes
0:06:08	like one user is sarcastic about out of their tweet and that we use they
0:06:12	can reply to another with so we have collected the full trade
0:06:15	and labels are provided by the authors so that's one different from the discussion forum
0:06:20	where it's not just one delegating here the labels are provided and we use that
0:06:25	have sex sarcasm and has tech sarcastic to understand what to do sarcastic to each
0:06:30	other
0:06:31	again
0:06:32	it sort of on about in balance that but it's still close like two thousand
0:06:36	and thirteen thousand it for sarcastic and on circuits degrees respectively
0:06:41	and more than thirty bucks of the data we had a couple of sentences as
0:06:45	a context
0:06:45	also it should be given as like into it as max for the contextual characteristics
0:06:50	here
0:06:52	looking to the model can conversation context help in sarcasm detection
0:06:57	our baseline
0:06:59	features for svm linear kernel with discrete features
0:07:02	and do that the features with actually a part one very well in previous the
0:07:06	sarcasm detection research and that's where you're using
0:07:09	n-gram features we use like bigram and trigram and unigram we also used two sets
0:07:15	of lexicons like they are lexical for sentiment and
0:07:18	liwc louis for pragmatic features
0:07:21	absolutely like alteration of the sentiments between the point text and sorry a sarcastic post
0:07:27	and then we used a set of sarcasm markers which are basically indicators of sarcasm
0:07:32	a and that's been used in many research and linguistic and communication signs are for
0:07:37	instance like morpho syntactic features so use of various interjections questions examine some signs
0:07:44	and then when you're doing like speech recognition i'm sarcasm the voice into an snr
0:07:49	modulations is like they have like strong features
0:07:52	but one opinion to natural language and people like typing the put like this kind
0:07:55	of typographic feature set this quotation mark a different type of you multi phones
0:07:59	you move these are used also capitalisation such as this what the time showing like
0:08:04	never
0:08:05	and finally a list of intensify record is also shown to be a strong features
0:08:10	in sarcasm detection like the worst degrade based and beta next section
0:08:14	we also experience with the lstm network of which are able to a lower long
0:08:19	distance dependencies
0:08:20	and in our architecture we are using two lstm one read stuff context and one
0:08:25	reads the response
0:08:27	your six print with that consent base variations where first we used a lot and
0:08:31	sentence level attention and that's like a hierarchical model
0:08:35	and in the second we give the word embeddings static and we didn't do any
0:08:39	a bit attention on the words but we only put that consent on the sentence
0:08:43	of the context because we also one two
0:08:46	answer our second visits question that
0:08:47	what part of the context also help in identifying the sarcasm so that's why you're
0:08:52	doing the sentence level only
0:08:55	this is a schematic diagram of our utterancelevel for the sentence a sentence but seconds
0:09:01	and so you can see that the left hand side we have one lstm to
0:09:04	learn the content
0:09:05	and in right hand side we have one lstm to learn the response the real
0:09:09	there is a sentence embedding then you have the hidden layer then we have attention
0:09:13	and then you have the context vector
0:09:15	and what we mean here that we are concatenating the final vector representation from the
0:09:19	context and the response concatenating and then we're passing it to the softmax
0:09:25	a finally we also experimented with other variation the fella stimulus the conditional density and
0:09:30	that was introduced from the deep mine group for the a natural language inferencing tasks
0:09:34	like textual entailment
0:09:36	what we're doing here is that again we are using two lstm
0:09:39	but the response lstm is conditioned on they represent on the context of spam
0:09:44	so which means that
0:09:45	the sale state is for the response is initialized by the final state of the
0:09:50	context and
0:09:51	that's
0:09:52	been shown that it's like really a strong character structure architecture for
0:09:57	nestling in inferencing task
0:10:01	more some more details that we split the data into a dt in general like
0:10:04	training is like eighty percent data ten percent data we used for the parameter tuning
0:10:09	and we use two sets of word embeddings for discussion forums so we use that
0:10:13	standard go would be ten few hundred dimensional what effect model that for twitter we
0:10:18	used as basic model that was been trained on tweets and we have used you
0:10:21	know this is for
0:10:24	no so the results are this talk about some of the macro-average the that allowed
0:10:28	us point out something important how values and
0:10:31	the more i thought the number seven in their in more details in the paper
0:10:36	here first we look into the comparison of the is em results with the response
0:10:41	and context bus response we said that into a terry there is a some of
0:10:46	that some part of the improvement is they're using the context
0:10:49	but not in the discussion forum data and we suspect that
0:10:52	it's might be possible because a lot of and the context of soul all in
0:10:56	the discussion forums and
0:10:57	is v m which are mostly based on n-gram features for not every to learn
0:11:01	that
0:11:03	if you look into the lstm variations we only the we don't that instant model
0:11:08	we see that
0:11:09	the first observation is using context help
0:11:12	for both lstm models here
0:11:15	but at the same time they list in conditional model which was kind is the
0:11:19	weather response was conditioned on the context
0:11:21	it performs reasonably well so we see there is like
0:11:25	five to six percent improvement of if one for the discussion forum and for the
0:11:29	preacher results more
0:11:31	and finally that the results from using attention over the sentences with the response and
0:11:37	with the context and we see that sort the treated it to the context and
0:11:41	response performs past and so the discussion forum it just like comparable to the conditional
0:11:47	now we i'm not showing the results we when we do the attentional what in
0:11:52	sentence because that are phones a little more than that
0:11:55	and what we think there isn't could be that
0:11:57	are we don't have enough training data and when you're putting attention what on the
0:12:00	warden sentence level that are too many parameters to model so in the future we
0:12:04	hope that
0:12:05	well when you're we can put more data under discussion forum at which are especially
0:12:09	to
0:12:11	experiment in that direction and that might prove better
0:12:15	no
0:12:16	we go to the second research question that and the identify
0:12:20	what of the context figure out the sarcastic reply
0:12:23	on what we're doing here that's basically we're looking to that happens and wait and
0:12:27	if the indicate what part of the context
0:12:30	triggers the sarcastic reply here
0:12:33	so we evaluated i mention weight with a crowdsourcing experiment on amazon mechanical turk where
0:12:38	we asked i don't parse that we
0:12:41	poster sarcastic reply on the context from the discussion forum and we have that are
0:12:46	those that are can you identify one or more sentences from the context
0:12:52	but you think that we got the sarcastic reply
0:12:54	and we selected the
0:12:56	dark colours where like really
0:12:59	much higher qualifications so we thought that due to be able to do that starts
0:13:04	we're and with our own the eighty five heats here and we selected only the
0:13:07	post with the context length are between three to seven sentence because we think that
0:13:11	if we put context much longer which will be a hard task for that are
0:13:14	percent they might be interested to do that
0:13:17	and we use the major regenerating and interestingly when you compared to that can wait
0:13:22	we found that
0:13:22	ask for more than forty possible but i'm the sentence that got the highest
0:13:27	attention weight is also
0:13:29	comparable to what that are course the thinking and the put the majority voting for
0:13:33	that
0:13:34	so if you see into the same example that i should be for are regarding
0:13:39	the reading have it up because i will
0:13:42	left hand side of the sentences and the right hand side with show that heat
0:13:46	map
0:13:46	so for the heat map also in the left hand side that engine the weight
0:13:49	and the right hand side it just how we represent the majority voting
0:13:53	we see that the for sentence was selected from that can generate and also but
0:13:57	are cars
0:13:58	but at the same time that can send don't put much weight into the other
0:14:02	sentences for the target still think that
0:14:05	there is some wait for the second
0:14:06	and part sentence
0:14:12	we also looked into to analyze that consent weight in the light of like how
0:14:17	it's actually telling us about sarcasm
0:14:20	and we in the paper we actually discussed what various characteristic of sarcasm but in
0:14:24	the interest of time i'll us basically look into one issue which no less the
0:14:29	context in country
0:14:30	so you can think of the conquered context incongruity is characteristics of sarcasm and
0:14:36	with this very simple example i can say that okay say
0:14:40	i'll going to the
0:14:42	emergency room one every week so there is a sentiment like positive sentiment loss
0:14:47	but there is a negative situation like going to the abundance a room on every
0:14:51	weekend and so you can think of this early in the country
0:14:54	or inconsistency between the sentiment of the situation and they're a lot of research in
0:14:59	a linguistics and communication and also in philadelphia where people talk about this context in
0:15:04	one greedy and the think that
0:15:06	this one of the very strong characteristic of sarcasm
0:15:09	a here in our research we actually found out that a lot of time that
0:15:13	i've been sent especially for that which you're the one example am showing
0:15:17	has actually puts more weight into that includes features here so for instance in this
0:15:23	example of the context which was that
0:15:26	it talking about like advertisement saying right now it just totally do you press
0:15:30	about c of mediocre s
0:15:32	and the response was like all really got see what you're doing
0:15:35	and we see that depressed and mediocre actually got more weight
0:15:40	with the
0:15:41	lots one responses got see so you can see that there is something some sort
0:15:44	of incongruity and there are some more examples in our paper is that we
0:15:49	i showed that how this context incongruity is happening here
0:15:54	so at the same time we also looked into sarcasm markers are which is basically
0:15:58	indicators like explicit indicators of sarcasm
0:16:02	and we seems like especially here i can send a concluding much weights on the
0:16:06	markers such as various multi columns anymore jeez
0:16:09	you can think of and those actually actions and put more weight and
0:16:12	that means the model of thinking that these are like some features
0:16:16	a one interesting thing was we found out that sometime in the context you have
0:16:20	a like cost like anymore tickle like a set face
0:16:23	and then in the response maybe you have a like smiley face
0:16:27	and that attention models actually learning that these opposite characters of emotive content may be
0:16:33	small feature and that would be
0:16:34	a small weight over there so that so interesting observation
0:16:38	at the same data like interjections like are and those words actually have received a
0:16:44	lot of attention which so
0:16:47	but
0:16:48	one of constant here that
0:16:51	however
0:16:52	plus the classification task was not forced only to run on the attention the weight
0:16:56	and it's again
0:16:58	not does it say tall in two thousand fifteen to have discussed
0:17:02	on this topic that
0:17:03	the integration based on any attention rate has to be taken
0:17:07	care taken with here and so you have to be little courses here because
0:17:10	the classification was not only working on this with
0:17:13	so
0:17:15	we're interpreting here and we are mostly like observing the races
0:17:19	so to confuse
0:17:21	we found out that for sarcasm detection when you're using contextual information from the dialogs
0:17:25	it shows like better accuracy
0:17:27	also we i think to identify what portion of the context may trigger the sarcasm
0:17:32	and we ran some experience on the interpretation of the context to analyze a different
0:17:37	attitude of such as a like context incorporate the n
0:17:39	other characteristics or read discussed in the paper
0:17:43	a sort of future work we are interested in to like for large scale experiment
0:17:47	and we hope that there will be like more data released from the discussion forum
0:17:51	that we can use for more parameter tuning especially like into the award and second
0:17:56	sentence level able to attention models
0:18:00	also we saw yesterday on posted at how the comments which are reply to the
0:18:05	sarcastic was where you stand
0:18:07	in this kind of analysis so we also interesting
0:18:11	another thing here to observe you have that the three the data is a self-labeling
0:18:15	sarcastic data so people are
0:18:17	posting to its and they are posting the highest tax themselves
0:18:21	but in the discussion forum the sarcasm was perceived to because you have other annotators
0:18:25	coming and then they are not doing so
0:18:28	we are a very interesting to do this sort of analysis that what's the difference
0:18:31	between cells level and possible sarcasm
0:18:35	and finally we follow the in our experiment after doing a lot of error analysis
0:18:39	that
0:18:40	there are specific aspect of conversation like you more than on that users have used
0:18:46	many times and when we also downloaded or like the mechanical turk or they were
0:18:50	actually able to identify those here morrison ones which could actually trigger sarcasm
0:18:55	but i want model actually couldn't so that's one interesting direction that
0:19:00	we hope that will continue or research into
0:19:03	thank you
0:19:11	you questions
0:19:16	you like this slide on an attention experiment i guess for you had the one
0:19:20	example
0:19:23	a series of tweets and i think you where you compared against a crowdsource workers
0:19:27	this can you in that i think i
0:19:30	this is like for attention or no so this is like a we're showing the
0:19:34	discussion forums one so we experiment with that are course only on the discussion forum
0:19:38	and we
0:19:39	proposed like the sarcastic pose
0:19:41	and also the context for the context for like multiple sentences so we have that
0:19:45	doctors that
0:19:46	okay like
0:19:47	we already telling that this is a sarcastic or stand this the context that actually
0:19:51	has triggered the sub doesn't
0:19:52	so how about that can identify the tweet sentence is more important
0:19:56	and we just like five doctors and they put their voting on the sentences that
0:20:00	they think that okay it's one is more important than is to an effect your
0:20:04	and here the right hand side it's sort of the heat map but it's showing
0:20:08	the method of importing
0:20:10	and then we compare the majority voting with that mention that was and here for
0:20:14	this to be example yes one
0:20:16	actually has got a maximum attention the weight from or experiment at the same time
0:20:20	that are first thing that actually also like doctors think that this of the thing
0:20:24	is that actually
0:20:25	i might be triggering the sarcasm
0:20:48	in this
0:20:49	sponded then
0:20:52	again
0:20:56	so this the macro-average f
0:21:00	sorry the macro-average if once we are showing here between the sarcastic and on circuits
0:21:04	to capture
0:21:10	so we basically to the macro-average if one from the
0:21:14	both binary classes so in the paper we actually shortly
0:21:18	separately all categories but just for the interest of time taken that macro-average report
0:21:30	so c plus are means that we're doing the c is like the context
0:21:34	and the response and the all like all the response model
0:21:39	is that
0:21:48	right this normal questions then that's thanks to john again

The Role of Conversation Context for Sarcasm Detection in Online Interactions

Oral Session 3: Modeling Semantics and Pragmatics

Debanjan Ghosh, Alexander Richard Fabbri and Smaranda Muresan